IROS 2024 Papers

Integrating visual observation with linguistic instruction holds significant promise for enhancing robot navigation across unstructured environments and enriches the human-robot interaction experience. However, while panoramic RGB views furnish robots with extensive environmental visuals, current methods significantly overlook crucial semantic and depth cues. This incomplete representation may lead to misinterpretation or inadequate execution of language instructions, thereby impeding navigation performance and adaptability. In this paper, we introduce SEAT, a semantic-depth aware cross-modal transformer model. Our approach incorporates an efficient panoramic multi-type visual encoder to capture comprehensive environmental details. To mitigate the rigidity of feature mapping stemming from the freezing of pre-training encoders, we propose a novel region query pre-training task. Additionally, we leverage an improved dual-scale cross-modal transformer to facilitate the integration of instructions, topological memory, and action prediction. Extensive experiments on three language-guided robot navigation datasets demonstrate the efficacy of our model, achieving competitive navigation success rates with fewer parameters and computational load. Furthermore, we validate SEAT’s effectiveness in real-world scenarios by deploying it on a mobile robot across various environments. The code is available at https://github.com/CrystalSixone/SEAT.

Vehicles capable of operating in more than one environment have been developed to solve real problems. Among them, the hybrid unmanned aerial-underwater vehicle (HUAUV) is receiving attention from the robotics community, mainly with a quadrotor-like configuration. However, this vehicle presents high energy consumption because of the larger mass required compared to the only aerial vehicle, limiting its autonomy. This work addresses the trajectory planning problem for a HUAUV. The method is based on Rapidly-exploring Random Trees (RRTs), a highly customizable planning technique. In addition, we propose two new heuristics to increase the energy efficiency of the hybrid vehicle. The first consists of biasing the tree expansion towards the environment with the lowest navigation cost, while the second one assigns estimated costs to nodes in the tree and chooses the least expensive trajectories. These techniques are evaluated in physically realistic simulation experiments performed in 135 scenarios. A comparative analysis of their performances is presented relative to the state of the art. We show that using efficient heuristics can significantly contribute to reducing energy consumption and even increase the average velocity in the missions performed by these vehicles.

Fixed-wing small uncrewed aerial vehicles (sUAVs) possess the capability to remain airborne for extended durations and traverse vast distances. However, their operation is susceptible to wind conditions, particularly in regions of complex terrain where high wind speeds may push the aircraft beyond its operational limits, potentially raising safety concerns. Moreover, wind impacts the energy required to follow a path, especially in locations where the wind direction and speed are not favorable. Incorporating wind information into mission planning is essential to ensure both safety and energy efficiency. In this paper, we propose a sampling-based planner using the kinematic Dubins aircraft paths with respect to the ground, to plan energy-efficient paths in non-uniform wind fields. We study the characteristics of the planner with synthetic and real-world wind data and compare its performance against baseline cost and path formulations. We demonstrate that the energy-optimized planner effectively utilizes updrafts to minimize energy consumption, albeit at the expense of increased travel time. The ground-relative path formulation facilitates the generation of safe trajectories onboard sUAVs within reasonable computational timeframes.

Multirotor unmanned aerial vehicle is a prevailing type of aerial robots with wide real-world applications. The energy efficiency of the robot is a critical aspect of its performance, determining the range and duration of missions that can be performed. This paper studies the energy-optimal planning of the multirotor, which aims at finding the optimal ordering of waypoints with the minimum energy consumption for missions in 3D space. The study is performed based on a previously developed model capturing first-principle energy dynamics of the multirotor. We found that in majority of the cases (up to 95%) the solutions of the energy-optimal planning are different from those of the traditional traveling salesman problem which minimizes the total distance. The difference can be as high as 14.9%, with the average at 1.6%-3.3% and 90th percentile at 3.7%-6.5% depending on the range and number of waypoints in the mission. We then identified and explained the key features of the minimum-energy order by correlating to the underlying flight energy dynamics. It is shown that instead of minimizing the distance, coordination of vertical and horizontal motion to promote aerodynamic efficiency is the key to optimizing energy consumption.

Energy sharing among modular robots enables sustainable operation of the system by maintaining energy balance among the modules. In this paper, we propose a novel energy sharing mechanism for FreeSN, a modular self-reconfigurable robot consisting of node and strut modules. Utilizing the feature that our modules are connected in a face-to-face manner, our method successfully establishes an energy sharing channel at almost any point on a sphere by placing transmission intermediaries at the interfacing face between modules, which is facilitated by the combination of brush contact and shell decomposition. Such mechanism also allows the utilization of the node module’s inner space for extra energy storage. A prototype of this energy sharing system has been implemented on FreeSN and rigorously tested. Our findings indicate that energy sharing is reliably established between modules; for strut modules positioned randomly on a node module’s surface, the probability of forming a valid connection is 56.6%. With orientation adjustment, a connection is achievable at nearly any position on the sphere, barring a few exceptional points. As a result, the operational endurance of the strut modules, which provide all the driving forces in the system, is markedly enhanced. This technique also holds potential for broader application across other freeform robotic platforms that incorporate conductive spherical surfaces for sliding connections.

This study introduces an innovative actuator that resembles a motor with a non-uniform permanent magnetic field. We have developed a prototype of the actuator by combining a standard motor, characterized by a uniform magnetic field, with a custom rotary magnetic spring exhibiting a non-uniform magnetic field. We have also presented a systematic computational approach to customize the magnetic field to minimize the energy consumption of the actuator when used for a user-defined oscillatory task. Experiments demonstrate that this optimized actuator significantly lowers energy consumption in a typical oscillatory task, such as pick-and-place or oscillatory limb motion during locomotion, compared to conventional motors. Our findings imply that incorporating task-optimized non-uniform permanent magnetic fields into conventional motors and direct-drive actuators could enhance the energy efficiency of robotic systems.

To this day, only a small number of industrial robots is used in assembly. One key reason for this is that specific contact situations require the introduction of force-control schemes. The parameters for those schemes are hard to select in practice, because they require in-depth expertise about the robot and the process. Learning-from-Demonstration (LfD) provides a powerful approach to intuitively parameterize robot programs by demonstrating the task at hand. However, when dimensions increase by including force or orientation, many LfD algorithms are hard to verify, understand and maintain, requiring expert knowledge to make adaptions, effectively making it a "black-box". This property renders them ineffective for usage in industrial applications. We build upon a system of composable skills, that can be easily adapted by experts without the need to demonstrate the task again. This approach to skill-based robot programming promises to address the issues of readability and maintainability by sequencing robot movements in skills and breaking them down into understandable (sub-)goals. In this paper, we combine skill-based programming with LfD, preserving both maintainability and intuitive parameterization. We present (a) an approach to parameterize and create sequences of hierarchies of force- and/or position-controlled robot skills from a LfD model, (b) which can be adapted by a user by hand with few, basic and understandable parameters, and (c) show its applicability on the real-world example of terminal clamp assembly. We achieve a reduction in teach-in time of 53.8% for variants, increased robustness against variance, and efficient tight stacking of clamps with a gap of ≤ 1mm.

Dementia affects 50 million people worldwide, underscoring the urgent need for effective interventions to enhance their well-being. While reminiscence intervention shows promise, its implementation is hindered by limited human resources, making machine-aided systems a viable automated solution for seamless photo-reminiscence sessions. In this paper, we introduce an empathetic response generation system specifically designed to enhance a question-only photo-reminiscence chatbot, with a focus on improving emotional context understanding and enhancing conversation engagement. We leverage Transformers to encode dialogue history, infer emotional states from user responses, and extract named entities. By combining template-based utterances with a retrieval chatbot, our system generates relevant and empathetic responses to user replies. Our system’s effectiveness is validated through human evaluations using a Likert-like scale to assess engagement levels. The results demonstrate that our approach surpasses both the question-only system and other models from existing works, including retrieval and generated models. This highlights our system’s potential to enhance interactions and engagement, advancing technology-driven interventions for dementia that improve well-being and quality of life.

In human-robot interaction (HRI), we study how humans interact with robots, but also the effects of robot behavior on human perception and well-being. Especially, the influence on humans by tandem robots with one human-controlled and one autonomous robot or even semi-autonomous multi-robot systems is not yet fully understood. Here, we focus on a leader-follower scenario and study how emotionally expressive motion patterns of a small, mobile follower robot affect the perception of a human operator controlling the leading robot. We examined three distinct emotional behaviors for the follower compared to a neutral condition: angry, happy, and sad. We asked participants to maneuver the leader robot along a set path while experiencing each follower behavior in a randomized order. We identified a significant shift in subjective attention toward the follower with emotionally expressive behaviors compared to the neutral condition. For example, the angry behavior significantly heightened participant stress levels and was considered the least preferred behavior. The happy behavior was the most preferred and associated with increased excitement by the participants. Integrating the proposed behaviors in robots can profoundly influence the human operator’s perceived attention, emotional state, and overall experience. These insights are valuable for future HRI tandem robot designs.

Depth estimationn is a critical topic for robotics and vision-related tasks. In monocular depth estimation, in comparison with supervised learning that requires expensive ground truth labeling, self-supervised methods possess great potential due to no labeling cost. However, self-supervised learning still has a large gap with supervised learning in 3D reconstruction and depth estimation performance. Meanwhile, scaling is also a major issue for monocular unsupervised depth estimation, which commonly still needs ground truth scale from GPS, LiDAR, or existing maps to correct. In the era of deep learning, existing methods primarily rely on exploring image relationships to train unsupervised neural networks, while the physical properties of the camera itself—such as intrinsics and extrinsics—are often overlooked. These physical properties are not just mathematical parameters; they are embodiments of the camera’s interaction with the physical world. By embedding these physical properties into the depth learning model, we can calculate depth priors for ground regions and regions connected to the ground based on physical principles, providing free supervision signals without the need for additional sensors. This approach is not only easy to implement but also enhances the effects of all unsupervised methods by embedding the camera’s physical properties into the model, thereby achieving an embodied understanding of the real world.

We present Embodiment Randomization, a simple, inexpensive, and intuitive technique for training robust behavior policies that can be transferred to multiple robot embodiments. While prior works require real-world data from multiple robots, or complex algorithmic adjustments to address the challenge of embodiment generalization, our approach leverages the power of simulation and large-scale reinforcement learning and can be easily integrated within existing policy learning methods. We show that policies trained with embodiment randomization implicitly perform system identification, enabling them to adapt to new embodiments during deployment. Our approach not only shows significant improvements in adapting to novel robot configurations, but also in generalizing from simulation to reality and contending with real-world perturbations, highlighting the potential of embodiment randomization in creating versatile and adaptable robotic navigation policies.

We introduce uncertainty-aware object instance segmentation (UncOS) and demonstrate its usefulness for embodied interactive segmentation. To deal with uncertainty in robot perception, we propose a method for generating a hypothesis distribution of object segmentation. We obtain a set of region-factored segmentation hypotheses together with confidence estimates by making multiple queries of large pre-trained models. This process can produce segmentation results that achieve state-of-the-art performance on unseen object segmentation problems. The output can also serve as input to a belief-driven process for selecting robot actions to perturb the scene to reduce ambiguity. We demonstrate the effectiveness of this method in real-robot experiments. Website: https://sites.google.com/view/embodied-uncertain-seg.

We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon tasks over a large workspace. Our system is modular: it deploys state of the art Large Language Models for task planning, Vision-Language models for semantic perception, and Point Cloud transformers for grasping. With semantic and physical safety in mind, these modules are interfaced with a real-time trajectory optimizer and a compliant tracking controller to enable human-robot proximity. We demonstrate performance for the following tasks: bi-arm sorting, bottle opening, and trash disposal tasks. These are done zero-shot where the models used have not been trained with any real world data from this bi-arm robot, scenes or workspace. Composing both learning- and non-learning-based components in a modular fashion with interpretable inputs and outputs allows the user to easily debug points of failures and fragilities. One may also in-place swap modules to improve the robustness of the overall platform, for instance with imitation-learned policies.

Soft robotic systems are inherently compliant, giving them unique capabilities not possessed by traditional rigid-bodied robot systems. Many soft systems rely on soft pneumatic actuators. One of the biggest downsides of such actuators is the need for bulky pressure-regulating devices and individual pneumatic supply lines. In this work, a miniaturized pressure-regulating system is developed and embedded into the unused space inside of a soft pneumatic McKibben actuator, enabling the simultaneous pressure control of multiple actuators connected to a single pneumatic supply line. This "valve-embedded" actuator is capable of regulating its internal pressure within 0.05 psi of a desired set point, even under external load. Compared to a conventional McKibben actuator driven by external valves, the valve-embedded actuator is experimentally shown to consistently achieve faster settling times. To showcase the practical application of the valve-embedded actuator on a robotic system, a 0.9m serial-linked robot driven by five independently controlled valve-embedded actuators was assembled, and was shown to achieve an average root mean square error of less than 1.5cm in a waypoint tracking experiment. The miniaturized pressure control system developed in this work is open source and could be embedded in any fluid-driven actuator, enabling more capable and densely actuated pneumatic soft robots.

Soft pneumatic actuators can accomplish various customizable deformation/motion through the distribution of cavities and gradients in stiffness. However, traditional manufacturing methods, say molding, struggle to produce soft actuators with both complex cavities and desirable stiffness distributions. Regular 3D printing methods usually need extra printheads for support materials to fabricate soft actuators with cavities. In addition, the printing quality and fidelity of the whole structure cannot be uniform due to the effect of gravity, especially for a soft actuator with overhang features. To fabricate a soft actuator of uniform fidelity but desirable stiffness distributions, we propose an embedded 3D printing approach with only one active mixing printhead. By adjusting the mixing ratio of the dual-component silicone, we can achieve designated stiffness gradients, ranging from 30.2 kPa to 198 kPa. With this approach, we successfully fabricate soft pneumatic actuators with overhang features, which exhibit programmable elongation and radial expansion. Additionally, we fabricate soft bending actuators which can achieve programmable workspaces due to their predetermined stiffness distribution.

Path planning has long been an important and active research area in robotics. To address challenges in high-dimensional motion planning, this study introduces the Force Direction Informed Trees (FDIT*), a sampling-based planner designed to enhance speed and cost-effectiveness in pathfinding. FDIT* builds upon the state-of-the-art informed sampling planner, the Effort Informed Trees (EIT*), by capitalizing on often-overlooked information in invalid vertices. It incorporates principles of physical force, particularly Coulomb’s law. This approach proposes the elliptical k-nearest neighbors search method, enabling fast convergence navigation and avoiding high solution cost or infeasible paths by exploring more problem-specific search-worthy areas. It demonstrates benefits in search efficiency and cost reduction, particularly in confined, high-dimensional environments. It can be viewed as an extension of nearest neighbors search techniques. Fusing invalid vertex data with physical dynamics facilitates force-direction-based search regions, resulting in an improved convergence rate to the optimum. FDIT* outperforms existing single-query, sampling-based planners on the tested problems in ℝ4 to ℝ16 and has been demonstrated on a real-world mobile manipulation task.

Accurate and reliable ego-localization is critical for autonomous driving. In this paper, we present EgoVM, an end-to-end localization network that achieves comparable localization accuracy to prior state-of-the-art methods, but uses lightweight vectorized maps instead of heavy point-based maps. To begin with, we extract BEV features from online multi-view images and LiDAR point cloud. Then, we employ a set of learnable semantic embeddings to encode the semantic types of map elements and supervise them with semantic segmentation, to make their feature representation consistent with BEV features. After that, we feed map queries, composed of learnable semantic embeddings and coordinates of map elements, into a transformer decoder to perform cross-modality matching with BEV features. Finally, we adopt a robust histogram-based pose solver to estimate the optimal pose by searching exhaustively over candidate poses. We comprehensively validate the effectiveness of our method using both the nuScenes dataset and a newly collected dataset. The experimental results show that our method achieves centimeter-level localization accuracy, and outperforms existing methods using vectorized maps by a large margin. Furthermore, our model has been extensively tested in a large fleet of autonomous vehicles under various challenging urban scenes.

Formal verification of robotic tasks requires a simple yet conformant model of the used robot. We present the first work on generating reachset conformant models for robotic contact tasks considering hybrid (mixed continuous and discrete) dynamics. Reachset conformance requires that the set of reachable outputs of the abstract model encloses all previous measurements to transfer safety properties. Aiming for industrial applications, we describe the system using a simple hybrid automaton with linear dynamics. We inject non-determinism into the continuous dynamics and the discrete transitions, and we optimally identify all model parameters together with the non-determinism required to capture the recorded behaviors. Using two 3-DOF robots, we show that our approach can effectively generate models to capture uncertainties in system behavior and substantially reduce the required testing effort in industrial applications.

Generative models are gaining momentum in both academic and industrial applications driven by the availability of large-scale datasets, especially in tasks involving Image-to-Image Translation. Meanwhile, poor human perception of nighttime environment has led to a demand for translation from night-vision infrared to day-vision RGB images. However, collecting such cross-modal training data at the same time is impossible due to the thermal imaging properties of infrared cameras, the challenge lies in constructing image pairs during the day and at night respectively, where the requirement for data alignment poses significant difficulties. In this paper, we propose a Pixel-level aligned Image Pair generation framework PIP to explore efficient colorization of high-resolution infrared images. Specifically, we first construct a 3D high-precision point cloud map for the purpose of establishing the correlation between day and night scenes. Corresponding point clouds of modal images are collected simultaneously during data acquisition to obtain image sensor poses by Global Matching with the map, which allows us to calculate the transformation relationship from infrared to RGB image coordinate systems based on the sensor parameters and depth information of the map. Leveraging the relationship, the pixel values of RGB image is projected onto the infrared image followed by optimization as the colored image. Accordingly, we present a dataset NUDT-PIP, the first of its kind containing large-scale pixel-level aligned cross-time infrared-RGB image pairs of complicated real road scenes. Experimental results demonstrate the reliability and strong applicability of our dataset in Image-to-Image Translation. Our code will be released at https://github.com/wjjjjyourFA/NUDT-PIP.