IROS 2024 Papers

Active reconstruction technique enables robots to autonomously collect scene data for full coverage, relieving users from tedious and time-consuming data capturing process. However, designed based on unsuitable scene representations, existing methods show unrealistic reconstruction results or the inability of online quality evaluation. Due to the recent advancements in explicit radiance field technology, online active high-fidelity reconstruction has become achievable. In this paper, we propose GS-Planner, a planning framework for active high-fidelity reconstruction using 3D Gaussian Splatting. With improvement on 3DGS to recognize unobserved regions, we evaluate the reconstruction quality and completeness of 3DGS map online to guide the robot. Then we design a sampling-based active reconstruction strategy to explore the unobserved areas and improve the reconstruction geometric and textural quality. To establish a complete robot active reconstruction system, we choose quadrotor as the robotic platform for its high agility. Then we devise a safety constraint with 3DGS to generate executable trajectories for quadrotor navigation in the 3DGS map. To validate the effectiveness of our method, we conduct extensive experiments and ablation studies in highly realistic simulation scenes.

Verbal communication plays a crucial role in human cooperation, particularly when the partners only have incomplete information about the task, environment, and each other’s mental state. In this paper, we propose a novel cooperative communication framework, Goal-Oriented Mental Alignment (GOMA). GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents’ mental states that are relevant to the goals. This approach enables an embodied assistant to reason about when and how to proactively initialize communication with humans verbally using natural language to help achieve better cooperation. We evaluate our approach against strong baselines in two challenging environments, Overcooked (a multiplayer game) and VirtualHome (a household simulator). Our experimental results demonstrate that large language models struggle with generating meaningful communication that is grounded in the social and physical context. In contrast, our approach can successfully generate concise verbal communication for the embodied assistant to effectively boost the performance of the cooperation as well as human users’ perception of the assistant.

The International Space Station (ISS) crew has to complete various tasks in a limited time. The Int-Ball project is one of the activities aimed at the efficient utilization of crew time. Int-Ball2 is an autonomous mobile robot that conducts imaging operations in place of crew members by remote control from the ground. This paper first introduces the design of the guidance, navigation and control (GNC) and propulsion systems of Int-Ball2, offering autonomous flight inside the ISS. This robot has unique features compared to other free-flying robots in space, including its small size for use in crowded environments and its high thrust despite its size. In addition, the navigation system uses a visual SLAM algorithm, which does not rely on external markers. The results of orbital performance verification tests for 6-DOF translational and rotational motion are also presented in this paper. Our analysis indicated that the proposed GNC algorithm provided sufficient performance to conduct the required photography. Finally, the results of ground tests simulating the microgravity environment are compared with the results of orbital experiments to evaluate the differences in the robot’s mobility performance in these tests. The results confirmed that the ground verification method was valid for achieving stable on-orbit mobility.

State-of-the-art LiDAR calibration frameworks mainly use non-probabilistic registration methods such as Iterative Closest Point (ICP) and its variants. These methods suffer from biased results due to their pair-wise registration procedure as well as their sensitivity to initialization and parameterization. This often leads to misalignments in the calibration process. Probabilistic registration methods compensate for these drawbacks by specifically modeling the probabilistic nature of the observations. This paper presents GMMCalib, an automatic target-based extrinsic calibration approach for multi-LiDAR systems. Using an implementation of a Gaussian Mixture Model (GMM)-based registration method that allows joint registration of multiple point clouds, this data-driven approach is compared to ICP algorithms. We perform simulation experiments using the digital twin of the EDGAR research vehicle and validate the results in a real-world environment. We also address the local minima problem of local registration methods for extrinsic sensor calibration and use a distance-based metric to evaluate the calibration results. Our results show that an increase in robustness against sensor miscalibrations can be achieved by using GMM-based registration algorithms. The code is open source and available on GitHub3.

In this paper, we present a novel motion planning algorithm that inherits the strengths of both optimization and search-based planners. Optimization-based planners use the gradient of an objective function to generate a desired path, whereas search-based planners operate on a graph capturing the salient topology of a robot’s free space. A class of optimization-based planners leverages prior information, modeled as a probability distribution of target locations in an environment, to guide path generation. We embrace one specific measure, referred to as ergodicity, which encourages a robot to spend a proportion of its time, weighted by the distribution, where it is likely to find targets of interest. Methods that minimize ergodicity were not designed to handle obstacles in the environment, and augmented approaches that add "soft" constraints for obstacles to the cost function may still yield a path that collides with an obstacle. In this work, we present a hybrid approach that first generates a graph of the environment’s free space, followed by searching the graph with ergodicity as a heuristic. Our approach not only restricts the search to the free space, thereby avoiding obstacles by design, but also generates trajectories with low ergodicity values. Extensive testing on 125 test scenarios with varying degrees of clutter, information distribution, and robot start locations illustrate the efficacy of our algorithm.

Humans can teleoperate robots to accomplish complex manipulation tasks. Imitation learning has emerged as a powerful framework that leverages human teleoperated demonstrations to teach robots new skills. However, the performance of the learned policies is bottlenecked by the quality, scale, and variety of the demonstration data. In this paper, we aim to lower the barrier to collecting large and high-quality human demonstration data by proposing a GEneraL framework for building LOw-cost and intuitive teleoperation systems for robotic manipulation (GELLO). Given a target robot arm, we build a GELLO controller device that has the same kinematic structure as the target arm, leveraging 3D-printed parts and economical off-the-shelf motors. GELLO is easy to build and intuitive to use. Through an extensive user study, we show that GELLO enables more reliable and efficient demonstration collection compared to other cost efficient teleoperation devices commonly used in the imitation learning literature such as virtual reality controllers and 3D spacemouses. We further demonstrate the capabilities of GELLO for performing complex bi-manual and contact-rich manipulation tasks. To make GELLO accessible to everyone, we have designed and built GELLO systems for 3 commonly used robotic arms: Franka, UR5, and xArm. All software and hardware are open-sourced and can be found on our website: https://wuphilipp.github.io/gello/.

In a gas distribution mapping (GDM) task, the objective of a mobile robot is to map the gas concentrations of an airborne chemical over a region of interest using onboard sensing. Given the limited battery budget available to the robot, covering the entire area to measure gas concentrations at every location might be infeasible. Assuming that the robot only has a budget for b meters of travel, in the rest of the locations, gas concentrations can be inferred using a supervised machine learning technique, namely the Gaussian Process (GP). In this paper, we propose a novel technique that combines deep reinforcement learning and GP regression to find an effective policy for GDM. We have implemented the proposed technique in Python within a 16×16 4-connected plane. We have used six types of Gaussian plumes to validate our presented approach. Compared to two popular baselines, our approach outperforms greedy and random exploration by 62% and 151% in terms of earned rewards, while outperforming them by 47% and 345%, respectively, in terms of the precision of gas distribution modeling in all test cases without obstacles. Our approach also improves the coverage of the exploration while consequently reducing the uncertainty in the prediction.

3D multi-object tracking is an important component of the perception module in autonomous driving systems. Due to the limitations of a single sensor, tracking methods based on either LiDAR or cameras always have certain deficiencies. Fusion-based tracking methods have received increasing attention. However, existing fusion-based tracking methods often underutilize image information, ignore the respective effects of appearance information and 2D detection results, and lack further analysis on the simultaneous use of both. This paper proposes a novel camera-LiDAR fusion tracking framework that primarily relies on the motion model using 3D objects. It fully leverages the appearance information and 2D detection results simultaneously from images and introduces three modules to reduce the number of false positive samples, false negative samples and ID switches, respectively. Besides, the entire tracking process does not require global processing and achieves online tracking. The proposed method achieves competitive results on the KITTI tracking dataset with 78.50% HOTA. Compared with EagerMOT using the same 3D and 2D detectors, the HOTA metric improved by 4.11%. Code is available on https://github.com/zengwz/FusionTrack.

Prosthetic limb abandonment remains an unsolved challenge as amputees consistently reject their devices. Current prosthetic designs often fail to balance human-like performance with acceptable device weight, highlighting the need for optimised designs tailored to modern tasks. This study aims to provide a comprehensive dataset of joint kinematics and kinetics essential for performing activities of daily living (ADL), thereby informing the design of more functional and user-friendly prosthetic devices. Functionally required Ranges of Motion (ROM), velocities, and torques for the Glenohumeral (rotation), elbow, Radioulnar, and wrist joints were computed using motion capture data from 12 subjects performing 24 ADLs. Our approach included the computation of joint torques for varying mass and inertia properties of the upper limb, while torques induced by the manipulation of experimental objects were considered by their interaction wrench with the subject’s hand. Joint torques pertaining to individual ADL scaled linearly with limb and object mass and mass distribution, permitting their generalisation to not explicitly simulated limb and object dynamics with linear regressors (LRM), exhibiting coefficients of determination R = 0.99 ± 0.01. Exemplifying an application of data-driven prosthesis design, we optimise wrist axes orientations for two serial and two differential joint configurations. Optimised axes reduced peak power requirements, compared to anatomical configurations, by exploiting high torque correlations (r = −0.84, p < 0.05) between Ulnar deviation and wrist flexion/extension joints. This study offers critical insights into the functional requirements of upper limb prostheses, providing a valuable foundation for data-driven prosthetic design that addresses key user concerns and enhances device adoption.

We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mango. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

Robots are especially useful in cold, remote, and inhospitable environments such as polar regions and extraterrestrial settings. Due to subfreezing temperatures and limited resources in these environments, robots made of ice are particularly advantageous. In this paper we demonstrate how the solid and liquid phases of water, and transitions between these phases, can be leveraged into common robot designs for modular robots, robot arms, rovers, and soft robots. We explore how robots can utilize structural elements made of ice and exploit the phase change between ice and water to augment their capabilities. Additionally, we do a scaling analysis of ice structural elements to provide insight on their performance at different length scales and ambient temperatures.

Multi-robot rendezvous and exploration are fundamental challenges in the domain of mobile robotic systems. This paper addresses multi-robot rendezvous within an initially unknown environment where communication is only possible after the rendezvous. Traditionally, exploration has been focused on rapidly mapping the environment, often leading to suboptimal rendezvous performance in later stages. We adapt a standard frontier-based exploration technique to integrate exploration and rendezvous into a unified strategy, with a mechanism that allows robots to re-visit previously explored regions thus enhancing rendezvous opportunities. We validate our approach in 3D realistic simulations using ROS, showcasing its effectiveness in achieving faster rendezvous times compared to exploration strategies.

Hierarchical control for robotics has long been plagued by the need to have a well defined interface layer to communicate between high-level task planners and low-level policies. With the advent of LLMs, language has been emerging as a prospective interface layer. However, this has several limitations. Not all tasks can be decomposed into steps that are easily expressible in natural language (e.g. performing a dance routine). Further, it makes end-to-end finetuning on embodied data challenging due to domain shift and catastrophic forgetting. We introduce our method – Latent Codes as Bridges (LCB) – as an alternate architecture to overcome these limitations. LCB uses a learnable latent code to act as a bridge between LLMs and low-level policies. This enables LLMs to flexibly communicate goals in the task plan without being entirely constrained by language limitations. Additionally, it enables end-to-end finetuning without destroying the embedding space of word tokens learned during pre-training. Through experiments on Language Table and Calvin, two common language based benchmarks for embodied agents, we find that LCB outperforms baselines (including those w/ GPT-4V) that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.

Co-design optimization strategies usually rely on simplified robot models extracted from CAD. While these models are useful for optimizing geometrical and inertial parameters for robot control, they might overlook important details essential for prototyping the optimized mechanical design. For instance, they may not account for mechanical stresses exerted on the optimized geometries and the complexity of assembly-level design. In this paper, we introduce a co-design framework aimed at improving both the control performance and mechanical design of our robot. Specifically, we identify the robot links that significantly influence control performance. The geometric characteristics of these links are parameterized and optimized using a multi-objective evolutionary algorithm to achieve optimal control performance. Additionally, an automated Finite Element Method (FEM) analysis is integrated into the framework to filter solutions not satisfying the required structural safety margin. We validate the framework by applying it to enhance the mechanical design for flight performance of the jet-powered humanoid robot iRonCub.

Soft pneumatic bending actuators (SPBAs) are commonly employed in soft robotics due to their unique characteristics, including safety, low weight, speed, and load capacity. However, the combination of pneumatics with soft materials causes SPBAs to exhibit nonlinearities and infinite degrees of freedom, complicating their dynamic modeling. In this work, we present how the dynamics of SPBAs can be adjusted to a fractional order model (FOM), showing an approach for their empirical identification. We also present a method for designing fractional order controllers (FOCs) for this type of actuators, based on the inversion of the empirical FOM. This modeling and control is applied to a modular SPBA made of a smart hydrogel, which endows the actuators with self-healing, self-adhesion, and self-sensing capabilities.

UAVs and other fast moving robots often need to keep track of distant objects. Conventional zoom cameras commit to a particular viewpoint, and carrying multiple zoom cameras for multi-object tracking is not feasible for power limited robotic systems. We present a dual camera setup that allows tracking of multiple targets at nearly 1km distance with high-resolution. Our setup includes a wide angle camera providing a conventional resolution view and a MEMS driven zoom camera that can query a specific region within the wide angle camera (WAC). We built and calibrated the two-camera system and implemented a real-time image fusion pipeline. We show multi-object tracking and stabilization in real world scenarios.

Executing bimanual manipulation tasks on humanoid robots introduces additional challenges due to inherent spatial and temporal coordination between both hands. In our previous work, we proposed the Bimanual Manipulation Taxonomy, which defines categories of bimanual manipulation strategies based on the coordination and physical interaction between both hands, the role of each hand in the task, and the symmetry of arm movements during task execution. In this work, we build upon this taxonomy and provide a formalization of temporal and spatial constraints associated with each category of the taxonomy. This formalization uses Petri nets to represent temporal constraints and differentiates between relative and global targets. We incorporate these constraints in a category-specific controller to enable reactive adaptation of the behavior according to the respective coordination constraints. We evaluated our approach in simulation and in real-world experiments on the humanoid robot ARMAR-6. The results demonstrate that category-specific constraints can be enforced when needed while maintaining flexibility to accommodate additional constraints.

Assistive devices are to be designed with the objective of use in daily-life as well as broad adoption by end users. In this context, it is necessary to tackle usability challenges by properly detecting and acting in accordance to user intents while minimizing the device installation complexity as well. In the case of physical assistive devices, using force/torque sensors is advantageous to detect user intent compared to EMG interfaces, but it remains difficult to correctly translate the detected intent into actuator motions. Focusing on upper-limb assistive robots, the user voluntary force is commonly used with a controller based on an admittance approach which leads to relatively poor reactivity and requires the user to develop force throughout the movement which can lead to fatigue, particularly for people with upper-limb impairments. This work proposes a Force-Triggered (FT) controller which can initiate and maintain movement only from short force impulses. The user voluntary forces are retrieved from total interaction forces by subtracting the passive component measured beforehand during a calibration phase. This paper presents the design of the proposed FT controller and its preliminary testing on pick-and-place tasks compared to an admittance strategy. This experiment was performed with one participant without impairment, equipped with an upper-limb exoskeleton prototype designed from recommendations of physical medicine therapists. This preliminary work highlights the potential of the proposed FT controller. Also, it provides directions for future work and clinical trials with end-users to assess the proposed FT approach usability while used alone or in the form of an hybrid controller between FT and admittance strategies.

In this article, we propose a generalization of a Deep Learning State-of-the-Art architecture such as Retentive Networks so that it can accept video sequences as input. With this generalization, we design a force/velocity predictor applied to the medium-distance Human-Robot collaborative object transportation task. We achieve better results than with our previous predictor by reaching success rates in testset of up to 93.7% in predicting the force to be exerted by the human and up to 96.5% in the velocity of the human-robot pair during the next 1 s, and up to 91.0% and 95.0% respectively in real experiments. This new architecture also manages to improve inference times by up to 32.8% with different graphics cards. Finally, an ablation test allows us to detect that one of the input variables used so far, such as the position of the task goal, could be discarded allowing this goal to be chosen dynamically by the human instead of being pre-set.