IROS 2024 Papers

Facial mimicry is crucial for human-like robots in human-robot interaction. The challenge is that the high diversity of facial expressions proposes difficulties in programming a robotic face to mimic human facial expressions using traditional methods. In this paper, we present a data-driven method to retarget human facial expressions to robotic faces without human effort. Our data collection is fully automatic, where only a robotic face and Apple ARKit are involved to sample actuator commands and record the resulting facial blendshape values. We trained a neural network that predicts blendshape values from commands, which is then used as a surrogate model to optimize command values to resemble given facial expressions. Experiments show that the proposed method has achieved lower error in terms of facial blendshape values than baselines. Moreover, the response time can be reduced to 0.2 seconds via TCP/IP through WiFi, offering great potential for real-time application. Our method is a novel framework for retargeting facial expressions to robotic faces, which can be incorporated into various human-robot interaction systems.

In this paper, a force servo for hydraulic robot joints is designed and applied to an inverted pendulum demo. First, the nonlinear nominal model of the hydraulic cylinder is reviewed, and the nominal integrator is introduced. Second, through a partial fusion of our modeling and control techniques, a force servo is designed based on a modification of the nominal integrator as an intermediate stage. Finally, against the non-short hydraulic pipes (1.9 m), the response is improved by the designed force servo in the presence of rotational motion effects.

Global localization is essential in enabling robot autonomy, and collaborative localization is key for multi-robot systems, allowing for more efficient planning and execution of tasks. In this paper, we address the task of collaborative global localization under computational and communication constraints. We propose a method which reduces the amount of information exchanged and the computational cost. We also analyze, implement and open-source seminal approaches, which we believe to be a valuable contribution to the community. We exploit techniques for distribution compression in near-linear time, with error guarantees. We evaluate our approach and the implemented baselines on multiple challenging scenarios, simulated and real-world. Our approach can run online on an onboard computer. We release an open-source C++/ROS2 implementation of our approach, as well as the baselines.1

In order to improve the terrain adaptability and energy efficiency of wheel-legged robot in complex environment, a dual-mode navigation system based on robot energy consumption model is proposed. Firstly, the obstacle trafficability is evaluated according to the maximum obstacle crossing capability of the robot, and the two-dimensional grid map is preprocessed. Secondly, the established energy consumption model is integrated into the evaluation function of A* algorithm, and the eight-adjacency expansion mode is improved to search the surrounding nodes according to the obstacle characteristics of the robot. In the obstacle-crossing area, the obstacle-crossing and obstacle-bypassing modes are intelligently switched based on the principle of minimum energy consumption. Finally, a dual-mode robot navigation system is built, and the experimental results show that the proposed navigation system reduces the average energy consumption, path length, and steering angle by 16.8%, 24.7%, and 31.18%, respectively.

This paper proposes a method for calibrating the external parameters of the LDS sensor and ToF depth camera based on three cylinders. This method obtains the scanning data of the side surfaces of the three cylinders at different postures by changing the posture of the robot. For the single-line laser plane scanned by the LDS sensor, three elliptical contours are obtained by intersecting with the side surfaces of the three cylinders respectively. The Random Sample Consensus (RANSAC) algorithm is used to obtain the coordinates of the center points of the three elliptical contours and two random points on each elliptical contour. For the three-dimensional point cloud image of the cylinder scanned by the ToF depth camera, the RANSAC algorithm is used to fit the central axis of the three cylinders. The nonlinear optimization equation is established using the three center points obtained from the three elliptical contours and the distances from the two random points on each elliptical contour to their corresponding central axes. In this paper, we propose to use a fusion method of the Powell algorithm and the BFGS algorithm to solve the nonlinear optimization equations to obtain the transformation matrix between the LDS sensor and the ToF depth camera. Finally, simulation and actual test are carried out based on the proposed method, and the influence of the initial value of the calibration parameter on the calibration result is discussed. The accuracy of the calibration algorithm in this paper is verified through comparative experiments of the calibration algorithm. The results show that the calibration accuracy of the proposed method is better than that of the traditional planar calibration method, and it has the characteristics of simple operation and high calibration accuracy.

Recent advancements in visual localization and mapping have demonstrated considerable success in integrating point and line features. However, expanding the localization framework to include additional mapping components frequently results in increased demand for memory and computational resources dedicated to matching tasks. In this study, we show how a lightweight neural network can learn to represent both 3D point and line features, and exhibit leading pose accuracy by harnessing the power of multiple learned mappings. Specifically, we utilize a single transformer block to encode line features, effectively transforming them into distinctive point-like descriptors. Subsequently, we treat these point and line descriptor sets as distinct yet interconnected feature sets. Through the integration of self- and cross-attention within several graph layers, our method effectively refines each feature before regressing 3D maps using two simple MLPs. In comprehensive experiments, our indoor localization findings surpass those of Hloc and Limap across both point-based and line-assisted configurations. Moreover, in outdoor scenarios, our method secures a significant lead, marking the most considerable enhancement over state-of-the-art learning-based methodologies. The source code and demo videos of this work are publicly available at: https://thpjp.github.io/pl2map/.

This paper introduces a new method for safety-aware robot learning, focusing on repairing policies using predictive models. Our method combines behavioral cloning with neural network repair in a two-step supervised learning framework. It first learns a policy from expert demonstrations and then applies repair subject to predictive models to enforce safety constraints. The predictive models can encompass various aspects relevant to robot learning applications, such as proprioceptive states and collision likelihood. Our experimental results demonstrate that the learned policy successfully adheres to a predefined set of safety constraints on two applications: mobile robot navigation, and real-world lower-leg prostheses. Additionally, we have shown that our method effectively reduces repeated interaction with the robot, leading to substantial time savings during the learning process.

In this paper, we introduce a new method that first utilizes 3D Gaussian splatting in street-level localization problem. Robust localization with street-level real-world images such as street view is a major issue for autonomous vehicle, augmented reality (AR) navigation, and outdoor mobile robots. The objective is to determine the position and orientation of a query image that matches a street view database composed of RGB images. However, given the limited information available in the street view images, accurately determining the location solely based on this data presents a significant challenge. To address this challenge, we propose a novel method called renderable street view map-based localization (RSM-Loc). This approach enhances the localization process by augmenting 2D street view images into a renderable 3D map using 3D Gaussian splatting, to resolve street-level localization problems. Upon receiving a query RGB image without geometry information, the proposed method renders 2D images from a pre-made renderable map and compares image pose similarities between the rendered images and the query image. Through iterations of this process, the proposed method eventually estimates the pose of the given query image. The experimental results demonstrate that RSM-Loc outperforms the baselines with neural-field-based localization. Additionally, we conduct deep analysis on the proposed method to show that our method can serve as a new concept for the street-level localization problem.

Autonomous robotic grasping under complex, clustered, and unstructured environments is a fundamental but challenging task. To achieve human-like rationality in dealing with the grasping task, the agent requires hybrid intelligence from multilateral aspects. This paper introduces RelationGrasp, a unified framework employing a transformer encoder-decoder structure to simultaneously achieve open-vocabulary object detection, manipulation relationship inference, and grasp pose detection. A unique object-oriented prompt learning mechanism is designed to seamlessly bridge the grasp pose and manipulation relationship branches, delivering high fidelity of object-grasp affiliation for object-aware grasping and grasp sequence planning. By formulating the relationship detection as an adjacency matrix regression task under multi-task learning, our framework significantly increases the relationship accuracy with reduced computational overhead. Moreover, to facilitate the robust and adaptive deployment of the proposed RelationGrasp to novel environments, we propose a consistency-based self-supervised adaptation strategy to adapt the pre-trained network to new scenarios and improve grasp accuracy on unseen objects. Our proposed network achieved state-of-the-art performance on various public dataset such as VMRD, OCID, etc., in both grasp detection and manipulation relationship classification, and real-world robot experiments has also been conducted to show the practical usages.

An excellent representation is crucial for reinforcement learning (RL) performance, especially in vision-based reinforcement learning tasks. The quality of the environment representation directly influences the achievement of the learning task. Previous vision-based RL typically uses explicit or implicit ways to represent environments, such as images, points, voxels, and neural radiance fields. However, these representations contain several drawbacks. They cannot either describe complex local geometries or generalize well to unseen scenes, or require precise foreground masks. Moreover, these implicit neural representations are akin to a "black box", significantly hindering interpretability. 3D Gaussian Splatting (3DGS), with its explicit scene representation and differentiable rendering nature, is considered a revolutionary change for reconstruction and representation methods. In this paper, we propose a novel Generalizable Gaussian Splatting framework to be the representation of RL tasks, called GSRL. Through validation in the RoboMimic environment, our method achieves better results than other baselines in multiple tasks, improving the performance by 10%, 44%, and 15% compared with baselines on the hardest task. This work is the first attempt to leverage generalizable 3DGS as a representation for RL.

Free-roaming dollies enhance filmmaking with dynamic movement, but challenges in automated camera control remain unresolved. Our study advances this field by applying Reinforcement Learning (RL) to automate dolly-in shots using free-roaming ground-based filming robots, overcoming traditional control hurdles. We demonstrate the effectiveness of combined control for precise film tasks by comparing it to independent control strategies. Our robust RL pipeline surpasses traditional Proportional-Derivative controller performance in simulation and proves its efficacy in real-world tests on a modified ROSBot 2.0 platform equipped with a camera turret. This validates our approach’s practicality and sets the stage for further research in complex filming scenarios, contributing significantly to the fusion of technology with cinematic creativity. This work presents a leap forward in the field and opens new avenues for research and development, effectively bridging the gap between technological advancement and creative filmmaking.

This paper presents an Active Search policy that balances between moving the camera and removing occluding objects to search for and retrieve a target object in clutter. While both types of action can reveal unobserved parts of a scene, they typically vary in execution complexity and time. Our proposed method explicitly reasons about the occluded spaces in the scene where the target object may be hidden, and uses reinforcement learning to compute the value of each action with the ultimate goal of finding and retrieving the target object in minimal time. Results in simulation and real-world experiments demonstrate a significant improvement in both task execution speed and success rate compared to baseline grasping strategies.

The precise and safe control of heavy material handling machines presents numerous challenges due to the hard-to-model hydraulically actuated joints and the need for collision-free trajectory planning with a free-swinging end-effector tool. In this work, we propose an RL-based controller that commands the cabin joint and the arm simultaneously. It is trained in a simulation combining data-driven modeling techniques with first-principles modeling. On the one hand, we employ a neural network model to capture the highly nonlinear dynamics of the upper carriage turn hydraulic motor, incorporating explicit pressure prediction to handle delays better. On the other hand, we model the arm as velocity-controllable and the free-swinging end-effector tool as a damped pendulum using first principles. This combined model enhances our simulation environment, enabling the training of RL controllers that can be directly transferred to the real machine. Designed to reach steady-state Cartesian targets, the RL controller learns to leverage the hydraulic dynamics to improve accuracy, maintain high speeds, and minimize end-effector tool oscillations. Our controller, tested on a mid-size prototype material handler, is more accurate than an inexperienced operator and causes fewer tool oscillations. It demonstrates competitive performance even compared to an experienced professional driver.

In this paper, we present a complete refractive Structure-from-Motion (RSfM) framework for underwater 3D reconstruction using refractive camera setups (for both, flat-and dome-port underwater housings). Despite notable achievements in refractive multi-view geometry over the past decade, a robust, complete and publicly available solution for such tasks is not available at present, and often practical applications have to resort to approximating refraction effects by the intrinsic (distortion) parameters of a pinhole camera model. To fill this gap, we have integrated refraction considerations throughout the entire SfM process within the state-of-the-art, open-source SfM framework COLMAP. Numerical simulations and reconstruction results on synthetically generated but photo-realistic images with ground truth validate that enabling refraction does not compromise accuracy or robustness as compared to in-air reconstructions. Finally, we demonstrate the capability of our approach for large-scale refractive scenarios using a dataset consisting of nearly 6000 images. The implementation is released as open-source at: https://cau-git.rz.uni-kiel.de/inf-ag-koeser/colmap_underwater.

Bronchoscopy reveals air passages and internal tissues for accurate diagnosis of various lung diseases. Robot-assisted bronchoscopy using an airway tree model can help path planning before surgery and navigation during surgery. In airway tree modeling, though volumetric deep learning methods have achieved good performance for airway segmentation, it remains a challenge due to the breakages and leakages. Some existing methods adopt post-processing using traditional methods like morphological and fuzzy connected algorithms. Also, some methods convert the volumetric data to point cloud format to refine segmentation. In this paper, we develop a new point cloud-based approach to refine volumetric segmentation. To address the breakage issue, we approach it as a regression problem of the branch extension direction and length. To tackle the leakage issue, we approach it as a segmentation task to eliminate leakages caused by breakage filling and from volumetric segmentation. Moreover, the direction information of branches is crucial for constructing the airway tree while point clouds do not naturally encode it. To introduce this information, we propose a directional feature aggregation, which first decomposes features of neighboring points based on their locations and aggregates decomposed features to aid the network in capturing the directional information effectively. Our proposed model has been evaluated on two public datasets, and the results show that our refinement can improve the volumetric segmentation.

Multi-robot systems have gained increasing interest across various fields such as medicine, environmental monitoring, and more. Despite the evident advantages, the coordination of the swarm arises significant challenges for human operators, particularly concerning the cognitive burden needed for efficiently controlling the robots. In this study, we present a novel approach for enabling a human operator to effectively control the motion of multiple robots. Leveraging a shared control data-driven approach, we enable a single user to control the 9 degrees of freedom related to the pose and shape of a swarm. Our methodology was evaluated through an experimental campaign conducted in simulated 3D environments featuring a narrow cylindrical path, which could represent, e.g., blood vessels, industrial pipes. Subjective measures of cognitive load were assessed using a post-experiment questionnaire, comparing different levels of autonomy of the system. Results show substantial reductions in operator cognitive load when compared to conventional teleoperation techniques, accompanied by enhancements in task performance, including reduced completion times and fewer instances of contact with obstacles. This research underscores the efficacy of our approach in enhancing human-robot interaction and improving operational efficiency in multi-robot systems.

We propose MR.HuBo (Motion Retargeting leveraging a HUman BOdy prior), a cost-effective and convenient method to collect high-quality upper body paired 〈robot, human〉 pose data, which is essential for data-driven motion retargeting methods. Unlike existing approaches which collect 〈robot, human〉 pose data by converting human MoCap poses into robot poses, our method goes in reverse. We first sample diverse random robot poses, and then convert them into human poses. However, since random robot poses can result in extreme and infeasible human poses, we propose an additional technique to sort out extreme poses by exploiting a human body prior trained from a large amount of human pose data. Our data collection method can be used for any humanoid robots, if one designs or optimizes the system’s hyperparameters which include a size scale factor and the joint angle ranges for sampling. In addition to this data collection method, we also present a two-stage motion retargeting neural network that can be trained via supervised learning on a large amount of paired data. Compared to other learning-based methods trained via unsupervised learning, we found that our deep neural network trained with ample high-quality paired data achieved notable performance. Our experiments also show that our data filtering method yields better retargeting results than training the model with raw and noisy data. Our code and video results are available on https://sites.google.com/view/mr-hubo/.

Non-rigid point cloud registration remains a significant challenge in 3D computer vision due to the complexity of structural deforms, lack of overlaps, and sensitivity to initialization. This paper introduces a framework inspired by the recent success in recurrent architecture, adapted to accommodate the unique characteristics of point clouds. More specifically, we design a recurrent update network block for progressively refining local registration results under a local rigidity assumption, starting from an initial global SE(3) alignment. Through comparison, our method consistently outperforms competing methods in standard metrics, achieving a 33% reduction in EPE on the 4DLoMatch benchmark compared to the second-best method. To the best of our knowledge, the proposed method is the first to successfully demonstrate that the recurrent update strategy can effectively address the non-rigid registration task with large displacement, significant deform, and low overlap. The source code and the model will be released at http://dummy.url/.

Recognizing failures during task execution and implementing recovery procedures is challenging in robotics. Traditional approaches rely on the availability of extensive data or a tight set of constraints, while more recent approaches leverage large language models (LLMs) to verify task steps and replan accordingly. However, these methods often operate offline, necessitating scene resets and incurring in high costs. This paper introduces Recover, a neuro-symbolic framework for online failure identification and recovery. By integrating ontologies, logical rules, and LLM-based planners, Recover exploits symbolic information to enhance the ability of LLMs to generate recovery plans and also to decrease the associated costs. In order to demonstrate the capabilities of our method in a simulated kitchen environment, we introduce OntoThor, an ontology describing the AI2Thor simulator setting. Empirical evaluation shows that OntoThor’s logical rules accurately detect all failures in the analyzed tasks, and that Recover considerably outperforms, for both failure detection and recovery, a baseline method reliant solely on LLMs. Supplementary material, including the OntoThor ontology, is available at: https://recover-ontothor.github.io.