IROS 2024 Papers

Unmanned aerial vehicles (UAVs) for contact-based tasks at height can greatly improve the safety of the human workers involved. However, performing contact-based tasks with typical under-actuated UAVs is non-trivial. Due to their coupled translational and rotational dynamics and their limited station-keeping performance under physical disturbances, it is difficult to maintain precise and consistent contact. We address these problems in the context of physical interaction with vertical, cylindrical target objects, such as trees. We present a novel UAV design with a pair of tilt-rotors and a landing gear that can reconfigure into a front-mounted, two-fingered gripper. While the tilt-rotors provide horizontal force toward the target object without pitching the UAV forward, the reconfigurable landing gear enables the UAV to obtain support from the target object. Such support results in an approximately 80% improvement in position- and heading-keeping performance. Moreover, the landing gear is designed as a cable-driven under-actuated system, which requires only one actuator to control both the reconfiguration and the grasping (i.e., five degrees of freedom in total). Such a minimalist design helps keep the UAV power consumption for interactions low. This marks progress towards safe, high-precision physical interaction against vertical, cylindrical target objects. Our UAV in action: https://youtu.be/D-65vldox_A.

We introduce Reality Fusion, a novel robot teleoperation system that localizes, streams, projects, and merges a typical onboard depth sensor with a photorealistic, high resolution, high framerate, and wide FoV rendering of the complex remote environment represented as 3D Gaussian splats (3DGS). Our framework enables robust egocentric and exocentric robot teleoperation in immersive VR, with the 3DGS effectively extending spatial information of a depth sensor with limited FoV and balancing the trade-off between data streaming costs and data visual quality. We evaluated our framework through a user study with 24 participants, which revealed that Reality Fusion leads to significantly better user performance, situation awareness, and user preferences. To support further research and development, we provide an open-source implementation with an easy-to-replicate custom-made telepresence robot, a high-performance virtual reality 3DGS renderer, and an immersive robot control package.1

Data augmentation methods to enhance perception performance in adverse weather have recently attracted considerable attention. Most of the LiDAR data augmentation methods post-process the existing dataset by physics-based models or machine-learning methods. However, due to the limited environmental annotations and the fixed vehicle trajectories in existing datasets, it is challenging to edit the scene and expand the diversity of traffic flow and scenario. To this end, we propose a simulator-based physical modeling approach to augment LiDAR data in rainy weather, enhancing the performance of the perception model. We complete the modeling task of the rainy weather effect in the CARLA simulator and establish a data collection pipeline for LiDAR. Furthermore, we pay special attention to the spray generated by vehicles in rainy weather and simulate this phenomenon through the Spray Emitter method we developed. In addition, considering the influence of different weather conditions on point cloud intensity, we develop a prediction network to forecast the intensity of the LiDAR echo. This enables us to complete the rainy weather simulation of 4D point cloud data. In the experiment, we observe that the model augmented by our synthetic dataset improves the performance for 3D object detection in rainy weather. Both code and dataset are available at https://github.com/PJLab-ADG/PCSim#rainypcsim.

In the context of unstructured and unknown environment, the autonomous navigation still faces many challenges, such as assessing rough terrain and deciding how to safely navigate complex terrain. In this work, we propose a robust and practical off-road navigation framework that has been successfully deployed on a vibroseis truck for land exploration. First, in degraded wild scenes, a tightly coupled lidar-GNSS-inertial fusion odometry and mapping framework is adopted to construct a local point cloud map around the vehicle in real-time and provide precise localization. Then, based on amplitude-frequency characteristic analysis and point cloud PCA, a multi-layer terrain assessment map containing terrain roughness, obstacles and slope information is obtained. Finally, combining Gaussian distribution based adaptive sampler and Bayesian sequentially updated proposal distribution, a local graph is efficiently built to obtain multiple path solutions under constrained conditions. Both simulations and field experiments show that the proposed navigation framework can decide how to travel on a flat road even in harsh terrain conditions, naturally suppressing frequent attitude angle changes and preventing vehicle accidents.

This paper presents an innovative approach for executing robotic needle insertion within deformable living organs. The objective is to maintain the insertion pivot point on the skin, which remains stationary. At the same time, the organs undergo displacement and deformation due to respiration. Therefore, real-time control and precise needle steering are crucial. The proposed method relies on isolated objective constraints to ensure the objectives while steering the needle along a predefined trajectory. The needle insertion process benefits from Finite Element (FE) models to simulate the environment and address the inverse problem to drive the robot’s end effector (EE) by re-evaluating the objective functions in the constraint space for each time step. So, the desired motion of the robot’s EE could be calculated at a small cost for non-linear functions in real-time, resulting in better precision and reducing stress caused to the organs.

To achieve safe legged locomotion, it is crucial to generate motion in real-time considering various constraints in robots and environments. In this study, we propose a lightweight real-time perceptive motion control system for the newly developed six-wheeled-telescopic-legged robot, Tachyon 3. In the proposed method, analytically smoothed constraints including Smooth Separating Axis Theorem (SSAT) as a novel higher order differentiable collision detection for 3D shapes is applied to the Control Barrier Function (CBF). The proposed system integrating the CBF achieves online motion generation in a short control cycle of 1 ms that satisfies joint limitations, environmental collision avoidance and safe convex foothold constraints. The efficiency of SSAT is shown from the collision detection time of 1 µs or less and the CBF constraint computation time for Tachyon 3 of several µs. Furthermore, the effectiveness of the proposed system is verified through the stair-climbing motion, integrating online recognition in a simulation and a real machine.

The growing interest in connected autonomous vehicles (CAVs) has intensified the focus on technologies and algorithms that enhance behavior, comfort, and safety. Among these, the concept of Digital Twins (DT) represents an emerging field of research that is now beginning to be applied to autonomous systems. Traditional Advanced Driver-Assistance Systems (ADAS) can prevent real-time collisions using sensor data. However, we propose that employing a DT can enable the accounting for complex, simulated decisions before they occur in reality. This paper introduces an initial model of a Digital Twin, founded on an internal simulator aligned with vehicle control architecture, for real-time hazard prediction and effective decision-making. Our DT synchronizes with the vehicle’s state to simulate various hazardous scenarios in advance, allowing for preemptive actions. To support our hypothesis, we introduce an algorithm for the early detection of potential collisions between CAVs and pedestrians through the unsupervised simulation of diverse traffic scenarios. This solution integrates the CORTEX cognitive architecture with CARLA for internal simulation, leveraging probabilistic models to select optimal scenarios. Employing data from external pedestrian cameras, a particle filter predicts the most probable pedestrian trajectories via DT simulations, thereby informing safe maneuvers. Although the algorithm itself is established, the novelty of our approach lies in incorporating a simulator within the digital twin. This simulator, informed by real-time data on the vehicle’s and environment’s state, facilitates appropriate responses to unpredictable behaviors. We have conducted extensive tests with an actual autonomous electric vehicle on a university campus to validate the system’s predictive and adaptive functions.

Dexterous telemanipulation is crucial in advancing human-robot systems, especially in tasks requiring precise and safe manipulation. However, it faces significant challenges due to the physical differences between human and robotic hands, the dynamic interaction with objects, and the indirect control and perception of the remote environment. Current approaches predominantly focus on mapping the human hand onto robotic counterparts to replicate motions, which exhibits a critical oversight: it often neglects the physical interaction with objects and relegates the interaction burden to the human to adapt and make laborious adjustments in response to the indirect and counter-intuitive observation of the remote environment. This work develops an End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD) framework to address telemanipulation tasks. EFOLD models telemanipulation as a Markov Game, introducing multiple end-effect features to interpret the human operator’s commands during interaction with objects. These features are used by a Deep Reinforcement Learning policy to control the robot and reproduce such end effects. EFOLD was evaluated with real human subjects and two end-effect extraction methods for controlling a virtual Shadow Robot Hand in telemanipulation tasks. EFOLD achieved real-time control capability with low command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084 rad).

This paper presents an algorithm that finds a centroidal motion and footstep plan for a Spring-Loaded Inverted Pendulum (SLIP)-like bipedal robot model substantially faster than real-time. This is achieved with a novel representation of the dynamic footstep planning problem, where each point in the environment is considered a potential foothold that can apply a force to the center of mass to keep it on a desired trajectory. For a biped, up to two such footholds per time step must be selected, and we approximate this cardinality constraint with an iteratively reweighted l1-norm minimization. Along with a linearizing approximation of an angular momentum constraint, this results in a quadratic program can be solved for a contact schedule and center of mass trajectory with automatic gait discovery. A 2 s planning horizon with 13 time steps and 20 surfaces available at each time is solved in 142 ms, roughly ten times faster than comparable existing methods in the literature. We demonstrate the versatility of this program in a variety of simulated environments.

Robots that autonomously operate in human living environments require the ability to adapt to unpredictable changes and flexibly handle a variety of tasks. Particularly, coordinated bimanual motions are essential for enabling tasks that are difficult with just one hand, such as grasping bulky objects, transporting heavy loads, and precision work. Traditional methods of generating robot motions typically involve executing pre-programmed motions, making it challenging to adapt to complex and unpredictable environmental changes. To address this issue, our research focuses on generating diverse motions that can flexibly adapt to environmental changes based on Deep Predictive Learning from a small amount of real-world data. Previous Deep Predictive Learning models have generated the motions of a robot’s left and right arms by a single LSTM, making it difficult to operate them independently. Therefore, we propose a new Hierarchical Deep Predictive Learning model specialized for generating coordinated bimanual motions. This model comprises three components: a Left-LSTM, which learns the body and visual information on the robot’s left side, a Right-LSTM that performs a similar function for the right side, and a Union-LSTM which integrates this information at a higher level. To verify the effectiveness of the proposed model, we conducted bimanual grasping experiments with multiple different objects using two different robots. The experimental results showed that independent of hardware, our model demonstrated a higher success rate compared to the traditional approach, indicating its enhanced capability in coordinating bimanual motions.

Bird’s-Eye-View (BEV) segmentation is a essential technology for safe and efficient navigation. This is even more necessary in indoor driving, where there are dynamic and unstructured objects such as people and robot. However, since there is no way to generate training data, most of the researches have been conducted mainly in outdoor environments. In this paper, we propose an innovative approach to address this challenge. We automatically generate BEV training data for indoor environments based on the physics engine of a simulator. This eliminates the needs for tons of real data and virtual environments. We also propose a lightweight network architecture capable of BEV panoptic segmentation in real-time. Based on this network and simple image processing, our framework shows fast but also robust performance in real-world environments with no gap between simulation and reality. The network can inference at a speed of about 61 FPS on AGX Orin in FP16 mode. Furthermore, it outperforms existing algorithms in the semantic segmentation task, achieving 14% higher mIoU.

Robot swarms are increasingly being applied in various domains. However, due to the inherent limitation imposed by low real-time communication bandwidth, the synchronization of environmental information among multiple robots remains a persistent and challenging problem in practical applications. In response to this challenge, we introduce a comprehensive framework for synchronizing occupancy grid maps (OGMs) in practical multi-robot systems that operate under communication bandwidth constraints. In our research, we elaborately design the data structure of transmitted local OGMs and employ the Hilbert space-filling curve for voxel sorting. By adopting this approach, data redundancy is effectively increased, resulting in lower information entropy for compression and significantly reducing the volume of communication data. Finally, our framework outperforms the benchmark method by reducing the average and maximum bandwidth usage by more than 10 times in high-resolution scenarios. Moreover, our method has been successfully applied in the multi-UAV autonomous navigation application, demonstrating its real-time and bandwidth-efficient nature, as well as its practical value.

Over the past decade, lidars have become a cornerstone of robotics state estimation and perception thanks to their ability to provide accurate geometric information about their surroundings in the form of 3D scans. Unfortunately, most of nowadays lidars do not take snapshots of the environment but sweep the environment over a period of time (typically around 100 ms). Such a rolling-shutter-like mechanism introduces motion distortion into the collected lidar scan, thus hindering downstream perception applications. In this paper, we present a novel method for motion distortion correction of lidar data by tightly coupling lidar with Inertial Measurement Unit (IMU) data. The motivation of this work is a map-free dynamic object detection based on lidar. The proposed lidar data undistortion method relies on continuous preintegrated of IMU measurements that allow parameterising the sensors’ continuous 6-DoF trajectory using solely eleven discrete state variables (biases, initial velocity, and gravity direction). The undistortion consists of feature-based distance minimisation of point-to-line and point-to-plane residuals in a non-linear least-square formulation. Given undistorted geometric data over a short temporal window, the proposed pipeline computes the spatiotemporal normal vector of each of the lidar points. The temporal component of the normals is a proxy for the corresponding point’s velocity, therefore allowing for learning-free dynamic object classification without the need for registration in a global reference frame. We demonstrate the soundness of the proposed method and its different components using public datasets and compare them with state-of-the-art lidar-inertial state estimation and dynamic object detection algorithms.

Semantic segmentation plays a pivotal role in many robotic applications requiring high-level scene understanding, such as smart farming, where the precise identification of trees or plants can aid navigation and crop monitoring tasks. While deep-learning-based semantic segmentation approaches have reached outstanding performance in recent years, they demand large amounts of labeled data for training. Inspired by modern Unsupervised Domain Adaptation (UDA) techniques, in this paper, we introduce a two-step training pipeline specifically tailored to challenging natural scenes, where the availability of annotated data is often quite limited. Our strategy involves the initial training of a powerful domain adaptive architecture, followed by a refinement stage, where segmentation masks predicted by the Segment Anything Model (SAM) are used to improve the accuracy of the predictions on the target dataset. These refined predictions serve as pseudo-labels to supervise the training of a final distilled architecture for real-time deployment. Extensive experiments conducted in two real-world scenes demonstrate the effectiveness of the proposed method. Specifically, we show that our pipeline enables the training of a MobileNetV3 that achieves significant mIoU gains of 3.60% and 11.40% on our two datasets compared to the DAFormer while only demanding 1/15 of the latter’s inference time. Code and datasets are available at https://github.com/VIS4ROB-lab/nature_uda_rt_segmentation.

Curb following is a key technology for autonomous road sweeping vehicles. Currently, existing implementations primarily involve pre-recording waypoints during human driving and subsequently retracing them autonomously. Moreover, existing research related to this topic predominately focuses on curb detection for driver assistance, yet the resultant curb detection outcomes remain underutilized in the development of autonomous curb following systems. To fill this gap, this paper proposes a real-time path generation and alignment control approach to facilitate autonomous curb following. Firstly, a segmented path generation algorithm is introduced that progressively generates reference path segments while ensuring the overall continuity of the reference path. Secondly, a parameterized alignment control algorithm is developed to accurately navigate the vehicle along the planned reference path with proved stability. Real public road experiments have been conducted to validate the proposed approach. The experimental results demonstrate the efficacy of the proposed methodologies across various curb following scenarios, including common concave, convex, and straight-concave curbs, thereby showcasing the practical viability of our methods in real-world applications.

The expanding use of automated vision, assistance systems, and augmented reality applications in marine settings calls for reliable and accurate horizon detection and locking. Traditional methods utilizing Inertial Measurement Units (IMU) or feature-based computer vision techniques often yield inconsistent results, particularly when unmanned surface vehicles or boats are subject to high-speed movement or choppy waters. Addressing this, our work introduces a computer vision (CV)-based solution for real-time horizon locking. Employing real-time semantic segmentation, we accurately differentiate between sky, land or water in the frame, enabling computational locking of the horizon’s position. This stable visual reference significantly improves the performance and reliability of on-board systems for autonomous navigation, augmented reality overlays, and multi-object tracking. Supported by a dataset collected under various marine conditions, our method has proven to achieve high accuracy with low computational latency, making it a promising avenue for wide-scale implementation on automated and semi-automated systems.

Robots interacting with humans must be safe, reactive and adapt online to unforeseen environmental and task changes. Achieving these requirements concurrently is a challenge as interactive planners lack formal safety guarantees, while safe motion planners lack flexibility to adapt. To tackle this, we propose a modular control architecture that generates both safe and reactive motion plans for human-robot interaction by integrating temporal logic-based discrete task level plans with continuous Dynamical System (DS)-based motion plans. We formulate a reactive temporal logic formula that enables users to define task specifications through structured language, and propose a planning algorithm at the task level that generates a sequence of desired robot behaviors while being adaptive to environmental changes. At the motion level, we incorporate control Lyapunov functions and control barrier functions to compute stable and safe continuous motion plans for two types of robot behaviors: (i) complex, possibly periodic motions given by autonomous DS and (ii) time-critical tasks specified by Signal Temporal Logic (STL). Our methodology is demonstrated on the Franka robot arm performing wiping tasks on a whiteboard and a mannequin that is compliant to human interactions and adaptive to environmental changes.

How do people use their faces and bodies to test the interactive abilities of a robot? Making lively, believable agents is often seen as a goal for robots and virtual agents but believability can easily break down. In this Wizard-of-Oz (WoZ) study, we observed 1169 nonverbal interactions between 20 participants and 6 types of agents. We collected the nonverbal behaviors participants used to challenge the characters physically, emotionally, and socially. The participants interacted freely with humanoid and non-humanoid forms: a robot, a human, a penguin, a pufferfish, a banana, and a toilet. We present a human behavior codebook of 188 unique nonverbal behaviors used by humans to test the virtual characters. The insights and design strategies drawn from video observations aim to help build more interaction-aware and believable robots, especially when humans push them to their limits.

We propose a new orientation-aware scene descriptor ReLoc-Aligner for re-localization of a 3D point cloud. Re-localization within a 3D point cloud map is crucial for conducting Simultaneous Localization and Mapping (SLAM). Existing re-localization or place recognition methods of 3D LiDAR sensor data aim to estimate the current position of the sensor robustly to orientation changes. However, they do not determine the current orientation of the sensor within a 3D point cloud map, which limits their applications to re-localization or loop closing in SLAM. On the other hand, existing methods capable of orientation estimation tend to be slower than them. Our scene descriptor has a property of orientation awareness that enables us to extract the orientation difference between two scans directly from the descriptor. This is useful for the registration of point clouds from a good initial estimate, which leads to better re-localization of a scan. We propose a training method for the new descriptor. In addition, we develop fast querying and re-localization methods using the descriptors. Intensive experiments demonstrate that the proposed method is superior to the existing state-of-the-art methods in both place recognition and orientation estimation.