IROS 2024 Papers

Despite the elegantly simple operation principle, the design of everting vine robots still presents two complexities: a tip-device required for buckle-free retraction; a pressurised base chamber to maintain inflation while allowing the everted body to be winded onto a motorised reel. We create a new type of vine robots with a unique eversion method: instead of using a single everting tube, our design comprises multiple interconnected tubes that bend to induce an overall eversion. Our design eliminates the pressurised base and the tip-device, while allowing buckle-free retraction. It also enables new designs and functionalities beyond conventional vine robots, including drone-based vine robots capable of cantilevered operations, everting grippers that grasp then transport objects into the robot body, and solid-state vine robots without the need for inflation.

Training robots to perform complex control tasks from high-dimensional pixel input using reinforcement learning (RL) is sample-inefficient, because image observations are comprised primarily of task-irrelevant information. By contrast, humans are able to visually attend to task-relevant objects and areas. Based on this insight, we introduce Visual Saliency-Guided Reinforcement Learning (ViSaRL). Using ViSaRL to learn visual representations significantly improves the success rate, sample efficiency, and generalization of an RL agent on diverse tasks including DeepMind Control benchmark, robot manipulation in simulation and on a real robot. We present approaches for incorporating saliency into both CNN and Transformer-based encoders. We show that visual representations learned using ViSaRL are robust to various sources of visual perturbations including perceptual noise and scene variations. ViSaRL nearly doubles success rate on the real-robot tasks compared to the baseline which does not use saliency.

X-ray to CT registration is an essential technique to provide on-site guidance for clinicians and medical robots by aligning preoperative information with intraoperative images. Current methods focus on local registration with small capture ranges and necessitate a manual initial alignment before precise registration. Some existing global methods are likely to fail in thoracic surgeries because of the respiratory motion and the nearly colinear nature of vertebrae landmarks. In this study, we propose a vertebrae-based global X-ray to CT registration method with the assistance of clinical setups for thoracic surgeries. Firstly, vertebrae centroids are automatically localized by CNN-based networks in CT and X-ray for establishing 2D/3-D correspondences. Then, inspired by clinical setup, we address the degradation of colinear landmarks of 6-DoF pose estimation by introducing a 4-DoF solver. Considering the inaccurate priori and landmark mislocalization, the solver is embedded into the Adaptive Error-Aware Estimator (AE2) to simultaneously estimate weights and aggregate candidate poses. Finally, the whole method is trained in an end-to-end manner for better performance. Evaluations on both the public LIDC-IDRI dataset and clinical dataset demonstrate that our method outperforms existing optimization-based and learningbased approaches in terms of registration accuracy and success rate. Our code: https://github.com/LiuLiluZJU/2P-AE2

To address the setup and changeover time issues in high-mix, low-volume production systems, we developed an end-effector capable of uniformly scooping, holding, and transporting a wide variety of objects, and demonstrated this system with prototype. Our experiments showed that the prototype was successful in scooping up and transporting a wide variety of objects and could be applied to high-mix low-volume production systems. In addition, the load testing of the spatula and modeling of objects that can be tilted backwards provided insight into further improvements of the scooping performance of this mechanism.

Hexapod robots are potentially suitable for carrying out tasks in cluttered environments since they are stable, compact, and light weight. They also have multi-joint legs and variable height bodies that make them good candidates for tasks such as stairs climbing and squeezing under objects in a typical home environment or an attic. Expanding on our previous work on joist climbing in attics, we train a legged hexapod equipped with a depth camera and visual inertial odometry (VIO) to perform three tasks: climbing stairs, avoiding obstacles, and squeezing under obstacles such as a table. Our policies are trained with simulation data only and can be deployed on low-cost hardware not requiring real-time joint state feedback. We train our model in a teacher-student model with 2 phases: In phase 1, we use reinforcement learning with access to privileged information such as height maps and joint feedback. In phase 2, we use supervised learning to distill the model into one with access to only onboard observations, consisting of egocentric depth images and robot pose captured by a tracking VIO camera. By manipulating available privileged information, constructing simulation terrains, and refining reward functions during phase 1 training, we are able to train the robots with skills that are robust in non-ideal physical environments. We demonstrate successful sim-to-real transfer and achieve high success rates across all three tasks in physical experiments.

Trajectory prediction plays a crucial role in autonomous driving, but it is challenging due to the multi-modal nature of future trajectories. Behavior information is frequently employed to capture more diverse modalities of future trajectories. Traditional behavior information is typically hard-encoded, which is often inaccurate and inadequate for reflecting future multimodality. Therefore, we introduce the concept of soft vehicle behavior, which is represented as a probability distribution over a predefined comprehensive set of behaviors. This approach allows for a more rational depiction of vehicle behavior and captures potential future driving modalities. Based on it, we propose a new soft-behavior-constrained vehicle trajectory prediction framework. The framework consists of a backbone and a lightweight and plug-and-play behavior prediction module, which is used to imbue soft behavior constraints to assist in representation learning. We integrated the behavior prediction module into five representative trajectory predictors and achieved improvements of at least 4.2% in minFDE(K=5) on the nuScenes dataset and 0.5% in minFDE(K=6) on the Argoverse 1 motion forecasting dataset. These universal increments prove the effectiveness and generalizability of soft behavior constraints in vehicle trajectory prediction.

In this paper we propose a new algorithm for robots searching a hazardous, communications-denied area to gather information using a robot fleet that has a limited number of agents. The centralized algorithm uses robot survival along search paths as a sensor event for a distributed sensor network. As agents are lost to hazards, the search behavior adjusts to prioritize agent longevity in order to maximize information gain. In the past, related work solving this problem has assumed an infinite number of agents. In contrast, we assume that the number of agents is finite. We use Bayesian inference to update target and hazard belief maps of an area using data from the probability of survival of prior agents’ paths as well as sensor readings from the agents along those paths. Using those belief maps, the algorithm can construct paths that maximize information gain, in expectation, while taking into account the predicted decrease in future information collected when losing an agent. This behavior increases the likelihood that agents survive longer, allowing them to collect more data.Using simulations with various fleet sizes and probabilities for hazards disabling agents, we compare our algorithm to work that does not account for attrition. The results show an increase in the longevity of the fleet when hazards are more effective at disabling agents. In nearly all cases, this contributes to an increased rate in information gain when the fleet size is small. Small sized fleets, in our case 10 or less agents, do not meet a threshold of collected information necessary to direct agents away from hazards. Large fleets, over 200 agents in our scenario, collect most of the information before Our algorithm causes a noticeable change in agent behavior (as compared to existing techniques). We find that the proposed method provides the greatest advantage for mid-sized fleets, between 20 and 100 agents, and when hazards have an increased probability of immobilizing agents. Show More

As a part of the perception results of intelligent driving systems, static object detection (SOD) in 3D space provides crucial cues for driving environment understanding. With the rapid deployment of deep neural networks for SOD tasks, the demand for high-quality training samples soars. The traditional, also reliable, way is manual labelling over the dense LiDAR point clouds and reference images. Though most public driving datasets adopt this strategy to provide SOD ground truth (GT), it is still expensive and time-consuming in practice. This paper introduces VRSO, a visual-centric approach for static object annotation. Experiments on the Waymo Open Dataset show that the mean reprojection error from VRSO annotation is only 2.6 pixels, around four times lower than the Waymo Open Dataset labels (10.6 pixels). VRSO is distinguished in low cost, high efficiency, and high quality: (1) It recovers static objects in 3D space with only camera images as input, and (2) manual annotation is barely involved since GT for SOD tasks is generated based on an automatic reconstruction and annotation pipeline.

Autonomous exploration plays a crucial role in robotics applications like rescue and scene reconstruction. This work addresses the challenges of autonomous exploration in intricate unknown environments by presenting a novel UAV autonomous exploration method based on a new concept of the view-region. Our proposed approach leverages the view-region to replace the conventional viewpoint generation and selection process, streamlining the planning process for exploration. Simultaneously, we model the problem of maximizing frontier coverage within the field of view during exploration, and jointly optimize it with the exploration path optimization problem. This approach ensures exploration path safety and effectiveness while being aggressive. Additionally, a gimbal is incorporated beneath the camera, with an associated optimization problem designed to minimize UAV self-rotation and enhance exploration efficiency. Simulations and real-world experiments demonstrate that the proposed method outperforms existing state-of-the-art methods in terms of runtime and distance traveled.

We present VLPG-Nav, a visual language navigation method for guiding robots to specified objects within household scenes. Unlike existing methods primarily focused on navigating the robot toward objects, our approach considers the additional challenge of centering the object within the robot’s camera view. Our method builds a visual language pose graph (VLPG) that functions as a spatial map of VL embeddings. Given an open-vocabulary object query, we plan a viewpoint for object navigation using the VLPG. Despite navigating to the viewpoint, real-world challenges such as object occlusion, displacement, and the robot’s localization errors can prevent visibility. We build an object localization probability map that leverages the robot’s current observations and prior VLPG. When the object is not visible, the probability map is updated, and an alternate viewpoint is computed. In addition, we propose an object-centering formulation that locally adjusts the robot’s pose to center the object in the camera view. We evaluate the effectiveness of our approach through simulations and real-world experiments, evaluating its ability to successfully view and center the object within the camera’s field of view. VLPG-Nav demonstrates improved performance in locating the object, navigating around occlusions, and centering the object within the robot’s camera view, outperforming selected baselines in the evaluation settings.

State estimation is a central component of autonomous navigation. To date, many methods presented have a disruptive potential for application, such as visual-inertial odometry (VIO), wheel and leg odometry (for short, body odometry). However, most of them are prone to fail in some challenging conditions like high-dynamic street scenes and sustain aggressive movements. To this end, in this paper, we present a novel visual-inertial-velocity odometry (VIVO) framework which incorporates velocity measurement provided by the proprioceptive sensing into the MSCKF-based VIO in a tightly coupled fashion. Furthermore, considering that the imprecise extrinsic parameters can severely undermine the state estimation performance, we hence perform VIVO along with online calibration of the body odometry’s extrinsic parameters by adding them to the estimated state vector. The generic VIVO can be deployed for a broad spectrum of robot models ranging from wheeled robots to legged robots. Both simulation and real-world experiments are performed to extensively validate the robustness and accuracy of the proposed method in challenging scenarios using wheeled and legged robot models, respectively.

Autonomous mobile robots are an increasingly integral part of modern factory and warehouse operations. Obstacle detection, avoidance and path planning are critical safety-relevant tasks, which are often solved using expensive LiDAR sensors and depth cameras. We propose to use cost-effective low-resolution ranging sensors, such as ultrasonic and infrared time-of-flight sensors by developing VIRUS-NeRF - Vision, InfraRed, and UltraSonic based Neural Radiance Fields.Building upon Instant Neural Graphics Primitives with a Multiresolution Hash Encoding (Instant-NGP), VIRUS-NeRF incorporates depth measurements from ultrasonic and infrared sensors and utilizes them to update the occupancy grid used for ray marching. Experimental evaluation in 2D demonstrates that VIRUS-NeRF achieves comparable mapping performance to LiDAR point clouds regarding coverage. Notably, in small environments, its accuracy aligns with that of LiDAR measurements, while in larger ones, it is bounded by the utilized ultrasonic sensors. An in-depth ablation study reveals that adding ultrasonic and infrared sensors is highly effective when dealing with sparse data and low view variation. Further, the proposed occupancy grid of VIRUS-NeRF improves the mapping capabilities and increases the training speed by 46% compared to Instant-NGP. Overall, VIRUS-NeRF presents a promising approach for cost-effective local mapping in mobile robotics, with potential applications in safety and navigation tasks. The code can be found at https://github.com/ethz-asl/virus_nerf.

In this work, we introduce the Virtual In-Hand Eye Transformer (VIHE), a novel method designed to enhance 3D manipulation capabilities through action-aware view rendering. VIHE autoregressively refines actions in multiple stages by conditioning on rendered views posed from action predictions in the earlier stages. These virtual in-hand views provide a strong inductive bias for effectively recognizing the correct pose for the hand, especially for challenging high-precision tasks such as peg insertion. On 18 manipulation tasks in RLBench simulated environments, VIHE achieves a new state-of-the-art, with a 12% absolute improvement, increasing from 65% to 77% over the existing state-of-the-art model using 100 demonstrations per task. In real-world scenarios, VIHE can learn manipulation tasks with just a handful of demonstrations, highlighting its practical utility. Videos and code implementation can be found at our project site: https://vihe-3d.github.io.

Humans excel at efficiently navigating through crowds without collision by focusing on specific visual regions relevant to navigation. However, most robotic visual navigation methods rely on deep learning models pre-trained on vision tasks, which prioritize salient objects—not necessarily relevant to navigation and potentially misleading. Alternative approaches train specialized navigation models from scratch, requiring significant computation. On the other hand, self-supervised learning has revolutionized computer vision and natural language processing, but its application to robotic navigation remains underexplored due to the difficulty of defining effective self-supervision signals. Motivated by these observations, in this work, we propose a Self-Supervised Vision-Action Model for Visual Navigation Pre-Training (VANP). Instead of detecting salient objects that are beneficial for tasks such as classification or detection, VANP learns to focus only on specific visual regions that are relevant to the navigation task. To achieve this, VANP uses a history of visual observations, future actions, and a goal image for self-supervision, and embeds them using two small Transformer Encoders. Then, VANP maximizes the information between the embeddings by using a mutual information maximization objective function. We demonstrate that most VANP-extracted features match with human navigation intuition. VANP achieves comparable performance as models learned end-to-end with half the training time and models trained on a large-scale, fully supervised dataset, i.e., ImageNet, with only 0.08% data. 1

Simultaneous localization and mapping (SLAM) in highly dynamic environments is challenging due to the correlation complexity between moving objects and the camera pose. Many methods have been proposed to deal with this problem; however, the moving properties of dynamic objects with a moving camera remain unclear. Therefore, to improve SLAM’s performance, minimizing disruptive events of moving objects with a physical understanding of 3D shapes and dynamics of objects is needed. In this paper, we propose a robust method, V3D-SLAM, to remove moving objects via two lightweight reevaluation stages, including identifying potentially moving and static objects using a spatial-reasoned Hough voting mechanism and refining static objects by detecting dynamic noise caused by intra-object motions using Chamfer distances as similarity measurements. Through our experiment on the TUM RGB-D benchmark on dynamic sequences with ground-truth camera trajectories, the results show that our methods outperform most other recent state-of-the-art SLAM methods. Our source code is available at https://github.com/tuantdang/v3d-slam.

Cooperative LiDAR systems integrating vehicles and road infrastructure, termed V2I calibration, exhibit substantial potential, yet their deployment encounters numerous challenges. A pivotal aspect of ensuring data accuracy and consistency across such systems involves the calibration of LiDAR units across heterogeneous vehicular and infrastructural endpoints. This necessitates the development of calibration methods that are both real-time and robust, particularly those that can ensure robust performance in urban canyon scenarios without relying on initial positioning values. Accordingly, this paper introduces a novel approach to V2I calibration, leveraging spatial association information among perceived objects. Central to this method is the innovative Overall Intersection over Union (oIoU) metric, which quantifies the correlation between targets identified by vehicle and infrastructure systems, thereby facilitating the real-time monitoring of calibration results. Our approach involves identifying common targets within the perception results of vehicle and infrastructure LiDAR systems through the construction of an affinity matrix. These common targets then form the basis for the calculation and optimization of extrinsic parameters. Comparative and ablation studies conducted using the DAIR-V2X dataset substantiate the superiority of our approach. For further insights and resources, our project repository is accessible at https://github.com/MassimoQu/v2i-calib.

The ability to construct concise scene representations from sensor input is central to the field of robotics. This paper addresses the problem of robustly creating a 3D representation of a tabletop scene from a segmented RGBD image. These representations are then critical for a range of downstream manipulation tasks. Many previous attempts to tackle this problem do not capture accurate uncertainty, which is required to subsequently produce safe motion plans. In this paper, we cast the representation of 3D tabletop scenes as a multi-class classification problem. To tackle this, we introduce V-PRISM, a framework and method for robustly creating probabilistic 3D segmentation maps of tabletop scenes. Our maps contain both occupancy estimates, segmentation information, and principled uncertainty measures. We evaluate the robustness of our method in (1) procedurally generated scenes using open-source object datasets, and (2) real-world tabletop data collected from a depth camera. Our experiments show that our approach outperforms alternative continuous reconstruction approaches that do not explicitly reason about objects in a multi-class formulation.

Research has indicated that exoskeletons can assist human movement, but due to the influence of additional weight and challenges in control strategy design, only a few exoskeletons effectively reduce the wearers’ metabolic costs during running. This paper proposes an innovative and efficient hip-assisted running exoskeleton (HARE) designed to facilitate the flexion and extension movements of the joint along the sagittal plane. In the field of structural engineering, we propose implementing an active-passive combination constant force suspension system, hereinafter referred to as CFS, to effectively mitigate the impact of inertial forces during running. The decoupled transmission mechanism allows the CFS and assist mechanisms to operate independently, ensuring the tension of the cables. The flexible structural design can reduce the locomotion limitation on human bodies and reduce the additional energy burden on the body. In control strategy designing, the joint torque-generating strategy provides personalized assistance strategies for wearers to actively optimize the control parameters. Meanwhile, the safety control strategy based on abnormal gait recognition can ensure human safety. Experiments have shown that compared to not wearing exoskeletons, this device can reduce the energy consumption of the human body by 5.33 % at a speed of 9 km/h. This demonstrates its potential in human motion assistance processes.

Collision-free motion planning with trajectory optimization is inherently nonconvex. Some of this nonconvexity is fundamental: the robot might need to make a discrete decision to go left around an obstacle or right around an obstacle. Some of the nonconvexity is potentially more benign: we might want to penalize high-order derivatives of our continuous trajectories in order to encourage smoothness. Recently, Graphs of Convex Sets (GCS) have been applied to trajectory optimization, addressing the fundamental nonconvexity with efficient online optimization over a "roadmap" represented by an approximate convex decomposition of the configuration space. In this paper, we explore some of the most useful nonconvex costs and constraints and the suitability of combining convex "global" optimization using GCS with nonconvex trajectory optimization for rounding the local solutions. We find that for many applications, this combination can lead to a small number of nonconvex optimizations finding extremely good solutions to the nonconvex trajectory optimization problem.