IROS 2024 论文列表

Recognizing and grasping novel-category objects remains a crucial yet challenging problem in real-world robotic applications. Despite its significance, limited research has been conducted in this specific domain. To address this, we seamlessly propose a novel framework that integrates open-vocabulary learning into the domain of robotic grasping, empowering robots with the capability to adeptly handle novel objects. Our contributions are threefold. Firstly, we present a large-scale benchmark dataset specifically tailored for evaluating the performance of open-vocabulary grasping tasks. Secondly, we propose a unified visual-linguistic framework that serves as a guide for robots in successfully grasping both base and novel objects. Thirdly, we introduce two alignment modules designed to enhance visual-linguistic perception in the robotic grasping process. Extensive experiments validate the efficacy and utility of our approach. Notably, our framework achieves an average accuracy of 71.2% and 64.4% on base and novel categories in our new dataset, respectively. Our code and dataset are available at https://github.com/cv516Buaa/OVGNet.

Vehicle-to-infrastructure cooperative 3D object detection (VIC3D) is a task that leverages both vehicle and roadside sensors to jointly perceive the surrounding environment. However, considering the high speed of vehicles, the real-time requirements, and the limitations of communication bandwidth, roadside devices transmit the results of perception rather than raw sensor data or feature maps in our real-world scenarios. And affected by various environmental factors, the transmission delay is dynamic. To meet the needs of practical applications, we present OTVIC, which is the first multi-modality and multi-view dataset with online transmission from real scenes for vehicle-to-infrastructure cooperative 3D object detection. The ego-vehicle receives the results of infrastructure perception in real-time, collected from a section of highway in Chengdu, China. Moreover, we propose LfFormer, which is a novel end-to-end multi-modality late fusion framework with transformer for VIC3D task as a baseline based on OTVIC. Experiments prove our fusion framework’s effectiveness and robustness. Our project is available at https://sites.google.com/view/otvic.

High Definition (HD) Maps have long been favored for their precise depictions of static road elements. However, their accessibility constraints and vulnerability to rapid environmental changes impede the widespread deployment of highly map-reliant autonomous driving tasks, such as motion forecasting. In this context, we propose to leverage OpenStreetMap (OSM) as a promising alternative to HD Maps for long-term motion forecasting. The contributions of this work are threefold: firstly, we extend the application of OSM to long-horizon forecasting, doubling the forecasting horizon compared to previous studies. Secondly, through an expanded observation landscape and the integration of intersection priors, our OSM-based approach exhibits competitive performance, narrowing the gap with HD-map-based models. Lastly, we conduct an exhaustive context-aware analysis, providing deeper insights in motion forecasting across diverse scenarios as well as conducting class-aware comparisons. This research not only advances long-term motion forecasting with coarse map representations but additionally offers a scalable solution within the domain of autonomous driving.

Goal-oriented grasping in dense clutter, a fundamental challenge in robotics, demands an adaptive policy to handle occluded target objects and diverse configurations. Previous methods typically learn policies based on partially observable segments of the occluded target to generate motions. However, these policies often struggle to generate optimal motions due to uncertainties regarding the invisible portions of different occluded target objects across various scenes, resulting in low motion efficiency. To this end, we propose OPG-Policy, a novel framework that leverages amodal segmentation to predict occluded portions of the target and develop an adaptive push-grasp policy for cluttered scenarios where the target object is partially observed. Specifically, our approach trains a dedicated amodal segmentation module for diverse target objects to generate amodal masks. These masks and scene observations are mapped to the future rewards of grasp and push motion primitives via deep Q-learning to learn the motion critic. Afterward, the push and grasp motion candidates predicted by the critic, along with the relevant domain knowledge, are fed into the coordinator to generate the optimal motion implemented by the robot. Extensive experiments conducted in both simulated and real-world environments demonstrate the effectiveness of our approach in generating motion sequences for retrieving occluded targets, outperforming other baseline methods in success rate and motion efficiency.

Recent advancements in artificial hand development have primarily concentrated on enhancing adaptive grasping, dexterity, as well as the integration of biomimetic skin. However, few designs have successfully combined lightweight, cost-effective solutions, and tactile sensing along with adaptive grasping in a human-sized prototype. We propose, an open-source, highly integrated artificial hand. It leverages a compliant linkage mechanism for versatile grasping capabilities, featuring six degrees of actuation and MEMS-based tactile sensors on every fingertip.

Obstacle detection and tracking represent a critical component in robot autonomous navigation. In this paper, we propose ODTFormer, a Transformer-based model that addresses both obstacle detection and tracking problems. For the detection task, our approach leverages deformable attention to construct a 3D cost volume, which is decoded progressively in the form of voxel occupancy grids. We further track the obstacles by matching the voxels between consecutive frames. The entire model can be optimized in an end-to-end manner. Through extensive experiments on DrivingStereo and KITTI benchmarks, our model achieves state-of-the-art performance in the obstacle detection task. We also report comparable accuracy to state-of-the-art obstacle tracking models while requiring only a fraction of their computation cost, typically ten-fold to twenty-fold less. Our code is available on https://github.com/neu-vi/ODTFormer.

In this paper, we introduce a novel overlap-based bidirectional point set registration approach, i.e., Overlap-guided Bidirectional Hybrid Mixture Registration (OBHMR), which incorporates geometric information (i.e., normal vectors) in both the correspondence and transformation stages and formulates the optimization objective of registration in a bidirectional manner. More importantly, to address the issue of partial-to-full registration, OBHMR utilises the predicted point-wise overlap score using networks to formulate the overlap-guided Hybrid Mixture Model consisting of the Gaussian Mixture Model (GMM) and Fisher Mixture Model (FMM). OBHMR contains four components: (1) the overlap-guided correspondence network that estimates the correspondence probabilities and calculates the point-wise overlap score; (2) the learning posterior module that estimates the overlap-guided HMM parameters; (3) the transformation module that computes the rigid transformation by formulating the optimisation objective in a bidirectional registration way, given correspondences and overlap-guided HMM parameters. Experiments using 1457 human femur and 1301 human hip models demonstrate significant improvements in partial-to-full registration performance (p < 0.01) under different overlapping ratios, compared to state-of-the-art registration approaches. Furthermore, individual contributions of three modules (i.e., additional normal vectors, overlap score estimation module and the bidirectional mechanism) in OBHMR have been validated in ablation studies. The results demonstrate OBHMR’s capability of tackling the challenging partial-to-full registration problems in computer-assisted orthopedic surgery. The codes are available at https://github.com/Dxinz/DeepOBHMR.

Bathymetry mapping of static water bodies like lakes is essential for sustainable ecosystem development strategies. However, bathymetry mapping using (i) traditional manual sampling approaches using single-beam echosounder (SBE) has large mapping errors and (ii) performing multi-beam survey is very expensive. Alternatively, performing lawn-mower SBE surveys is time consuming due to limited field-of-view. In order to address the above issues, in this paper, we present an on-the-way sampling approach with Gaussian Process Upper Confidence Bound (GPUCB) algorithm called as OAS-GPUCB that can adaptively sample the lake to minimize the bathymetry error while reducing the distance travelled to achieve a given mapping accuracy. We validate the proposed approach using simulations on actual lake bathymetry maps and also carry out real-world experiments using an Autonomous Surface Vehicle(ASV) with SBE. Further, we compare OAS-GPUCB to lawn-mower, GPUCB, and GPUCB with fixed radius approaches. The results consistently show that the proposed approach can achieve less than 10% bathymetry error while achieving distance reduction of more than 55%compared to the lawn-mower approach, and more than 90% less distance travelled compared to GPUCB and GPUCB with fixed radius approaches. The results shows the general applicability of OAS-GPUCB for bathymetry mapping of water bodies without any prior information maps.

Tracked robots with reconfigurable mechanisms exhibit great maneuverability due to their adaptability to complex ground conditions. Reconfigurable tracked robots with geometry-changing tracks show further obstacle-crossing capabilities with compact dimensions. However, existing systems face deployment limitations due to either complex transmission mechanisms or unsustainable designs when maintaining the tension in the tracks. To address these challenges, we introduce a novel design of a reconfigurable tracked robot with geometry-changing tracks, which achieves strong terrain traversability with good mechanical properties. We achieve the elliptical trajectory of key planetary wheels through a novel Quad-slider Elliptical Trammel Mechanism (Qs-ETM), allowing the tracks to maintain fixed tension while changing their geometry. Furthermore, the combination of direct drive motors significantly enhances its mechanical properties and agility. A detailed analysis of the kinematic and dynamic characteristics has been conducted and proved with a series of simulations. We built a fully functional prototype of the design and tested it in real-world experiments to validate its advantages. The result shows that our design can reduce the torque required by up to 68.3% and the shear stress of the flipper by up to 67.1%.

The differential mechanism can reduce the number of actuators and efficiently distribute force or power. We proposed a novel multiport output twisted string actuator (MO-TSA) with self-differential mechanism that employs a single actuator to achieve multiport outputs. The differential MO-TSA is adaptively controlled in accordance with the force differences at each output port, thus replacing the traditional differential gears and whiffletree mechanisms. Inspired by the hand muscles, we designed one hand glove using the MO-TSA, aiming to enhance the range of achievable grasp configurations. The hand glove is capable of performing various grasps with a single actuator, resulting in a lighter and simpler hand design and revolutionizing the field of twisted string actuators (TSAs) by offering a streamlined solution for achieving versatile actuation.

LiDAR is an essential sensor for intelligent vehicles. Recently, LiDARs used in vehicles produced by different companies have significant differences in their scanning patterns. Some vehicles use mechanical and solid-state (repetitive) LiDARs, while others use prism-based (non-repetitive) LiDARs. The scanning pattern of a LiDAR has a profound impact on its scanning performance. To investigate the influence of LiDAR scanning patterns, we created the "Repetitive-or-not" dataset, which is collected simultaneously by LiDARs with both repetitive and non-repetitive scanning patterns in the CARLA simulation environment. Using this dataset, we conducted a comprehensive statistical analysis of the scanning ability of repetitive and non-repetitive LiDARs. Furthermore, we looked into the effects of these two LiDAR scanning patterns on the performance of various 3D object detection algorithms. Finally, we explored the domain gap in the point cloud data produced by repetitive and non-repetitive LiDARs. Through an in-depth investigation of the "Repetitive-or-not" dataset, we have discovered that non-repetitive LiDAR shows great promise. This conclusion is primarily supported by its superior object scanning capabilities.

As robots become smarter and more ubiquitous, optimizing the power consumption of intelligent compute becomes imperative towards ensuring the sustainability of technological advancements. Neuromorphic computing hardware makes use of biologically inspired neural architectures to achieve energy and latency improvements compared to conventional von Neumann computing architecture. Applying these benefits to robots has been demonstrated in several works in the field of neurorobotics, typically on relatively simple control tasks. Here, we introduce an example of neuromorphic computing applied to the real-world industrial task of object insertion. We trained a spiking neural network (SNN) to perform force-torque feedback control using a reinforcement learning approach in simulation. We then ported the SNN to the Intel neuromorphic research chip Loihi interfaced with a KUKA robotic arm. At inference time we show latency competitive with current CPU/GPU architectures, and one order of magnitude less energy usage in comparison to traditional low-energy edge-hardware. We offer this example as a proof of concept implementation of a neuromoprhic controller in real-world robotic setting, highlighting the benefits of neuromorphic hardware for the development of intelligent controllers for robots.

We present an efficient and scalable learning-based autonomous exploration system for mobile robots navi-gating unknown indoor environments. Our system incorporates three network models trained to identify the frontier region (FR), to evaluate the detected FR regions based on their proximity to the robot (A*-Net), and to measure the coverage reward at the FR regions (Viz-Net). Our method employs an active window of the map that moves along with the robot, offering scalable exploration capabilities while maintaining a high rate of exploration coverage owing to the two exploratory measures utilized by A*-Net (proximity) and Viz-Net (coverage). Consequently, Our system completes over 99% coverage in a large-scale benchmarking world, scaling up to 135m × +80m. In contrast, other state-of-the-art approaches completed only less than 40% of the same world with a 30% slower exploration speed than ours.

We present NeuralLabeling, a labeling approach and toolset for annotating 3D scenes using either bounding boxes or meshes and generating segmentation masks, affordance maps, 2D bounding boxes, 3D bounding boxes, 6DOF object poses, depth maps, and object meshes. NeuralLabeling uses Neural Radiance Fields (NeRF) as a renderer, allowing labeling to be performed using 3D spatial tools while incorporating geometric clues such as occlusions, relying only on images captured from multiple viewpoints as input. To demonstrate the applicability of NeuralLabeling to a practical problem in robotics, we added ground truth depth maps to 30000 frames of transparent object RGB and noisy depth maps of glasses placed in a dishwasher captured using an RGBD sensor, yielding the Dishwasher30k dataset. We show that training a simple deep neural network with supervision using the annotated depth maps yields a higher reconstruction performance than training with the previously applied weakly supervised approach. We also show how instance segmentation and depth completion datasets generated using NeuralLabeling can be incorporated into a robot application for grasping transparent objects placed in a dishwasher with an accuracy of 83.3%, compared to 16.3% without depth completion. Supplementary URI: https://florise.github.io/neural_labeling_web/.

Learning autonomous driving capabilities requires diverse and realistic training data. This has led to exploring generative techniques as an alternative to real-world data collection. In this paper we propose a method for synthesising photo-realistic urban driving scenes, along with semantic, instance and depth ground-truth. Our model relies on Bird’s Eye View (BEV) representations due to their compositionality and scene content control capabilities, reducing the need for traditional simulators. We employ a two-stage process: first, a 3D scene representation is extracted from BEV semantic, instance and style maps using a neural field. After rendering the semantic, instance, depth and style maps from a ground-view perspective, a second stage based on a diffusion model is used to generate the photo-realistic scene. We extend our prior work - NeuralFloors, to include multiple-view outputs, style manipulation for finer control at the object level through instance-wise style maps and cross-frame consistency via auto-regressive training. The proposed system is evaluated extensively on the KITTI-360 dataset, showing improved realism and semantic alignment for generated images.

The multi-agent trajectory planning problem is a difficult problem in robotics due to its computational complexity and real-world environment complexity with uncertainty, non-linearity, and real-time requirements. Many existing solutions are either search-based or optimization-based approaches with simplified assumptions of environment, limited planning speed, and limited scalability in the number of agents. In this work, we first attempt to reformulate single-agent and multi-agent trajectory planning problems as query problems over an implicit neural representation of trajectories. We formulate such implicit representations as Neural Trajectory Models (NTM) which can be queried to generate nearly optimal trajectory in complex environments. We conduct experiments in simulation environments and demonstrate that NTM achieve (1) sub-millisecond planning time using GPUs, (2) almost avoiding all collisions, and (3) generating almost shortest paths. We also demonstrate that the same NTM framework can also be used for refining low-quality and conflicting multi-agent trajectories into nearly optimal solutions efficiently. (Open source code is available at https://github.com/laser2099/neural-trajectory-model)

Autonomous vehicles demand detailed maps to maneuver reliably through traffic, which need to be kept up-to-date to ensure a safe operation. A promising way to adapt the maps to the ever-changing road-network is to use crowd-sourced data from a fleet of vehicles. In this work, we present a mapping system that fuses local submaps gathered from a fleet of vehicles at a central instance to produce a coherent map of the road environment including drivable area, lane markings, poles, obstacles and more as a 3D mesh. Each vehicle contributes locally reconstructed submaps as lightweight meshes, making our method applicable to a wide range of reconstruction methods and sensor modalities. Our method jointly aligns and merges the noisy and incomplete local submaps using a scene-specific Neural Signed Distance Field, which is supervised using the submap meshes to predict a fused environment representation. We leverage memory-efficient sparse feature-grids to scale to large areas and introduce a confidence score to model uncertainty in scene reconstruction. Our approach is evaluated on two datasets with different local mapping methods, showing improved pose alignment and reconstruction over existing methods. Additionally, we demonstrate the benefit of multi-session mapping and examine the required amount of data to enable high-fidelity map learning for autonomous vehicles.

In robotics, acquiring new skills through Imitation Learning (IL) is crucial for handling diverse complex tasks. However, model-free IL faces challenges of data inefficiency and prolonged training time, whereas model-based methods struggle to obtain accurate nonlinear models. To address these challenges, we developed Neural ODE-based Imitation Learning (NODE-IL), a novel model-based imitation learning framework that employs Neural Ordinary Differential Equations (Neural ODEs) for learning task dynamics and control policies. NODE-IL comprises (1) Dynamic-NODE for learning the continuous differentiable task’s transition dynamics model, and (2) Control-NODE for learning a long-horizon control policy in an MPC fashion, which are trained holistically. Extensively evaluated on challenging manipulation tasks, NODE-IL demonstrates significant advantages in data efficiency, requiring less than 70 samples to achieve robust performance. It outperforms Behavioral Cloning from Observation (BCO) and Gaussian Process Imitation Learning (GP-IL) methods, achieving 70% higher average success rate, and reducing translation errors for high-precision tasks, which demonstrates its robustness and accuracy, as an effective and efficient imitation learning approach for learning complex manipulation tasks.

We integrate neural networks into kinodynamic motion planning and present the Learning for KinoDynamic Tree Expansion (L4KDE) method. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree expansion, the procedure in which new nodes are added to an ever-expanding tree. We study the kinodynamic variants of tree-based planning, where we have known system dynamics and kinematic constraints. In the interest of quickly selecting nodes to connect newly sampled coordinates, existing methods typically cannot optimise the finding of nodes that have a low cost to transition to sampled coordinates. Instead, they use metrics like Euclidean distance between coordinates as a heuristic for selecting candidate nodes to connect to the search tree. We propose L4KDE to address this issue. L4KDE uses a neural network to predict transition costs between queried states, which can be efficiently computed in batch, providing much higher quality estimates of transition cost compared to commonly used heuristics while maintaining almost-surely asymptotic optimality guarantee. We empirically demonstrate the significant performance improvement provided by L4KDE on a variety of challenging system dynamics,with the ability to generalise across different instances of the same model class and in conjunction with a suite of modern tree-based motion planners.