IROS 2024 Papers

State-of-the-art approaches for 6D object pose estimation assume the availability of CAD models and require the user to manually set up physically-based rendering (PBR) pipelines for synthetic training data generation. Both factors limit the application of these methods in real-world scenarios. In this work, we present a pipeline that does not require CAD models and allows training a state-of-the-art pose estimator requiring only a small set of real images as input. Our method is based on a NeuS2 [1] object representation, that we learn through a semi-automated procedure based on Structure-from-Motion (SfM) and object-agnostic segmentation. We exploit the novel-view synthesis ability of NeuS2 and simple cut-and-paste augmentation to automatically generate photorealistic object renderings, which we use to train the correspondence-based SurfEmb [2] pose estimator. We evaluate our method on the LINEMOD-Occlusion dataset, extensively studying the impact of its individual components and showing competitive performance with respect to approaches based on CAD models and PBR data. We additionally demonstrate the ease of use and effectiveness of our pipeline on self-collected real-world objects, showing that our method outperforms state-of-the-art CAD-model-free approaches, with better accuracy and robustness to mild occlusions. To allow the robotics community to benefit from this system, we will publicly release it at https://www.github.com/ethz-asl/neusurfemb.

Real-time high-accuracy optical flow estimation is a crucial component in various applications, including localization and mapping in robotics, object tracking, and activity recognition in computer vision. While recent learning-based optical flow methods have achieved high accuracy, they often come with heavy computation costs. In this paper, we propose a highly efficient optical flow architecture, called NeuFlow, that addresses both high accuracy and computational cost concerns. The architecture follows a global-to-local scheme. Given the features of the input images extracted at different spatial resolutions, global matching is employed to estimate an initial optical flow on the 1/16 resolution, capturing large displacement, which is then refined on the 1/8 resolution with lightweight CNN layers for better accuracy. We evaluate our approach on Jetson Orin Nano and RTX 2080 to demonstrate efficiency improvements across different computing platforms. We achieve a notable 10-80 speedup compared to several state-of-the-art methods, while maintaining comparable accuracy. Our approach achieves around 30 FPS on edge computing platforms, which represents a significant breakthrough in deploying complex computer vision tasks such as SLAM on small robots like drones. The full training and evaluation code is available at https://github.com/neufieldrobotics/NeuFlow.

Visual servoing for the development of autonomous robotic systems capable of administering UltraSound (US) guided regional anesthesia requires real-time segmentation of nerves, needle tip localization and needle trajectory extrapolation. First, we recruited 227 patients to build a large dataset of 41,000 anesthesiologist annotated images from US videos of brachial plexus nerves and developed models to localize nerves in the US images. Generalizability of the best suited model was tested on the datasets constructed from separate US scanners. Using these nerve segmentation predictions, we define automated anesthesia needle targets by fitting an ellipse to the nerve contours. Next, we developed an image analysis tool to guide the needle toward their targets. For the segmentation of the needle, a natural RGB pre-trained neural network was first fine-tuned on a large US dataset for domain transfer and then adapted for the needle using a small dataset. The segmented needle’s trajectory angle is calculated using Radon transformation and the trajectory is extrapolated from the needle tip. The intersection of the extrapolated trajectory with the needle target guides the needle navigation for drug delivery. The needle trajectory’s average error was within acceptable range of 5 mm as per experienced anesthesiologists. The entire dataset has been released publicly for further study by the research community at https://github.com/Regional-US/

As a crucial part of mobile robotics and autonomous driving, Visual Place Recognition (VPR) is usually addressed by recognizing its similar reference images from a pre-obtained database. However, VPR always suffers from environmental changes, such as weather, illumination, perceptual-aliasing and so on. To address this, we firstly introduce a robust and discriminative global descriptor aggregation technique that normalizes the spatial and channel dimensions of features. A Spatial-Channel Embedding (SCE) module is proposed to learn the spatial and scale information of features which make global features more discriminative. Meanwhile, the traditional re-ranking methods (e.g. RANSAC) for geometric consistency verification are time-consuming. Here we propose a Neighborhood Consensus Guided Matching (NCGM) module, which uses Neighborhood Consensus to filter the features from patch-level matching to achieve more accurate matching while reduces the time consumption. Through extensive experiments on multiple benchmarks, we demonstrate that our method outperforms several state-of-the-art methods while maintaining lower time consumption and storage requirements.

Recent advancements in deep learning have significantly enhanced the development of efficient models for multi-modal path prediction within urban environments, offering approaches to navigate complex environments accurately. Despite their performance, models grounded in deep learning techniques frequently encounter challenges related to interpretability. This limitation not only hampers their practical application but also complicates the process of diagnosing and rectifying errors within these systems, which is a critical factor for ensuring reliability and safety in realworld deployments. In this paper we propose NeSyMoF, a Neuro-Symbolic model for Motion Forecasting, to address this critical gap by combining the predictive power of deep neural networks with the interpretable logic inherent in symbolic reasoning. Data processing in NeSyMoF involves extracting pertinent features from the agent’s environment and channeling them into a neuro-symbolic reasoning module. The neurosymbolic reasoning module generates first-order logic rules that describe and condition the path prediction process, thereby providing clear explanations and intentions behind the forecasts of the model. We evaluate our model with the Argoverse benchmark for path forecasting, as it includes challenging driving situations, necessary to extensively evaluate our model. The results of our evaluation show that NeSyMoF outperforms state-of-the-art interpretable models for single-mode predictions while providing logic-based explanations for its forecasts, that articulate the reasoning behind predictions, making NeSyMoF more adapted for human-centric applications.

Inverse Synthetic Aperture Radar (ISAR) imaging presents a formidable challenge when it comes to small everyday objects due to their limited Radar Cross-Section (RCS) and the inherent resolution constraints of radar systems. Existing ISAR reconstruction methods including backprojection (BP) often require complex setups and controlled environments, rendering them impractical for many real-world noisy scenarios. In this paper, we propose a novel Analysis-through-Synthesis (ATS) framework enabled by Neural Radiance Fields (NeRF) for high-resolution coherent ISAR imaging of small objects using sparse and noisy Ultra-Wideband (UWB) radar data with an inexpensive and portable setup. Our end-to-end framework integrates ultra-wideband radar wave propagation, reflection characteristics, and scene priors, enabling efficient 2D scene reconstruction without the need for costly anechoic chambers or complex measurement test beds. With qualitative and quantitative comparisons, we demonstrate that the proposed method outperforms traditional techniques and generates ISAR images of complex scenes with multiple targets and complex structures in Non-Line-of-Sight (NLOS) and noisy scenarios, particularly with limited number of views and sparse UWB radar scans. This work represents a significant step towards practical, cost-effective ISAR imaging of small everyday objects, with broad implications for robotics and mobile sensing applications.

Reconfigurable microswarms have received extensive attention recently. In this work, we propose a control strategy for a ribbon-like swarm to perform navigated locomotion with a stable pattern, and perform controllable splitting into double subswarms to reach two targets simultaneously. Two different behaviors of the ribbon-like swarm are firstly investigated, i.e., locomotion with a stable pattern, and controllable splitting. The two behaviors of the ribbon-like swarm are realized based on different aspect ratio of the swarm. Subsequently, we propose a morphology controller to keep the aspect ratio of the swarm within a desired range. The morphology controller consists of a feedforward controller and a PD controller. The feedforward controller containing a fitted model, and a fuzzy logic controller for online compensation of the model error. The control strategy combining the morphology planning, morphology controller, path planning, and motion controller is developed. Using the proposed control strategy, the ribbon-like swarm can be navigated to follow a desired path with a stable pattern while avoiding obstacles, and finally perform controllable splitting into double subswarms to reach two predefined targets simultaneously.

The pursuit of 3D reconstruction from 2D images for nanomanipulation under scanning electron microscopy stands as a critical research endeavor. Previous methods either necessitates additional lighting which is difficult in standard SEM devices or relies on feature matching with low resolution and precision, further constraining reconstruction performance. In this paper, we propose a novel robot-assisted nanoscale 360° reconstruction approach, which simplifies SEM setups and maximizes the utilization of robot motion and feedback. By harnessing a nanorobotic system, we capture 360°multi-view images automatically with precise mapping information and camera postures. Sequentially, neural radiance field reconstruct the pixel-wise structure and synthesizing images from diverse perspectives. Experimental results using two real datasets demonstrates our approach’s efficacy, achieving PSNR of 28.1 and SSIM of 0.93 for nanotube reconstruction, and PSNR of 32.8 and SSIM of 0.98 for AFM cantilever reconstruction. These results validate the reliability and robustness of our proposed robot-assisted reconstruction method.

In recent years, Neural Radiance Fields (NeRF) have emerged as a powerful tool for 3D reconstruction and novel view synthesis. However, the computational cost of NeRF rendering and degradation in quality due to the presence of artifacts pose significant challenges for its application in real-time and robust robotic tasks, especially on embedded systems. This paper introduces a novel framework that integrates NeRF-derived localization information with Visual-Inertial Odometry (VIO) to provide a robust solution for real-time robotic navigation. By training an absolute pose regression network with augmented image data rendered from a NeRF and quantifying its uncertainty, our approach effectively counters positional drift and enhances system reliability. We also establish a mathematically sound foundation for combining visual inertial navigation with camera localization neural networks, considering uncertainty under a Bayesian framework. Experimental validation in a photorealistic simulation environment demonstrates significant improvements in accuracy compared to a conventional VIO approach.

In this work, we propose a novel method, NLNS-MASPF, to solve the Multi-Agent Scheduling and Pathfinding (MASPF) problem. The problem exhibits a bi-level structure, consisting of High-level Scheduling and Low-level Pathfinding. Our method applies a graph neural network in the high-level scheduling process and utilizes a MAPF solver with a schedule segmenting technique in the low-level pathfinding process. Through these approaches, NLNS-MASPF has experimentally demonstrated superior performance compared to the previous state-of-the-art MASPF algorithm, LNS-PBS, in solving the MASPF problem.

Reinforcement learning with domain randomization (DR) has been proposed as a promising approach for learning robust policies to environmental changes. However, for DR to work well in real-world environments, it is necessary to design appropriate DR distributions for model parameters. This paper proposes Normalizing Flow-based Parameter Distribution Estimation (NFPDE), a new estimation method for DR distributions. NFPDE models the target distribution by a flow-based generative model using normalizing flow and estimates the target distribution based on an offline dataset collected a priori in the target environment. Through numerical experiments on the OpenAI gym environment, we show that NFPDE can estimate the target distribution more accurately and efficiently than the previous estimation methods. We also show that the estimated DR distributions can improve the robustness of trained policies.

We propose a novel, vision-only object-level SLAM framework for automotive applications representing 3D shapes by implicit signed distance functions. Our key innovation consists of augmenting the standard neural representation by a normalizing flow network. As a result, achieving strong representation power on the specific class of road vehicles is made possible by compact networks with only 16-dimensional latent codes. Furthermore, the newly proposed architecture exhibits a significant performance improvement in the presence of only sparse and noisy data, which is demonstrated through comparative experiments on synthetic data. The module is embedded into the back-end of a stereo-vision based framework for joint, incremental shape optimization. The loss function is given by a combination of a sparse 3D point-based SDF loss, a sparse rendering loss, and a semantic mask-based silhouette-consistency term. We furthermore leverage semantic information to determine keypoint extraction density in the front-end. Finally, experimental results on real-world data reveal accurate and reliable performance comparable to alternative frameworks that make use of direct depth readings. The proposed method performs well with only sparse 3D points obtained from bundle adjustment, and eventually continues to deliver stable results even under exclusive use of the mask-consistency term.

Loop-closure detection, also known as place recognition, aiming to identify previously visited locations, is an essential component of a SLAM system. Existing research on lidar-based loop closure heavily relies on dense point cloud and 360 FOV lidars. This paper proposes an out-of-the-box NDT (Normal Distribution Transform) based global descriptor, NDT-Map-Code, designed for both on-road driving and underground valet parking scenarios. NDT-Map-Code can be directly extracted from the NDT map without the need for a dense point cloud, resulting in excellent scalability and low maintenance cost. The NDT representation is leveraged to identify representative patterns, which are further encoded according to their spatial location (bearing, range, and height). Experimental results on the NIO underground parking lot dataset and the KITTI dataset demonstrate that our method achieves significantly better performance compared to the state-of-the-art.

The impressive capabilities of Large Language Models (LLMs) have led to various efforts in enabling robots to be controlled through natural language instructions, opening exciting possibilities for human-robot interaction. The goal is for the motor-control task to be performed accurately, efficiently and safely while also enjoying the flexibility imparted by LLMs to specify and adjust the task through natural language. In this work, we demonstrate how a careful layering of an LLM in combination with a Model Predictive Control (MPC) formulation allows for accurate and flexible robotic control via natural language while taking into consideration safety constraints. In particular, we rely on the LLM to effectively frame constraints and objective functions as mathematical expressions, which are later used in the motor-control module via MPC. The transparency of the optimization formulation allows for interpretability of the task and enables adjustments through human feedback. We demonstrate the validity of our method through extensive experiments on long-horizon reasoning, contact-rich, and multi-object interaction tasks. Our evaluations show that NARRATE outperforms current existing methods on these benchmarks and effectively transfers to the real world on two different embodiments.Videos, Code and Prompts at narrate-mpc.github.io

Collaboration with robots through physical contact offers a more intuitive, natural, and engaging operational experience, showcasing vast potential in the field of human-robot interaction. However, current physical interaction devices, such as collaborative robots and haptic feedback mechanisms, are limited by their singular modes of motion and feedback, hindering enhancements in interaction experiences. Herein, we present a multistable soft actuator capable of driving multimodal shape changes and passively conforming to user touch. This actuator can memorize and maintains any deformation with zero power consumption. Its structural mechanical properties can be dynamically adjusted to produce rich haptic feedback for the user, including changes in shape, elasticity, stiffness, and even sensations of rupture and weightlessness. Structurally, the mechanism consists of a network of pneumatic bistable units in series and parallel configurations, which can switch states under air pressure or external force, achieving extension, contraction, and omnidirectional bending. The input of air pressure can either impede or assist deformation, altering structural stiffness and resulting in varied loading curves. With its high safety in physical interactions, robust operability, and rich mechanical tactile feedback, the multistable soft actuator promises new design directions for physical human-robot interaction devices.

Multiple-suction-cup grasp is preferable for picking large and heavy objects in warehouses. Deep learning methods have been widely used to predict grasp point position for single-suction-cup grasp, but few studies have examined grasp pose detection for a gripper with multiple suction cups. This study proposes MultipleCupSuctionNet, which is a deep neural network for detecting multiple cup grasp pose. To address the challenge of direct regression of poses, this neural network first infers the surface mask to compute the surface normal to obtain the direction of the gripper z-axis. The feature maps of the surfaces are then affine-transformed to surface image coordinates, based on which gripper position and rotation angle around the z-axis are predicted. Such a neural network design makes grasp pose learning easier because there is no need to consider the orientation of the surfaces so that 2D poses respective to the surface are learned. Feature map affine transformation saves computation cost because there is no need to first transform images and then extract the features to obtain surface features. Further, for each predicted grasp pose, the overlap area between cup and surface is calculated to determine which cup should be used when grasping. MultipleCupSuctionNet exhibited competitive performance (80.1% prediction accuracy) particularly in dense scenes compared with state-of-the-art planners (Dex-Net and a model-free multiple-suction-cup grasp planner). Physical picking experiments were conducted using a robot employing the proposed neural network. The experimental results showed that our robot achieved an average success rate of 94.5% for picking common objects in warehouses.

Vision-and-Language Navigation (VLN) in continuous environments aims to navigate robot agents in unseen environments following natural language instructions. The majority of existing approaches rely on constructing semantic maps or topological maps to record information. However, semantic maps overlook the detailed information of objects and the correspondence among views during navigation, while topological maps lack the spatial representation between entities. To address these limitations, we propose a novel visual feature representation method for continuous VLN, called Multiple Visual Features in Topological Map (MV-Topo). MV-Topo utilizes three distinct visual encoders to extract visual features, which are integrated in the dynamically generated topological map. These fused features actively participate in the subsequent cross-modal planning to derive a long-term path towards a subgoal, effectively guiding the agent to reach the final location. We experimentally demonstrate the effectiveness of our approach and achieve competitive results on the full VLN-CE test splits. Notably, our method outperforms the state-of-the-art by 3.5% in terms of the Navigation Error (NE) metric, indicating that the utilization of multiple visual features significantly enhances the agent’s perception of semantic targets.

This study investigates the efficacy of a haptic interface, aiming to offer the walking frame users accurate, intuitive, and easily understandable directional cues. The research introduces a novel haptic feedback interface incorporated into a walking frame to enhance navigation assistance. The haptic handle encompasses three distinct haptic feedback modalities: vibration, skin stretch, and combined feedback. Ten participants, all in good health, engaged with the haptic handle for navigation. Across all three haptic feedback methods, 60% of participants found the combined feedback to be the most effective, while 40% favoured the vibration feedback; none selected the skin-stretch feedback. Comparative analysis revealed significant disparities between vibration and combined input regarding velocity (p-value: 0.04). These findings emphasize the haptic handle’s capacity to give users an instinctive perception of directional cues, thus offering a promising avenue for assistive navigation.

Due to the increasing behavioral and structural complexity of robots, it is challenging to predict the execution outcome after error detection. Anomaly detection methods can help detect errors and prevent potential failures. However, not every fault leads to a failure due to the system’s fault tolerance or unintended error masking. In practical applications, a robotic system should have a potential failure evaluation module to estimate the probability of failures when receiving an error alert. Subsequently, a decision-making mechanism should help to take the next action, e.g., terminate, degrade performance, or continue the execution of the task. This paper proposes a multimodal method for failure prediction for vision-based manipulation systems that suffer from potential camera faults. We inject faults into images (e.g., noise and blur) and observe manipulation failure scenarios (e.g., pick failure, place failure, and collision) that can occur during the task. Through extensive fault injection experiments, we created a FAULT-to-FAILURE dataset containing 4000 real-world manipulation samples. The dataset is subsequently used to train the failure predictor. Our approach processes the combination of RGB images, masked images, and planned paths to effectively evaluate whether a certain faulty image could potentially lead to a manipulation failure. Results demonstrate that the proposed method outperforms state-of-the-art models in terms of overall performance, requires fewer sensors, and achieves faster inference speeds. The analytical software prototype and dataset are available at Github: MultimodalFailurePrediction.