ICRA 2022 Papers

Grasping a wide range of novel objects in densely cluttered scenes is difficult due to irregular shapes of objects and the uncertainty in sensing. In this paper, a novel vacuum cup grasping method, based on uncertainty modeling of perception data and grasp geometric heuristics, is proposed to grasp unknown objects in densely cluttered scenes. The probabilistic signed distance function is proposed to both reconstruct the point cloud of a scene and explicitly model the uncertainty from depth images captured from a low-cost stereo camera. The quasi-static spring model is used to approximate seal formation between the suction cup and the reconstructed point cloud. A coarse-to-fine exploration procedure is proposed to refine the estimated point cloud, reduce uncertainties during the movement of the robot and redetermine the target grasp pose iteratively. Extensive experiments show that our proposed method achieves state-of-the-art performance on real-world grasping and outperforms existing methods by a large margin.

Deployment of deep neural networks (DNNs) for monocular depth estimation in safety-critical scenarios on resource-constrained platforms requires well-calibrated and efficient uncertainty estimates. However, many popular uncertainty estimation techniques, including state-of-the-art ensembles and popular sampling-based methods, require multiple inferences per input, making them difficult to deploy in latency-constrained or energy-constrained scenarios. We propose a new algorithm, called Uncertainty from Motion (UfM), that requires only one inference per input. UfM exploits the temporal redundancy in video inputs by merging incrementally the per-pixel depth prediction and per-pixel aleatoric uncertainty prediction of points that are seen in multiple views in the video sequence. When UfM is applied to ensembles, we show that UfM can retain the uncertainty quality of ensembles at a fraction of the energy by running only a single ensemble member at each frame and fusing the uncertainty over the sequence of frames. In a set of representative experiments using FCDenseNet and eight indistribution and out-of-distribution video sequences, UfM offers comparable uncertainty quality to an ensemble of size 10 while consuming only 11.3% of the ensemble's energy and running 6.4× faster on a single Nvidia RTX 2080 Ti GPU, enabling near ensemble uncertainty quality for resource-constrained, real-time scenarios.

Disparity and optical flow estimation are respectively 1D and 2D dense correspondence matching (DCM) tasks in nature. Unsupervised domain adaptation (UDA) is crucial for their success in new and unseen scenarios, enabling networks to draw inferences across different domains without manually-labeled ground truth. In this paper, we propose a general UDA framework (UnDAF) for disparity or optical flow estimation. Unlike existing approaches based on adversarial learning that suffers from pixel distortion and dense correspondence mismatch after domain alignment, our UnDAF adopts a straightforward but effective coarse-to-fine strategy, where a co-teaching strategy (two networks evolve by complementing each other) refines DCM estimations after Fourier transform initializes domain alignment. The simplicity of our approach makes it extremely easy to guide adaptation across different domains, or more practically, from synthetic to real-world domains. Extensive experiments carried out on the KITTI and MPI Sintel benchmarks demonstrate the accuracy and robustness of our UnDAF, advancing all other state-of-the-art UDA approaches for disparity or optical flow estimation. Our project page is available at https://sites.google.com/view/undaf.

We propose an efficient method for unsupervised learning of metric depth estimation from a single image in the context of unconstrained videos captured from UAVs. We combine the accuracy of an analytical solution based on odometry with the power of deep learning. First, we show how to correct the noisy odometric measurements by optimizing the alignment between the derotated optical flow and the projected linear speed in the image. Then, we detail an analytical depth estimation method based on optical flow and corrected camera velocities. Subsequently, the improved depth and camera veloc-ities obtained analytically are used, as additional cost terms, for training our novel unsupervised learning architecture for metric depth estimation. We extensively test on a recent UAV dataset, which we significantly extend by adding completely novel scenes. We outperform by significant margins different kinds of state-of-the-art approaches, ranging from analytical and unsupervised solutions to transformer-based architectures that require heavy computation and pre-training. The resulting algorithm could be deployed on embedded devices, being a good candidate for practical robotics use cases, such as obstacle avoidance and safe landing for UAV s.

Modular and truss robots offer the potential of high reconfigurability and great functional flexibility, but common implementations relying on rigid components often lead to highly complex actuation and control requirements. This paper introduces a new type of modular, compliant robot: TrussBot. TrussBot is composed of 3D-printed tetrahedral modules connected at the corners with compliant joints. We propose a truss geometry, analyze its deformation modes, and provide a simulation framework for predicting its behavior under applied loads and actuation. The TrussBot is geometrically constrained, thus requiring compliant joints to move. The TrussBot can be actuated through a network of tendons which pinch vertices together and apply a twisting motion due to the structure's connectivity. The truss was demonstrated in a physical prototype and compared to simulation results.

We present a framework for dynamic trajectory generation for autonomous navigation, which does not rely on HD maps as the underlying representation. High Definition (HD) maps have become a key component in most autonomous driving frameworks, which include complete road network information annotated at a centimeter-level that include traversable waypoints, lane information, and traffic signals. Instead, the presented approach models the distributions of feasible ego-centric trajectories in real-time given a nominal graph-based global plan and a lightweight scene representation. By embedding contextual information, such as crosswalks, stop signs, and traffic signals, our approach achieves low errors across multiple urban navigation datasets that include diverse intersection maneuvers, while maintaining real-time performance and reducing network complexity. Underlying datasets introduced are available online.

Global point cloud registration is an essential module for localization, of which the main difficulty exists in estimating the rotation globally without initial value. With the aid of gravity alignment, the degree of freedom in point cloud registration could be reduced to 4DoF, in which only the heading angle is required for rotation estimation. In this paper, we propose a fast and accurate global heading angle estimation method for gravity-aligned point clouds. Our key idea is that we generate a translation invariant representation based on Radon Transform, allowing us to solve the decoupled heading angle globally with circular cross-correlation. Besides, for heading angle estimation between point clouds with different distributions, we implement this heading angle estimator as a differentiable module to train a feature extraction network end-to-end. The experimental results validate the effectiveness of the proposed method in heading angle estimation and show better performance compared with other methods.

We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1–1 correspondence between a vertical scanline in the image, and rays passing through the camera location in an overhead map. This lets us formulate map generation from an image as a set of sequence-to-sequence translations. Posing the problem as translation allows the network to use the context of the image when interpreting the role of each pixel. This constrained formulation, based upon a strong physical grounding of the problem, leads to a restricted transformer network that is convolutional in the horizontal direction only. The structure allows us to make efficient use of data when training, and obtains state-of-the-art results for instantaneous mapping of three large-scale datasets, including a 15% and 30% relative gain against existing best performing methods on the nuScenes and Argoverse datasets, respectively.

Robotic grasping pose detection that predicts the configuration of the robotic gripper for object grasping is fundamental in robot manipulation. Based on point clouds, most of the existing methods predict grasp pose with the hierarchical PointNet++ backbone, while the non-local geometric information is underexplored. In this work, we address the 7-DoF (6- DoF with the grasp width) grasp detection by introducing a one- stage Transformer-based hierarchical multi-scale model dubbed TransGrasp. Empowered by TransGrasp, the point features are enhanced via acquiring multi-scale shape awareness in the whole scene. By directly modeling the long-range relevance, our pipeline is aware of object contour to avoid collisions and able to apply analogy reasoning for long-distance geometric structures. The evaluation results on the large scale GraspNet- 1Billion dataset demonstrate the effectiveness of the proposed TransGrasp. The real robot experiments on an ABB YUMI robot with an Azure Kinect DK camera and an ABB Smart two-finger gripper show high success rates in both single object and cluttered scenes.

Trajectory scaling has long been used to address velocity and acceleration constraints in robotic motion planning. In later years, reactive motion planning based on dynamical systems has become popular. The traditional scaling techniques are not always suitable to adopt directly when online modifications of the trajectories are made leading to feasibility problems. In this paper, we propose an approach which scales trajectories modelled as dynamical systems for improved feasibility. This is achieved by proactively scaling the trajectory as the acceleration limits are approached. Performance is illustrated by means of simulations and experiments on a UR10 robot.

Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory samples with partially-annotated captions. The model learns the meaning of each of the words without direct per-word supervision. At inference time, it generates a linguistic description of trajectories which captures maneuvers and interactions over an extended time interval. This generated description is used to refine predictions of the trajectories of multiple agents. We train and validate our model on the Argoverse dataset, and demonstrate improved accuracy results in trajectory prediction. In addition, our model is more interpretable: it presents part of its reasoning in plain language as captions, which can aid model development and can aid in building confidence in the model before deploying it.

State-of-the-art autonomous driving systems rely on high definition (HD) maps for localization and navigation. However, building and maintaining HD maps is time-consuming and expensive. Furthermore, the HD maps assume structured environment such as the existence of major road and lanes, which are not present in rural areas. In this work, we propose an end-to-end transformer networks based approach for map-less autonomous driving. The proposed model takes raw LiDAR data and noisy topometric map as input and produces precise local trajectory for navigation. We demonstrate the effectiveness of our method in real-world driving data, including both urban and rural areas. The experimental results show that the proposed method outperforms state-of-the-art multimodal methods and is robust to the perturbations of the topometric map. The code of the proposed method is publicly available at https://github.com/Jiaolong/trajectory-prediction.

Heterogeneous robotic systems in the field often encounter bodies of water with unknown traversability properties. One approach to measuring depth, current, soil composition, etc. is via an in situ underwater sensor being dragged by cable attached to a maneuvering airborne multicopter - which entails a novel motion planning and control problem with mixed resistive media. In this work we propose a framework to plan trajectories for future characterization sensors and payloads moving through mixed (air-water) media while considering uncertainty in the depth of the underwater ground surface. The methodology is applied to example underactuated systems with suspended payloads of increasing levels of complexity, including a cable robot and 4- and 8-DOF multicopter systems. Simulation studies employing trajectory optimization indicate that under certain payload configurations and task constraints, there are maneuvers in which it is more efficient to drag the payloads through water than through air. The paper also includes preliminary experiments with a testbed cable robot platform.

Tracks, wheels, and legs are all useful locomotion modes for Unmanned Ground Vehicles (UGVs), and ground robots that combine these mechanisms have the potential to climb over large obstacles. As robot morphologies include more degrees of freedom and obstacles become increasingly large and complex, UGVs will need to rely on automatic motion planning to compute the joint trajectories for traversal. This article presents a trajectory optimization formulation for multibody UGVs with combined wheel-leg and track-leg designs. We derive the dynamics and constraints for rolling wheels and circulating elliptical tracks. Using direct collocation, we formulate a model-based trajectory optimization where all constraints and objectives are written in closed-form with smooth and exact derivatives for tractable computation times with existing large-scale nonlinear optimization solvers (<1 minute). We demonstrate the trajectory optimization on numerous simulated planar wheel-leg and track-leg morphologies completing locomotion tasks, demonstrating full body dynamic coupling for the multibody system. Future work will extend this formulation to 3D and include contact planning.

This paper presents a novel control approach for autonomous systems operating under uncertainty. We combine Model Predictive Path Integral (MPPI) control with Covariance Steering (CS) theory to obtain a robust controller for general nonlinear systems. The proposed Covariance-Controlled Model Predictive Path Integral (CC-MPPI) controller addresses the performance degradation observed in some MPPI implementations owing to unexpected disturbances and uncertainties. Namely, in cases where the environment changes too fast or the simulated dynamics during the MPPI rollouts do not capture the noise and uncertainty in the actual dynamics, the baseline MPPI implementation may lead to divergence. The proposed CC-MPPI controller avoids divergence by controlling the dispersion of the rollout trajectories at the end of the prediction horizon. Furthermore, the CC-MPPI has adjustable trajectory sampling distributions that can be changed according to the environment to achieve efficient sampling. Numerical examples using a ground vehicle navigating in challenging environments demonstrate the proposed approach.

Detection of rare objects (e.g., traffic cones, traffic barrels and traffic warning triangles) is an important perception task to improve the safety of autonomous driving. Training of such models typically requires a large number of annotated data which is expensive and time consuming to obtain. To address the above problem, an emerging approach is to apply data augmentation to automatically generate cost-free training samples. In this work, we propose a systematic study on simple Copy-Paste data augmentation for rare object detection in autonomous driving. Specifically, local adaptive instance-level image transformation is introduced to generate realistic rare object masks from source domain to the target domain. Moreover, traffic scene context is utilized to guide the placement of masks of rare objects. To this end, our data augmentation generates training data with high quality and realistic characteristics by leveraging both local and global consistency. In addition, we build a new dataset named NM10k consisting 10k training images, 4k validation images and the corresponding labels with a diverse range of scenarios in autonomous driving. Experiments on NM10k show that our method achieves promising results on rare object detection. We also present a thorough study to illustrate the effectiveness of our local-adaptive and global constraints based Copy-Paste data augmentation for rare object detection. The data, development kit and more information of NM10k dataset are available online at: https://nullmax-vision.github.io.

We propose a method for robotic control of deformable objects using a learned nonlinear dynamics model. After collecting a dataset of trajectories from the real system, we train a recurrent neural network (RNN) to approximate its input-output behavior with a latent state-space model. The RNN internal state is low-dimensional enough to enable realtime nonlinear control methods. We demonstrate a closed-loop control scheme with the RNN model using a standard nonlinear state observer and model-predictive controller. We apply our method to track a highly dynamic trajectory with a point on the deformable object, in real time and on real hardware. Our experiments show that the RNN model captures the true system's frequency response and can be used to track trajectories outside the training distribution. In an ablation study, we find that the full method improves tracking accuracy compared to an open-loop version without the state observer.

Surgical instrument segmentation - in general a pixel classification task - is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Track-to-Segment Transformer that wisely exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions i.e., a set of class-bbox-mask pairs, by decoding query embeddings. Specifically, we introduce the prior query that encoded with previous temporal knowledge, to transfer tracking signals to current instances via identity matching. A contrastive query learning strategy is further applied to reshape the query feature space, which greatly alleviates the tracking difficulty caused by large temporal variations. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets, including two RAS benchmarks from EndoVis Challenges and one cataract surgery dataset CaDIs.

Soft robotics has seen an exponential growth in the past decade, in part because the transition to soft materials has made a wider range of applications possible. Tasks involving contact with fragile objects or unstructured environments are particularly amenable to devices based on soft materials. To date, research has primarily focused on the development of soft analogs to traditional sensors and actuators while controllers for soft robots have tended to rely on common rigid electronic components. We aspire to create a library of elastomer-based devices that can evolve soft controllers beyond component-level demonstrations and towards system-level completeness taking inspiration from electronic microcontrollers. Our approach combines microfluidic circuit designs with soft robotic fabrication techniques to create fluidic microcontroller components that are composed of soft materials and are of minimal size. We have identified the shift register, oscillator, and demultiplexer as key circuit elements for both individual functionality and for multi-component systems that can mimic microcontroller behaviors. In this paper, we present a review of fluidic circuits, fabrication processes, and implementation of these circuits into soft robotic platforms. In this work, we demonstrate a shift register, demultiplexer, and oscillator. They contain characteristics such as memory storage, data communication, and timing capabilities.