ICRA 2022 论文列表

Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific noise, which might worsen the features along with the propagation of the network. Therefore, this work proposes an effective and efficient cross-modality fusion module called Bi-directional Adaptive Attention Gate (BAA-Gate). Based on the attention mechanism, the BAA-Gate is devised to distill the informative features and recalibrate the representations asymptotically. Concretely, a bi-direction multi-stage fusion strategy is adopted to progressively optimize features of two modalities and retain their specificity during the propagation. Moreover, an adaptive interaction of BAA-Gate is introduced by the illumination-based weighting strategy to adaptively adjust the recalibrating and aggregating strength in the BAA-Gate and enhance the robustness towards illumination changes. Considerable experiments on the challenging KAIST dataset demonstrate the superior performance of our method with satisfactory speed.

Future Moon bases will likely be constructed using resources mined from the surface of the Moon. The difficulty of maintaining a human workforce on the Moon and communications lag with Earth means that mining will need to be conducted using collaborative robots with a high degree of autonomy. In this paper, we describe our solution for Phase 2 of the NASA Space Robotics Challenge, which provided a simulated lunar environment in which teams were tasked to develop software systems to achieve autonomous collaborative robots for mining on the Moon. Our 3rd place and innovation award winning solution shows how machine learning-enabled vision could alleviate major challenges posed by the lunar environment towards autonomous space mining, chiefly the lack of satellite positioning systems, hazardous terrain, and delicate robot interactions. A robust multi-robot coordinator was also developed to achieve long-term operation and effective collaboration between robots11A recording of our robots in action is available at [1]..

This paper presents an integrated motion planning system for autonomous vehicle (AV) parking in the presence of other moving vehicles. The proposed system includes 1) a hybrid environment predictor that predicts the motions of the surrounding vehicles and 2) a strategic motion planner that reacts to the predictions. The hybrid environment predictor performs short-term predictions via an extended Kalman filter and an adaptive observer. It also combines short-term predictions with a driver behavior cost-map to make long-term predictions. The strategic motion planner comprises 1) a model predictive control-based safety controller for trajectory tracking; 2) a search-based retreating planner for finding an evasion path in an emergency; 3) an optimization-based repairing planner for planning a new path when the original path is invalidated. Simulation validation demonstrates the effectiveness of the proposed method in terms of initial planning, motion prediction, safe tracking, retreating in an emergency, and trajectory repairing.

Ultrasound scanning is an imaging technique that aids medical professionals in diagnostics and interventional procedures. However, a trained human-in-the-loop (HITL) with a radiologist is required to perform the scanning procedure. We seek to create a novel ultrasound system that can provide imaging in the absence of a trained radiologist, say for patients in the field who suffered injuries after a natural disaster. One challenge of automating ultrasound scanning involves finding the optimal area to scan and then performing the actual scan. This task requires simultaneously maintaining contact with the surface while moving along it to capture high quality images. In this work, we present an automated Robotic Ultrasound System (RUS) to tackle these challenges. Our approach introduces a Bayesian Optimization framework to guide the probe to multiple points on the unknown surface. Our proposed framework collects the ultrasound images as well as the pose information at every probed point to estimate regions with high vessel density (information map) and the surface contour. Based on the information map and the surface contour, an area of interest is selected for scanning. Furthermore, to scan the proposed region, a novel 6-axis hybrid force-position controller is presented to ensure acoustic coupling. Lastly, we provide experimental results on two different phantom models to corroborate our approach.

This paper presents a novel strategy for autonomous teamed exploration of subterranean environments using legged and aerial robots. Tailored to the fact that subterranean settings, such as cave networks and underground mines, often involve complex, large-scale and multi-branched topologies, while wireless communication within them can be particularly challenging, this work is structured around the synergy of an onboard exploration path planner that allows for resilient long-term autonomy, and a multi-robot coordination framework. The onboard path planner is unified across legged and flying robots and enables navigation in environments with steep slopes, and diverse geometries. When a communication link is available, each robot of the team shares submaps to a centralized location where a multi-robot coordination framework identifies global frontiers of the exploration space to inform each system about where it should re-position to best continue its mission. The strategy is verified through a field deployment inside an underground mine in Switzerland using a legged and a flying robot collectively exploring for 45 min, as well as a longer simulation study with three systems.

This paper presents a novel planning and control strategy for competing with multiple vehicles in a car racing scenario. The proposed racing strategy switches between two modes. When there are no surrounding vehicles, a learning-based model predictive control (MPC) trajectory planner is used to guarantee that the ego vehicle achieves better lap timing performance. When the ego vehicle is competing with other surrounding vehicles to overtake, an optimization-based planner generates multiple dynamically-feasible trajectories through parallel computation. Each trajectory is optimized under a MPC formulation with different homotopic Bezier-curve reference paths lying laterally between surrounding vehicles. The time-optimal trajectory among these different homotopic trajectories is selected and a low-level MPC controller with control barrier function constraints for obstacle avoidance is used to guarantee the system's safety-critical performance. The proposed algorithm has the capability to generate collision-free trajectories and track them while enhancing the lap timing performance with steady low computational complexity, outper-forming existing approaches in both timing and performance for an autonomous racing environment. To demonstrate the performance of our racing strategy, we simulate with multiple randomly generated moving vehicles on the track and test the ego vehicle's overtaking maneuvers.

Autonomous Exploration Development Environment is an open-source repository released to facilitate development of high-level planning algorithms and integration of com-plete autonomous navigation systems. The repository contains representative simulation environment models, fundamental navigation modules, e.g., local planner, terrain traversability analysis, waypoint following, and visualization tools. Together with two of our high-level planner releases - TARE planner for exploration and FAR planner for route planning, we detail usage of the three open-source repositories and share experiences in integration of autonomous navigation systems. We use DARPA Subterranean Challenge as a use case where the repositories together form the main navigation system of the CMU-OSU Team. In the end, we discuss a few potential use cases in extended applications.

In most instances, flapping wing robots have emulated the “synchronous” actuation of insects in which the wingbeat timing is generated from a time-dependent, rhythmic signal. The internal dynamics of asynchronous insect flight muscle enable high-frequency, adaptive wingbeats with minimal direct neural control. In this paper, we investigate how the delayed stretch-activation (dSA) response of asynchronous insect flight muscle can be transformed into a feedback control law for flapping wing robots that results in stable limit cycle wingbeats. We first demonstrate - in theory and simulation - the mechanism by which asynchronous wingbeats self-excite. Then, we implement the feedback law on a dynamically-scaled robophysical model as well as on an insect-scale robotic flapping wing. Experiments on large- and small-scale robots demonstrate good agreement with the theory results and highlight how dSA parameters govern wingbeat amplitude and frequency. Lastly, we demonstrate that asynchronous actuation has several advantages over synchronous actuation schemes, including the ability to rapidly adapt or halt wingbeats in response to external loads or collisions through low-level feedback control.

The automatic generation of (dis)assembly sequences for complex technical products is a challenging field. Complex products like vehicles consist of numerous different components. Determining the sequence using a brute-force-approach by testing all components for disassembly one after another in a loop until all components are disassembled is laborious and costly. In industrial scenarios, a large proportion of the components are fasteners. In this paper, we propose a new framework which improves the disassembly sequencing generation by prioritizing fasteners during planning. Our proposed framework comprises a preprocessing in which fasteners are identified with a convolutional neural network within a dataset and a procedure that preferentially and automatically checks fasteners for disassembly. The algorithm takes initial and unavoidable collisions of the fasteners into account. We show the effectiveness of our approach on real-world data from the automotive industry. A new synthetic dataset of fasteners for training neural networks is available.

Automatic recognition of surgical workflow is a growing area of interest with significant potential to become part of context-aware decision-support systems in future enhanced ORs and clinical suites. Applications range from post-operative analysis to intra-operative monitoring to providing automated assistance to the clinical staff. This work proposes, for the first time, automatic tool presence and episodes recognition in bronchoscopy images, using a newly annotated video dataset obtained from robotic bronchoscopy procedures. A novel multi-task architecture that utilizes Vision Transformers for the episode recognition task is used to achieve improved accuracy while reducing the computational burden during the training stage. The method is thoroughly validated on clinical procedure video data obtained across 135 procedures, displaying improved performance and training times compared to various state-of-the-art methods.

Grasping a particular object may require a dedicated grasping movement that may also be specific to the robot end-effector. No generic and autonomous method does exist to generate these movements without making hypotheses on the robot or on the object. Learning methods could help to autonomously discover relevant grasping movements, but they face an important issue: grasping movements are so rare that a learning method based on exploration has little chance to ever observe an interesting movement, thus creating a bootstrap issue. We introduce an approach to generate diverse grasping movements in order to solve this problem. The movements are generated in simulation, for particular object positions. We test it on several simulated robots: Baxter, Pepper and a Kuka Iiwa arm. Although we show that generated movements actually work on a real Baxter robot, the aim is to use this method to create a large dataset to bootstrap deep learning methods.

Many robot control scenarios involve assessing system robustness against a task specification. If either the controller or environment are composed of “black-box” components with unknown dynamics, we cannot rely on formal verification to assess our system. Assessing robustness via exhaustive testing is also often infeasible if the number of possible environments is large compared to experiment cost. Given limited budget, we provide a method to choose experiment inputs which accurately reflect how robustly a system satisfies a given specification across the domain. By combining signal temporal logic metrics with adaptive experiment design, our method chooses each experiment by incrementally constructing a surrogate model of the specification robustness. This model then chooses experiments in areas of either high prediction error or high uncertainty. Our evaluation shows how this adaptive experiment design results in sample-efficient descriptions of system robustness. Further, we show how to use the constructed surrogate model to assess the behaviour of a data-driven control system under domain shift.

Given a heterogeneous group of robots executing a complex task represented in Linear Temporal Logic, and a new set of tasks for the group, we define the task update problem and propose a framework for automatically updating individual robot tasks given their respective existing tasks and capabilities. Our heuristic, token-based, conflict resolution task allocation algorithm generates a near-optimal assignment for the new task. We demonstrate the scalability of our approach through simulations of multi-robot tasks.

Recent advances in medical technology have produced a number of flexible instruments that are capable of traversing non-linear paths. This is of special interest in the field of neurosurgery. However, the non-rigid instruments have the disadvantage that path planning becomes increasingly difficult. In addition to anatomical risk factors, the mechanical properties and constraints of the specific instrument must also be considered. To support surgeons to deal with the increase in planning complexity, we present a novel method for both linear and arbitrary follow-the-leader flexible path planning. Our method is utilizing patient-specific image data, which is then used to generate a multi-objective problem consisting of conventional risk metrics for path planning in high-risk regions like the accumulated path cost or the distance to risk structures. Simultaneously, the path-problem is also constraint to mechanical properties of the instrument such as curvature or maximum operational length. Optimal paths can then be generated by solving a multi-objective problem by approximating the Pareto front. We show that our method can automatically generate linear and non-linear paths for neurosurgical interventions in the human brain in less than 2 minutes. Furthermore, we show that the proposed automated method generates paths with 87% reduced risk compared to standard of care plannings.

This paper presents a novel place recognition approach to autonomous vehicles by using low-cost, single-chip automotive radar. Aimed at improving recognition robustness and fully exploiting the rich information provided by this emerging automotive radar, our approach follows a principled pipeline that comprises (1) dynamic points removal from instant Doppler measurement, (2) spatial-temporal feature embedding on radar point clouds, and (3) retrieved candidates refinement from Radar Cross Section measurement. Extensive experimental results on the public nuScenes dataset demonstrate that existing visual/LiDAR/spinning radar place recognition approaches are less suitable for single-chip automotive radar. In contrast, our purpose-built approach for automotive radar consistently outperforms a variety of baseline methods via a comprehensive set of metrics, providing insights into the efficacy when used in a realistic system.

Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces Manipulation Primitive-augmented reinforcement Learning (MAPLE), a learning framework that augments standard reinforcement learning algorithms with a pre-defined library of behavior primitives. These behavior primitives are robust functional modules specialized in achieving manipulation goals, such as grasping and pushing. To use these heterogeneous primitives, we develop a hierarchical policy that involves the primitives and instantiates their executions with input parameters. We demonstrate that MAPLE out-performs baseline approaches by a significant margin on a suite of simulated manipulation tasks. We also quantify the compositional structure of the learned behaviors and highlight our method's ability to transfer policies to new task variants and to physical hardware. Videos and code are available at https://ut-austin-rpl.github.io/maple

The robustness of visual navigation policies trained through imitation often hinges on the augmentation of the training image-action pairs. Traditionally, this has been done by collecting data from multiple cameras, by using standard data augmentations from computer vision, such as adding random noise to each image, or by synthesizing training images. In this paper we show that there is another practical alternative for data augmentation for visual navigation based on extrapolating viewpoint embeddings and actions nearby the ones observed in the training data. Our method makes use of the geometry of the visual navigation problem in 2D and 3D and relies on policies that are functions of equivariant embeddings, as opposed to images. Given an image-action pair from a training navigation dataset, our neural network model predicts the latent representations of images at nearby viewpoints, using the equivariance property, and augments the dataset. We then train a policy on the augmented dataset. Our simulation results indicate that policies trained in this way exhibit reduced cross-track error, and require fewer interventions compared to policies trained using standard augmentation methods. We also show similar results in autonomous visual navigation by a real ground robot along a path of over 500m.

With recent advancements in CV (computer vision) and AI (Artificial Intelligence) technologies, pointing gesture is becoming an emerging trend for human-robot interaction. Its intuitive and deictic nature makes it an ideal way for giving commands, especially referring spatial information to the robots. In this paper, we propose an augmented pointing gesture estimation method to enable richer and programmable instructions to be given to the robots. We propose five pointing gestures and demonstrate the idea using a collaborative robot with a multi-finger robotic gripper. Experiments are designed and conducted to test the pointing accuracy in space and in gesture estimation. The results show that our proposed method can achieve a mean drift of 8.3 cm and an estimation accuracy of 94.08%.

Referring expressions are commonly used when referring to a specific target in people's daily dialogue. In this paper, we develop a novel task of audio-visual grounding referring expression for robotic manipulation. The robot leverages both the audio and visual information to understand the referring expression in the given manipulation instruction and the corresponding manipulations are implemented. To solve the proposed task, an audio-visual framework is proposed for visual localization and sound recognition. We have also established a dataset which contains visual data, auditory data and manipulation instructions for evaluation. Finally, extensive experiments are conducted both offline and online to verify the effectiveness of the proposed audio-visual framework. And it is demonstrated that the robot performs better with the audio-visual data than with only the visual data.