ICRA 2023 Papers

Diverse and realistic traffic scenarios are crucial for evaluating the AI safety of autonomous driving systems in simulation. This work introduces a data-driven method called TrafficGen for traffic scenario generation. It learns from the fragmented human driving data collected in the real world and then generates realistic traffic scenarios. TrafficGen is an autoregressive neural generative model with an encoder-decoder architecture. In each autoregressive iteration, it first encodes the current traffic context with the attention mechanism and then decodes a vehicle's initial state followed by generating its long trajectory. We evaluate the trained model in terms of vehicle placement and trajectories, and the experimental result shows our method has substantial improvements over baselines for generating traffic scenarios. After training, TrafficGen can also augment existing traffic scenarios, by adding new vehicles and extending the fragmented trajectories. We further demonstrate that importing the generated scenarios into a simulator as an interactive training environment improves the performance and safety of a driving agent learned from reinforcement learning. Model and data are available at https://metadriverse.github.io/trafficgen.

Data-driven simulation has become a favorable way to train and test autonomous driving algorithms. The idea of replacing the actual environment with a learned simulator has also been explored in model-based reinforcement learning in the context of world models. In this work, we show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving, and based on TrafficBots we obtain a world model tailored for the planning module of autonomous vehicles. Existing data-driven traffic simulators are lacking configurability and scalability. To generate configurable behaviors, for each agent we introduce a destination as nav-igational information, and a time-invariant latent personality that specifies the behavioral style. To improve the scalability, we present a new scheme of positional encoding for angles, allowing all agents to share the same vectorized context and the use of an architecture based on dot-product attention. As a result, we can simulate all traffic participants seen in dense urban scenarios. Experiments on the Waymo open motion dataset show TrafficBots can simulate realistic multi-agent behaviors and achieve good performance on the motion prediction task.

While there have been advancements in autonomous driving control and traffic simulation, there have been little to no works exploring their unification with deep learning. Works in both areas seem to focus on entirely different exclusive problems, yet traffic and driving are inherently related in the real world. In this paper, we present Traffic-Aware Autonomous Driving (TrAAD), a generalizable distillation-style method for traffic-informed imitation learning that directly optimizes for faster traffic flow and lower energy consumption. TrAAD focuses on the supervision of speed control in imitation learning systems, as most driving research focuses on perception and steering. Moreover, our method addresses the lack of co-simulation between traffic and driving simulators and provides a basis for directly involving traffic simulation with autonomous driving in future work. Our results show that, with information from traffic simulation involved in the supervision of imitation learning methods, an autonomous vehicle can learn how to accelerate in a fashion that is beneficial for traffic flow and overall energy consumption for all nearby vehicles.

This paper proposes a new model for onboard physical fault detection on autonomous unmanned aerial vehicles (UAV) through machine learning (ML) techniques. The proposal performs the detection task with high accuracies and minimal processing requirements while signaling an unreliable ML model to the operator, implemented in two main phases. First, a wrapper-based feature selection is performed to de-crease the feature extraction computational costs, coped with a classification assessment technique to identify ML model unreliability. Second, physical UAV faults are signaled through a multi-view rationale that evaluates a variety of UAV sensors while triggering alerts based on a sliding window scheme. Experiments performed on a real quadcopter UAV with a broken propeller use case shows the proposal's feasibility. Our model can decrease the false-positive rates up to only 0.4%, while also decreasing the computational costs by at least 43 % when compared to traditional techniques. Notwithstanding, it can identify ML model unreliability, signaling the UAV operator when model fine-tuning is needed.

While lower limb exoskeletons for above-ground locomotion have been emerging, few attempts have been made to develop an exoskeleton to augment human swimming. Such efforts are hindered by a lack of knowledge surrounding the kinematics and kinetics of human swimming. This paper presents the design of a robotic platform to be used as a finned swimming testbed; describes a controller to generate finned swimming movement; and presents experiments and associated experimental results conducted to explore thrust production resulting from a flutter kick swimming motion.

Visual classification under uncertainty is a complex computer vision problem. We present a thorough comparison of several variants of convolutional neural network (CNN) classification techniques in the context of ambiguous image data interpretation. We explore possible improvements in classification accuracy achieved by insertion of prior ambiguity information during the annotation process. This enables us to harness known similarities between individual classes and use them as probability distributions for soft ground-truth labels. We also present an approach based on Bayesian CNNs, offering the possibility of further interpretation of classification results in a problem where the neural network model is often considered as a black box. The presented techniques are verified on a practical spot weld inspection problem.

Loop detection plays a key role in visual Si-multaneous Localization and Mapping (SLAM) by correcting the accumulated pose drift. In indoor scenarios, the richly distributed semantic landmarks are view-point invariant and hold strong descriptive power in loop detection. The current semantic-aided loop detection embeds the topology between semantic instances to search a loop. However, current semantic-aided loop detection methods face challenges in dealing with ambiguous semantic instances and drastic viewpoint differences, which are not fully addressed in the literature. This paper introduces a novel loop detection method based on an incremen-tally created scene graph, targeting the visual SLAM at indoor scenes. It jointly considers the macro-view topology, micro-view topology, and occupancy of semantic instances to find correct correspondences. Experiments using handheld RGB-D sequence show our method is able to accurately detect loops in drastically changed viewpoints. It maintains a high precision in observing objects with similar topology and appearance. Our method also demonstrates that it is robust in changed indoor scenes.

Path planning and localization in low-light and inclement weather conditions are critical problems facing autonomous vehicle systems. Our proposed method applies a single modality, millimetre-wave radar perception system for the detection of roadside retro-reflectors. Radar-based perception tasks can be challenging to perform due to the sparse and noisy nature of radar data. We propose the use of an unsupervised learning approach for filtering radar point clouds through Density-Based Spatial Clustering of Applications with Noise (DBSCAN). The DBSCAN algorithm segments retro-reflector points from noise points, thus providing the autonomous vehicle with a predicted path for the road ahead. We tested the approach via indoor experiments that make use of Continental's ARS 408 radar, a mobile Husky A2000 robot, and a Vicon motion capture system for ground truth validation. The experimental results of the proposed system demonstrated a classification accuracy of 84.13 % and F1 score of 83.71 %.

Communication enables agents to cooperate to achieve their goals. Learning when to communicate, i.e., sparse (in time) communication, and whom to message is particularly important when bandwidth is limited. However, recent work in learning sparse individualized communication suffers from high variance during training, where decreasing communication comes at the cost of decreased reward, particularly in cooperative tasks. We use the information bottleneck to reframe sparsity as a representation learning problem, which we show naturally enables lossless sparse communication at lower budgets than prior art. In this paper, we propose a method for true lossless sparsity in communication via Information Maximizing Gated Sparse Multi-Agent Communication (IMGS-MAC). Our model uses two individualized regularization objectives, an information maximization autoencoder and sparse communication loss, to create informative and sparse communication. We evaluate the learned communication ‘language’ through direct causal analysis of messages in non-sparse runs to determine the range of lossless sparse budgets, which allow zero-shot sparsity, and the range of sparse budgets that will inquire a reward loss, which is minimized by our learned gating function with few-shot sparsity. To demonstrate the efficacy of our results, we experiment in cooperative multi-agent tasks where communication is essential for success. We evaluate our model with both continuous and discrete messages. We focus our analysis on a variety of ablations to show the effect of message representations, including their properties, and lossless performance of our model.

Manual labeling of gestures in robot-assisted surgery is labor intensive, prone to errors, and requires expertise or training. We propose a method for automated and explainable generation of gesture transcripts that leverages the abundance of data for image segmentation. Surgical context is detected using segmentation masks by examining the distances and intersections between the tools and objects. Next, context labels are translated into gesture transcripts using knowledge-based Finite State Machine (FSM) and data-driven Long Short Term Memory (LSTM) models. We evaluate the performance of each stage of our method by comparing the results with the ground truth segmentation masks, the consensus context labels, and the gesture labels in the JIGSAWS dataset. Our results show that our segmentation models achieve state-of-the-art performance in recognizing needle and thread in Suturing and we can automatically detect important surgical states with high agreement with crowd-sourced labels (e.g., contact between graspers and objects in Suturing). We also find that the FSM models are more robust to poor segmentation and labeling performance than LSTMs. Our proposed method can significantly shorten the gesture labeling process (~2.8 times).

Real-time remote manipulation requires careful operations by a user to ensure the safety of a robot, which is designed to follow user's commands, against dynamic obstacles. However, a user may give commands to a robot at the risk of collision with dynamic obstacles due to a user's unfamiliar control ability or unexpected situations. In this paper, we propose a risk-aware user command adjustment method to avoid potential collision with dynamic obstacles. Our method consists of a network that predicts the risk of dynamic obstacles and another network that synthesizes commands to avoid obstacles. Based on the predicted risk, our method decides an adjusted command between a user command and a command to avoid collisions. We evaluate our method in problems that face collisions with dynamic obstacles when following given commands and in problems with static obstacles. We show that our method improves safety against the risk of dynamic obstacles or follows user commands when there is no risk. We also demonstrate the feasibility of our method using the real fetch manipulator with seven-degrees-of-freedom.

Falling cat problem is well-known where cats show their super aerial reorientation capability and can land safely. For their robotic counterparts, a similar falling quadruped robot problem, has not been fully addressed, although achieving safe landing as the cats has been increasingly investigated. Unlike imposing the burden on landing control, we approach to safe landing of falling quadruped robots by effective flight phase control. Different from existing work like swinging legs and attaching reaction wheels or simple tails, we propose to deploy a 3-DoF morphable inertial tail on a medium-size quadruped robot. In the flight phase, the tail with its maximum length can self-right the body orientation in 3D effectively; before touch-down, the tail length can be retracted to about 1/4 of its maximum for impressing the tail's side-effect on landing. To enable aerial reorientation for safe landing in the quadruped robots, we design a control architecture, which is verified in a high-fidelity physics simulation environment with different initial conditions. Experimental results on a customized flight-phase test platform with comparable inertial properties are provided and show the tail's effectiveness on 3D body reorientation and its fast retractability before touch-down. An initial falling quadruped robot experiment is shown, where the robot Unitree A1 with the 3-DoF tail can land safely subject to non-negligible initial body angles.

With the rapid development of autonomous driving, the need for auto-labeling reference systems is becoming increasingly urgent. 3D multiple object tracking (MOT) is one of the most critical components of the reference system. In this work, we reviewed and rethought the common failure sources and limitations of the SOTA 3D MOT methods. We propose a set of innovative 3D MOT post-processing modules as a unified framework based on the observation. First, we design a self-learning-based detector to eliminate the outliers in each tracklet. Then a novel post-processing module, GGTrajRec, will recover the breakpoints and ID switches in the trajectories. Finally, a confidence-guided trajectory optimizer is implemented to ensure each trajectory's consistency. Extensive experiments on KITTI and nuScenes show that our method can improve the SOTA methods on most evaluation metrics by a remarkable margin. Currently, our results are second ranking on the KITTI tracking leaderboard. Specifically, our method offers the lowest FPs, highest DetRe, and AssRe values among all methods, which can significantly contribute to a stable and robust reference system for ADAS.

Modern robotic manipulation systems fall short of human manipulation skills partly because they rely on closing feedback loops exclusively around vision data, which reduces system bandwidth and speed. By developing autonomous grasping reflexes that rely on high-bandwidth force, contact, and proximity data, the overall system speed and robustness can be increased while reducing reliance on vision data. We are developing a new system built around a low-inertia, high-speed arm with nimble fingers that combines a high-level trajectory planner operating at less than 1 Hz with low-level autonomous reflex controllers running upwards of 300 Hz. We characterize the reflex system by comparing the volume of the set of successful grasps for a naive baseline controller and variations of our reflexive grasping controller, finding that our controller expands the set of successful grasps by 55% relative to the baseline. We also deploy our reflexive grasping controller with a simple vision-based planner in an autonomous clutter clearing task, achieving a grasp success rate above 90% while clearing over 100 items.

When humans interact with robots influence is inevitable. Consider an autonomous car driving near a human: the speed and steering of the autonomous car will affect how the human drives. Prior works have developed frameworks that enable robots to influence humans towards desired behaviors. But while these approaches are effective in the short-term (i.e., the first few human-robot interactions), here we explore long-term influence (i.e., repeated interactions between the same human and robot). Our central insight is that humans are dynamic: people adapt to robots, and behaviors which are influential now may fall short once the human learns to anticipate the robot's actions. With this insight, we experimentally demonstrate that a prevalent game-theoretic formalism for generating influential robot behaviors becomes less effective over repeated interactions. Next, we propose three modifications to Stackelberg games that make the robot's policy both influential and unpredictable. We finally test these modifications across simulations and user studies: our results suggest that robots which purposely make their actions harder to anticipate are better able to maintain influence over long-term interaction. See videos here: https://youtu.be/ydO83cgjZ2Q

A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (US) technologies to analyze muscle states. While estimating finger gestures has been done in the past by detecting prominent gestures, we are interested in detection, or inference, done in the context of fine motions that evolve over time. Examples include motions occurring when performing fine and dexterous tasks such as keyboard typing or piano playing. We consider this task as an important step towards higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. To this end, we present an end-to-end robotic system, which can successfully infer fine finger motions. This is achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from a sequence of US images. We evaluated our method by collecting data from a group of subjects and demonstrating how it can be used to replay music played or text typed. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.

Language-based communications are essential in human-robot interaction, especially for the majority of non-expert users. In this paper, we present SeeAsk, an open-world interactive visual grounding system to grasp specified targets with ambiguous natural language instructions. The main contribution of SeeAsk is that it can robustly handle open-world scenes in terms of both open-set objects and open-vocabulary interactions. Specifically, our SeeAsk is built upon modern large-scale vision-language pre-trained models and traditional decision-making process, and shows promising results to be deployed in real-world scenarios. SeeAsk outperforms previous state-of-the-art algorithms with a clear margin in terms of not only success rate but also asking smarter and more informative questions. User studies also demonstrate its advantages over previous works.

The texture recognition can provide clues for robots to interact with the external environment. The traditional tactile material recognition task is studied under the close-set assumption, which means that all types of materials are included in the training set. However, the open-set materials recognition for robots is of much greater significance because in the real-world applications, there is usually something that doesn't belong to any known class. Up to now, there is no researcher to further the discussion of this problem. To cope with unknown classes, this study proposes the Open set Material Recognition (OpenMR) based on General Convolutional Prototype Learning (GCPL). To handle the open space risk for GCPL caused by the lack of unknown samples in the training stage, we use Generative Adversarial Networks (GAN) to synthesize open-set samples as unknowns. The proposed framework is implemented and tested on two batches of tactile data collected in different exploratory motions on 8 material textures using the electronic skin. Compared with other open-set classifiers, experiments reveal that the proposed framework achieves competitive performance in both known classification and unknown detection.

This works deals with the problem of long-term autonomy in the context of multi-day field deployments of Micro Aerial Vehicle (MAV) systems. To truly depart from the necessity for human intervention for the crucial task of providing battery recharging, and to liberate from the need to operate in a confined range around specially installed infrastructure such as recharging pods, the MAV robot is required to harvest power on its own, but equally importantly also sustain prolonged periods of ambient power scarcity. This implies being able to sustain the battery charge overnight when using solar recharging, or even during multiple days of illumination inadequacy (e.g., due to degraded atmospheric lucidity and heavy overcast). We address this by presenting a Self-Sustainable Autonomous System architecture for MAVs centered around a specially tailored Power Management Stack, which is capable of achieving deep system hibernation, a feature that facilitates the aforementioned functionalities. We present a) continuous, b) multi-day successive, and c) externally-powered recharging that uses a legged robot-mounted Mobile Recharging Station. We conclude by demonstrating a challenging zero-intervention multi-day field deployment mission in the N.Nevada region.