ICRA 2022 论文列表

Robots need object-level scene understanding to manipulate objects while reasoning about contact, support, and occlusion among objects. Given a pile of objects, object recognition and reconstruction can identify the boundary of object instances, giving important cues as to how the objects form and support the pile. In this work, we present a system, SafePicking, that integrates object-level mapping and learning-based motion planning to generate a motion that safely extracts occluded target objects from a pile. Planning is done by learning a deep Q-network that receives observations of predicted poses and a depth-based heightmap to output a motion trajectory, trained to maximize a safety metric reward. Our results show that the observation fusion of poses and depth-sensing gives both better performance and robustness to the model. We evaluate our methods using the YCB objects in both simulation and the real world, achieving safe object extraction from piles.

We study the problem of safe multi-agent motion planning in cluttered environments. Existing multi-agent reinforcement learning-based motion planners only provide approximate safety enforcement. We propose a safe reinforcement learning algorithm that leverages single-agent reinforcement learning for target regulation and a subsequent convex optimization-based filtering that ensures the collective safety of the system. Our approach yields a safe, real-time implementable multi-agent motion planner that is simpler to train and enforces safety as hard constraints. Our approach can handle state and control constraints on the agents, and enforce collision avoidance among themselves and with static obstacles in the environment. Numerical simulations and hardware experiments show the efficacy of the approach.

One of the major functions brought by robots in Minimally Invasive Surgery is endoscope holding. This consists, for the user, in placing the camera at a desired location which the robot will maintain still once he/she releases it. This behavior is usually achieved with rigid position servoing, leading to possibly high forces generated and safety issues. Model-based weight compensation is an alternative solution. However, endoscopic cameras' weight is difficult to model as their gravity parameters can change during the same surgery. In this paper, an algorithm is presented as an option to cope with this variability in the gravity model without using rigid position servoing. The surgeon first positions the camera in a comanipulation mode (gravity compensation). When he/she releases the camera, if the gravity model is not accurate, the endoscope presents a drift. In this case, a controller brings the endoscope back to its release position by combining low gain position control and model adaptation. Once stabilized, the system is switched back to a zero-stiffness mode. Two in-vitro experiments were performed in which a user manipulates an endoscope whose configuration of mass is changed. In one case, the mass in the gravity model was set to half of the actual one. In the second case, a variable weight was attached to the endoscope. The algorithm successfully updated the model for each experiment reducing position errors by 95% and 57%, respectively.

Current safety mechanisms implementing industry standards for human-robot coexistence separate humans and robots through caging. Other approaches allowing humans to enter the workspace of manipulators do not provide formal safety guarantees. Thus, this study aims to facilitate the widespread adoption of collaborative robots by presenting SaRA, an extensible tool that performs set-based reachability analysis and formally guarantees safety. Our experimental results show that the set-based prediction of a human can be computed in a few microseconds, using SaRA, allowing for real-time consideration of many surrounding humans in an environment.

We present a motion planner for planning through space-time with dynamic obstacles, velocity constraints, and unknown arrival time. Our algorithm, Space-Time RRT*(ST-RRT*), is a probabilistically complete, bidirectional motion planning algorithm, which is asymptotically optimal with respect to the shortest arrival time. We experimentally evaluate ST-RRT* in both abstract (2D disk, 8D disk in cluttered spaces, and on a narrow passage problem), and simulated robotic path planning problems (sequential planning of 8DoF mobile robots, and 7DoF robotic arms). The proposed planner outperforms RRT-Connect and RRT* on both initial solution time, and attained final solution cost. The code for ST-RRT* is available in the Open Motion Planning Library (OMPL).

Road extraction is an essential step in building autonomous navigation systems. Detecting road segments is challenging as they are of varying widths, bifurcated throughout the image, and are often occluded by terrain, cloud, or other weather conditions. Using just convolution neural networks (ConvNets) for this problem is not effective as it is inefficient at capturing distant dependencies between road segments in the image which is essential to extract road connectivity. To this end, we propose a Spatial and Interaction Space Graph Reasoning (SPIN) module which when plugged into a ConvNet performs reasoning over graphs constructed on spatial and interaction spaces projected from the feature maps. Reasoning over spatial space extracts dependencies between different spatial regions and other contextual information. Reasoning over a projected interaction space helps in appropriate delineation of roads from other topographies present in the image. Thus, SPIN extracts long-range dependencies between road segments and effectively delineates roads from other semantics. We also introduce a SPIN pyramid which performs SPIN graph reasoning across multiple scales to extract multi-scale features. We propose a network based on stacked hourglass modules and SPIN pyramid for road segmentation which achieves better performance compared to existing methods. Moreover, our method is computationally efficient and significantly boosts the convergence speed during training, making it feasible for applying on large-scale high-resolution aerial images. Code available at: https://github.com/wgcban/SPIN_RoadMapper.git.

We present SMORS, the first Soft fully actuated MultirOtoR System for multimodal locomotion. Unlike conventional hexarotors, SMORS is equipped with three rigid and three continuously soft arms, with each arm hosting a propeller. We create a bridge between the fields of soft and aerial robotics by mechanically coupling the actuation of a fully actuated flying platform with the actuation of a soft robotic manipulator. Each rotor is slightly tilted, allowing for full actuation of the platform. The soft components combined with the platform's full actuation allow for a robust interaction, in the form of efficient multimodal locomotion. In this work, we present the dynamical model of the platform, derive a closed-loop control, and present simulation results fortifying the robustness of the platform under a jumping-flying maneuver. We demonstrate in simulations that our multimodal locomotion approach can be more energy-efficient than the flight with a hexarotor.

Panoptic segmentation aims to address semantic and instance segmentation simultaneously in a unified framework. However, an efficient solution of panoptic segmentation in applications like autonomous driving is still an open research problem. In this work, we propose a novel LiDAR-based panoptic system, called SMAC-Seg. We present a learnable sparse multi-directional attention clustering to segment multi-scale foreground instances. SMAC-Seg is a real-time clustering-based approach, which removes the complex proposal network to segment instances. Most existing clustering-based methods use the difference of the predicted and ground truth center offset as the only loss to supervise the instance centroid regression. However, this loss function only considers the centroid of the current object, but its relative position with respect to the neighbouring objects is not considered when learning to cluster. Thus, we propose to use a novel centroid-aware repel loss as an additional term to effectively supervise the network in order to differentiate each object cluster with its neighbours. Our experimental results show that SMAC-Seg achieves state-of-the-art performance among all real-time deployable networks on both large-scale public SemanticKITTI and nuScenes panoptic segmentation datasets.

Efficient exploration is a long-standing problem in reinforcement learning since extrinsic rewards are usually sparse or missing. A popular solution to this issue is to feed an agent with novelty signals as intrinsic rewards. In this work, we introduce SEMI, a self-supervised exploration policy by incentivizing the agent to maximize a new novelty signal: multisensory incongruity, which can be measured in two aspects, perception incongruity and action incongruity. The former represents the misalignment of the multisensory inputs, while the latter represents the variance of an agent's policies under different sensory inputs. Specifically, an alignment predictor is learned to detect whether multiple sensory inputs are aligned, the error of which is used to measure perception incongruity. A policy model takes different combinations of the multisensory observations as input, and outputs actions for exploration. The variance of actions is further used to measure action incongruity. Using both incongruities as intrinsic rewards, SEMI allows an agent to learn skills by exploring in a self-supervised manner without any external rewards. We further show that SEMI is compatible with extrinsic rewards and it improves sample efficiency of policy learning. The effectiveness of SEMI is demonstrated across a variety of benchmark environments including object manipulation and audio-visual games.

Depth completion refers to inferring the dense depth map from a sparse depth map with or without corre-sponding color image. Numerous neural networks have been proposed to accomplish this task. However, insufficient uti-lization of heteromorphic data and the fact that predicted dense depth prefers a sparse depth enormously damage the performance of approaches. To reduce data preference and fully utilize two modalities, this paper proposes a novel network that predicts high- and low-frequency components of dense depth separately. Specifically, the framework consists of a Low-Frequency(LF) branch and a High-Frequency(HF) branch. In the LF branch, we recover the low-frequency depth component from sparse depth through an Adaptive Graph-Generate Graph Attention Network, which can be seen as a low-pass filter. In the HF branch, we model the high-frequency component, e.g. boundaries, as residuals to mitigate the impact of data preferences. Moreover, in this branch, we propose an Attention-based Self-Fusion mechanism to efficiently fuse multi-scale features extracted from the sparse depth and color image. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on the KITTI benchmark and ranks 1st in root mean squared error among other published approaches.

In endoscopy, many applications (e.g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor graph optimization. The appearance and geometry priors are explicitly learned in an end-to-end differentiable training pipeline to master the task of pair-wise image alignment, one of the core components of the SLAM system. In our experiments, the proposed SLAM system is shown to robustly handle the challenges of texture scarceness and illumination variation that are commonly seen in endoscopy. The system generalizes well to unseen endoscopes and subjects and performs favorably compared with a state-of-the-art feature-based SLAM system. The code repository is available at https://github.com/lppllpp1920/SAGE-SLAM.git.

Building general-purpose robots to perform a diverse range of tasks in a large variety of environments in the physical world at the human level is extremely challenging. According to [1], it requires the robot learning to be sample-efficient, generalizable, compositional, and incremental. In this work, we introduce a systematic learning framework called SAGCI-system towards achieving these above four requirements. Our system first takes the raw point clouds gathered by the camera mounted on the robot's wrist as the inputs and produces initial modeling of the surrounding environment represented as a file of Unified Robot Description Format (URDF). Our system adopts a learning-augmented differentiable simulation that loads the URDF. The robot then utilizes the interactive perception to interact with the environment to online verify and modify the URDF. Leveraging the differentiable simulation, we propose a model-based learning algorithm combining object-centric and robot-centric stages to efficiently produce policies to accomplish manipulation tasks. We apply our system to perform articulated object manipulation tasks, both in the simulation and the real world. Extensive experiments demonstrate the effectiveness of our proposed learning framework. Supplemental materials and videos are available on our project webpage https://sites.google.com/view/egci.

Scene flow prediction is a challenging task that aims at jointly estimating the 3D structure and 3D motion of dynamic scenes. The previous methods concentrate more on point-wise estimation instead of considering the correspondence between objects as well as lacking the sensation of high-level semantic knowledge. In this paper, we propose a concise yet effective method for scene flow prediction. The key idea is to extend the view of all points for computing point cloud features into object-level, thus simultaneously modeling the relationships of the object-level and point-level via an improved transformer. In addition, we introduce a novel unsupervised loss called segmentation-aware loss, which can model semanticaware details to help predict scene flow more accurately and robustly. Since this loss can be trained without any ground truth, it can be used in both supervised training and self-supervised training. Experiments on both supervised training and self-supervised training demonstrate the effectiveness of our method. On supervised training, 3.8%, 22.58%, 10.90% and 21.82 % accuracy boosts than FLOT [23] can be observed on FT3Ds, KITTIs, FT3Do and KITTIo datasets. On self-supervised scheme, 48.23% and 48.96% accuracy boost than PointPWC-Net [40] can be observed on KITTIo and KITTIs datasets.

Providing safety guarantees for Autonomous Vehicle (AV) systems with machine-learning based controllers remains a challenging issue. In this work, we propose Simplex-Drive, a framework that can achieve runtime safety assurance for machine-learning enabled controllers of AVs. The proposed Simplex-Drive consists of an unverified Deep Reinforcement Learning (DRL)-based advanced controller (AC) that achieves desirable performance in complex scenarios, a Velocity-Obstacle (VO) based baseline safe controller (BC) with provably safety guarantees, and a verified mode management unit that monitors the operation status and switches the control authority between AC and BC based on safety-related conditions. We provide a formal correctness proof of Simplex-Drive and conduct a lane-changing case study in dense traffic scenarios. The simulation experiment results demonstrate that Simplex-Drive can always ensure the operation safety without sacrificing control performance, even if the DRL policy may lead to deviations from the safe status.

Despite significant developments in the design of surgical robots and automated techniques for objective evaluation of surgical skills, there are still challenges in ensuring safety in robot-assisted minimally-invasive surgery (RMIS). This paper presents a runtime monitoring system for the detection of executional errors during surgical tasks through the analysis of kinematic data. The proposed system incorporates dual Siamese neural networks and knowledge of surgical context, including surgical tasks and gestures, their distributional similarities, and common error modes, to learn the differences between normal and erroneous surgical trajectories from small training datasets. We evaluate the performance of the error detection using Siamese networks compared to single CNN and LSTM networks trained with different levels of contextual knowledge and training data, using the dry-lab demonstrations of the Suturing and Needle Passing tasks from the JIGSAWS dataset. Our results show that gesture specific task nonspecific Siamese networks obtain micro F1 scores of 0.94 (Siamese-CNN) and 0.95 (Siamese-LSTM), and perform better than single CNN (0.86) and LSTM (0.87) networks. These Siamese networks also outperform gesture nonspecific task specific Siamese-CNN and Siamese-LSTM models for Suturing and Needle Passing.

This paper studies a multi-robot visibility-based pursuit-evasion problem in which a group of pursuer robots are tasked with detecting an evader within a two dimensional polygonal environment. The primary contribution is a novel formulation of the pursuit-evasion problem that modifies the pursuers' objective by requiring that the evader still be de-tected, even in spite of the failure of any single pursuer robot. This novel constraint, whereby two pursuers are required to detect an evader, has the benefit of providing redundancy to the search, should any member of the team become unresponsive, suffer temporary sensor disruption/failure, or otherwise become incapacitated. Existing methods, even those that are designed to respond to failures, rely on the pursuers to replan and update their search pattern to handle such occurrences. In contrast, the proposed formulation produces plans that are inherently tolerant of some level of disturbance. Building upon this new formulation, we introduce an augmented data structure for encoding the problem state and a novel sampling technique to ensure that the generated plans are robust to failures of any single pursuer robot. An implementation and simulation results illustrating the effectiveness of this approach are described.

In a long-term large-scenario application, the multi-agent collaborative SLAM is expected to improve the robustness and efficiency of executing tasks for mobile agents. In this paper, a multi-agent collaborative visual-inertial SLAM system is proposed based on a centralized client-server (CS) architecture, where the clients run on smart mobiles. In general, multi-agent collaborative SLAM relies on robust and precise experience sharing and efficient communication among agents. The experience sharing requires the place recognition with a high recall and accuracy, the precise estimation of transformation between looping frames, and the map fusion with globally consistency. To this end, we devise an enhanced geometric verification, a re-projection optimization based on the error-aware weighting strategy, and a strategy of flexible fusion to meet these requirements. In addition, the multi-agent collaborative SLAM needs to exchange abundant information, which requires the efficient communication. Therefore, we design a CS collaborative loop detection mechanism which is more robust to network transmission. We perform extensive experiments on the EuRoc dataset and in real environments. Experimental results show that the proposed system achieves better results than state-of-the-art methods. Furthermore, we demonstrate the stability of the proposed collaborative SLAM in real environments with a bandwidth of 7.55Mbps.

We propose a system that uses semantic object detections to localize a microgravity free-flyer. Many applications require absolute localization in a known reference frame, such as the execution of waypoint trajectories defined by human operators. Classical geometric methods build a map of point features, which may not be able to be associated after lighting or environmental changes. By contrast, semantics remain invariant to changes up to the robustness of the detection algorithm and motion of the semantic objects. In this work, we describe our approaches for both offline semantic map generation as well as online localization against a semantic map, intended to run in real-time on the robot. We additionally demonstrate how our semantic localizer outperforms image-feature matching in some cases, and show the robustness of the algorithm to environmental changes. Crucially, we show in our experiments that when semantics are used to supplement point features, localization is always improved. To our knowledge, these experiments demonstrate the first use of learned semantics for localization on a free-flying robot in microgravity.

Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While other approaches involve automatically adjusting environment setups during training, they have been limited to simple environments where low-dimensional encodings can be used. Inspired by these approaches, we propose genetic curriculum, an algorithm that automatically identifies scenarios in which the agent currently fails and generates an associated curriculum to help the agent learn to solve the scenarios and acquire more robust behaviors. As a non-parametric optimizer, our approach uses a raw, non-fixed encoding of scenarios, reducing the need for expert supervision and allowing our algorithm to adapt to the changing performance of the agent. Our empirical studies show improvement in robustness over the existing state of the art algorithms, providing training curricula that result in agents being 2 - 8x times less likely to fail without sacrificing cumulative reward. We include an ablation study and share insights on why our algorithm outperforms prior approaches.