IROS 2021 Papers

In this paper we introduce a novel technique that aims to dynamically control a two-module bio-inspired soft-robotic arm in order to qualitatively reproduce a path defined by sparse way-points. The main idea behind this work is based on the assumption that a complex trajectory may be derived as a combination of a discrete set of parameterizable simple movements, as suggested by Movement Primitive (MP) theory. Capitalising on recent advances in this field, the proposed controller uses a Probabilistic MP (ProMP) model which initially creates an abstract mapping in the primitive-level between the task and the actuation space, and subsequently guides the movement’s composition by exploiting its unique properties - conditioning and blending. At the same time, a learning-based adaptive controller updates the composition parameters by estimating the inverse kinematics of the robot, while an auxiliary process through replanning ensures that the trajectory complies with the new estimation. The learning architecture is evaluated on both a simulation model, and a real soft-robotic arm. The research findings show that the proposed methodology constitutes a novel approach that successfully manages to simplify the trajectory control task for robots of complex dynamics when high-precision is not required.

Autonomous aerial videography is a challenging task, which involves collision avoidance against obstacles and visibility guaranteed target tracking in unstructured environments. In this paper, we organize a two micro aerial vehicle (MAV) team, which consists of a target agent responsible for a specific mission and a camera agent for filming the target agent. Especially, this paper focuses on trajectory planning of the camera agent to chase without occlusion of target agent. Our trajectory planner module includes two phases of guaranteeing target visibility. In the first phase, we generate homotopic safe flight corridor (SFC) to attain target-visible regions. In the subsequent phase, we generate a safe and smooth trajectory with the continuous visibility constraint based on the SFC, using quadratic programming (QP). Regardless of complexity of map, our planner converts an overall problem to a single QP and generates a steady flight trajectory without undesirable fluctuating motion, while guaranteeing all-time visibility. We validate our approach in Gazebo simulations and a real-world experiment.

Prior work in natural-language-driven navigation demonstrates success in systems deployed in synthetic environments or applied to large datasets, both real and synthetic. However, there is an absence of such frameworks being deployed and rigorously tested in real environments, unknown a priori. In this paper, we present a novel framework that uses spoken dialogue with a real person to interpret a set of navigational instructions into a plan and subsequently execute that plan in a novel, unknown, indoor environment. This framework is implemented on a real robot and its performance is evaluated in 39 trials across 3 novel test-building environments. We also demonstrate that our approach outperforms three prior vision-and-language navigation methods in this same environment.

The ability to detect that a grasped object is slipping from the robot gripper is a crucial skill for autonomous robotic manipulation. However, current solutions for automatic slip detection do not perform well in real-world unstructured settings, in which a wide variety of gripper-object interactions could occur. Tactile and force sensing are the most suitable sensory modalities to detect such events, and the recent technological advances in the field are generating novel interesting opportunities. In this work, we propose a data-driven method for automatic slip detection that leverages a novel sensor, which combines the advantages of tactile and force sensing, i.e. distributed measurements of normal and shear contact forces. Interestingly, our model is trained (and tested) uniquely with data obtained during routine robot operations (i.e. in the wild) rather than during a controlled data collection procedure. We compare different sets of tactile/force features to highlight the advantages provided by the different sensory modalities, and we report results that show good detection performances on our in-the-wild dataset, which we make publicly available.

This paper introduces a quadruped robot, Tachyon, which aims to achieve high payload, robust, and dynamic locomotion on the various terrain with high energy efficiency. Thanks to a novel compact series-parallel elastic actuator (SPEA) on the upper link and a four-bar linkage design in the knee joint for constant vertical foot force, the 41-kg robot can carry more than 20 kg of payloads with dynamic walking. The combination of a robust horizontal CoM stabilizer and SPEA joint torque controller provides low impedance force controllability of the whole-body even when only the knee joint can accurately detect its joint torque. The major performance of Tachyon is demonstrated by carrying a 20-kg rice bag—half of its body weight—and climbing stairs with dynamic locomotion. When the robot climbs the stairs, the SPEA parallel spring improves energy efficiency by 16% and is also effective for various other gaits. The robustness of the robot is also shown by its high flexibility and fall avoidance capability when an unknown disturbance of 400 N or more is applied.

In this paper, we propose TUPPer-Map, a metric-semantic mapping framework based on the unified panoptic segmentation and temporal data association. In contrast to the previous mapping method, our framework integrates the data association stage into the holistic pixel-level segmentation stage in an end-to-end fashion, taking advantage of both intra-frame and inter-frame spatial and temporal knowledge. Firstly, we unify two-branch instance segmentation network and semantic segmentation network into a single network by sharing the backbone net, maximizing the 2D panoptic segmentation performance. Next, we leverage geometric segmentation to refine the segments predicted via deep learning. Then, we design a novel deep learning based data association module to track the object instances across different frames. Optical flow of consecutive frames and alignment of ROI (Region of Interest) candidates are learned to predict the frame-consistent instance label. At last, 2D semantics are integrated into 3D volume by TSDF raycasting to build the final map. We evaluated the performance of our framework extensively over the SceneNN, ScanNet v2 and Cityscapes-VPS datasets. Our experimental results demonstrate the superiority of TUPPer-Map over existing semantic mapping methods. Overall, our work illustrates that using learning based data association strategy can enable a more unified perception network for 3D mapping.

Event cameras are bio-inspired vision sensors which measure per pixel brightness changes. They offer numerous benefits over traditional, frame-based cameras, including low latency, high dynamic range, high temporal resolution and low power consumption. Thus, these sensors are suited for robotics and virtual reality applications. To foster the development of 3D perception and navigation algorithms with event cameras, we present the TUM-VIE dataset. It consists of a large variety of handheld and head-mounted sequences in indoor and outdoor environments, including rapid motion during sports and high dynamic range scenarios. The dataset contains stereo event data, stereo grayscale frames at 20Hz as well as IMU data at 200Hz. Timestamps between all sensors are synchronized in hardware. The event cameras contain a large sensor of 1280x720 pixels, which is significantly larger than the sensors used in existing stereo event datasets (at least by a factor of ten). We provide ground truth poses from a motion capture system at 120Hz during the beginning and end of each sequence, which can be used for trajectory evaluation. TUM-VIE includes challenging sequences where state-of-the art visual SLAM algorithms either fail or result in large drift. Hence, our dataset can help to push the boundary of future research on event-based visual-inertial perception algorithms.

Lower Limb Exoskeletons (LLEs) are promising in gait rehabilitation for stroke survivors. In gait training of post-stroke patients with LLEs, one of the main challenges is how to generate appropriate gait patterns from the sound leg to the paretic leg for different patients with varying walking speeds. In this paper, we proposed a Synergetic Gait Prediction (SGP) model for rehabilitation LLEs with post-stroke patients, which can generate adaptive synergetic gait patterns for different patients with varying walking speeds. The proposed SGP model is based on Sequence-to-Sequence (Seq2Seq) neural networks with temporal attention mechanisms. In the training procedure of the proposed SGP model, a gait database with collected gait patterns from healthy subjects is employed to learn the parameters of SGP model. The SGP model takes current joint angles from the sound leg and a segment of observed history joint angles from both legs as input and predicts the future joint angles for the paretic leg. We compared the effectiveness of the SGP model with the Long Short Term Memory (LSTM) model, experimental results indicate that SGP model can generate synergetic gait patterns for different subjects via varying walking speeds with less prediction error.

Loop closure detection is an essential tool of Simultaneous Localization and Mapping (SLAM) to minimize drift in its localization. Many state-of-the-art loop closure detection (LCD) algorithms use visual Bag-of-Words (vBoW), which is robust against partial occlusions in a scene but cannot perceive the semantics or spatial relationships between feature points. CNN object extraction can address those issues, by providing semantic labels and spatial relationships between objects in a scene. Previous work has mainly focused on replacing vBoW with CNN derived features. In this paper we propose SymbioLCD, a novel ensemble-based LCD that utilizes both CNN-extracted objects and vBoW features for LCD candidate prediction. When used in tandem, the added elements of object semantics and spatial-awareness creates a more robust and symbiotic loop closure detection system. The proposed SymbioLCD uses scale-invariant spatial and semantic matching, Hausdorff distance with temporal constraints, and a Random Forest that utilizes combined information from both CNN-extracted objects and vBoW features for predicting accurate loop closure candidates. Evaluation of the proposed method shows it outperforms other Machine Learning (ML) algorithms - such as SVM, Decision Tree and Neural Network, and demonstrates that there is a strong symbiosis between CNN-extracted object information and vBoW features which assists accurate LCD candidate prediction. Furthermore, it is able to perceive loop closure candidates earlier than state-of-the-art SLAM algorithms, utilizing added spatial and semantic information from CNN-extracted objects.

This study analyzes the surface sliding behavior of space probes in simulated extraterrestrial environments. When a space probe lands on an extraterrestrial body, its landing gear (footpad, landing legs) contacts and slides along the ground surface. The influence of various parameters (i.e., footpad size, velocity, ground condition, atmospheric pressure, and gravity) on the friction behavior of footpads was experimentally evaluated herein. First, we developed an experimental system that can perform footpad sliding tests repeatedly. Subsequently, the system was used to perform tests under various conditions: 1) on ground under normal atmospheric conditions, 2) in a vacuum, and 3) in reduced gravity. The tests performed in a vacuum and in reduced gravity indicated that the friction behavior of the footpad is largely unaffected by atmospheric pressure and gravity. The findings obtained herein offer useful design and control guidelines for space probes landing on extraterrestrial bodies.

Autonomous surgical execution relieves tedious routines and surgeon’s fatigue. Recent learning-based methods, especially reinforcement learning (RL) based methods, achieve promising performance for dexterous manipulation, which usually requires the simulation to collect data efficiently and reduce the hardware cost. The existing learning-based simulation platforms for medical robots suffer from limited scenarios and simplified physical interactions, which degrades the real-world performance of learned policies. In this work, we designed SurRoL, an RL-centered simulation platform for surgical robot learning compatible with the da Vinci Research Kit (dVRK). The designed SurRoL integrates a user-friendly RL library for algorithm development and a real-time physics engine, which is able to support more PSM/ECM scenarios and more realistic physical interactions. Ten learning-based surgical tasks are built in the platform, which are common in the real autonomous surgical execution. We evaluate SurRoL using RL algorithms in simulation, provide in-depth analysis, deploy the trained policies on the real dVRK, and show that our SurRoL achieves better transferability in the real world.

This paper proposes an improved supervised autonomy framework for remote teleoperation of a quadrupedal bimanual mobile manipulator in an unknown environment, with the usage of advanced perception technology and allowing the operator to easily assist the robot with decision making for executing tasks on the fly. First, the perception system uses lightweight deep neural network-based Single Shot Detector (SSD) MobileNet on RGB images to detect objects and highlight them to the human operator via an intuitive interactive visualization interface. After object and action selections are made by the operator, segmentation of object point cloud and 3D surfaces based on random sample consensus is performed, followed by object pose localization by using keypoint extraction. Based on the localized object, mobile manipulation motion to perform the operator-selected action is planned and executed with the help of a state estimator for the hybrid wheel-legged robot. Thanks to the autonomy of the robot in perception and manipulation, the complexity of teleoperating the robot is reduced to specifying the essential task objectives. Experimental results on the real robot, with full system integration, for 2 task scenarios, namely passage clearing and object retrieval, demonstrate a high average success rate of 92.2% over a total of 90 trials.

Along with point features, line features play an important role in achieving robust Simultaneous Localization and Mapping (SLAM) under complex environments. This paper proposes a fast and effective method, namely Superline, to simultaneously detect line segments and generate robust descriptors for matching. The entire model is composed of a convolutional backbone and two task heads, i.e., detection head and description head respectively. A line selecting mechanism and a spatial pyramid Line-of-Interest (LOI) pooling module is specially designed in the description head to aggregate multi-scale information into line feature descriptors. The entire model is implemented end-to-end and can be trained on a dataset with only line annotations and without the need of providing ground truth matching. Comparative experimental results on Wireframe and York Urban datasets as well as applying the Superline features on SLAM applications demonstrate the superior performance of our method.

We propose Super Odometry, a high-precision multi-modal sensor fusion framework, providing a simple but effective way to fuse multiple sensors such as LiDAR, camera, and IMU sensors and achieve robust state estimation in perceptually-degraded environments. Different from traditional sensor-fusion methods, Super Odometry employs an IMU-centric data processing pipeline, which combines the advantages of loosely coupled methods with tightly coupled methods and recovers motion in a coarse-to-fine manner. The proposed framework is composed of three parts: IMU odometry, Visual-inertial odometry, and LiDAR-inertial odometry. The Visual-inertial odometry and LiDAR-inertial odometry provide the pose prior to constrain the IMU bias and receive the motion prediction from IMU odometry. To ensure high performance in real-time, we apply a dynamic octree that only consumes 10% of the running time compared with a static KD-tree. The proposed system was deployed on drones and ground robots, as part of Team Explorer’s effort to the DARPA Subterranean Challenge where the team won 1st and 2nd place in the Tunnel and Urban Circuits 1, respectively.

We present Success weighted by Completion Time (SCT), a new metric for evaluating navigation performance for mobile robots. Several related works on navigation have used Success weighted by Path Length (SPL) as the primary method of evaluating the path an agent makes to a goal location, but SPL is limited in its ability to properly evaluate agents with complex dynamics. In contrast, SCT explicitly takes the agent’s dynamics model into consideration, and aims to accurately capture how well the agent has approximated the fastest navigation behavior afforded by its dynamics. While several embodied navigation works use point-turn dynamics, we focus on unicycle-cart dynamics for our agent, which better exempli-fies the dynamics model of popular mobile robotics platforms (e.g., LoCoBot, TurtleBot, Fetch, etc.). We also present RRT*-Unicycle, an algorithm for unicycle dynamics that estimates the fastest collision-free path and completion time from a starting pose to a goal location in an environment containing obstacles. We experiment with deep reinforcement learning and reward shaping to train and compare the navigation performance of agents with different dynamics models. In evaluating these agents, we show that in contrast to SPL, SCT is able to capture the advantages in navigation speed a unicycle model has over a simpler point-turn model of dynamics. Lastly, we show that we can successfully deploy our trained models and algorithms outside of simulation in the real world. We embody our agents in a real robot to navigate an apartment, and show that they can generalize in a zero-shot manner. A video summary is available here: https://youtu.be/QOQ56XVIYVE

Path tracking controllers for non-holonomic vehicles are commonly designed with primary focus on robustness to different kinds of disturbances, vehicle dynamics and other effects. Subsequently, the path tracking behavior is improved by parameter optimization or controller extensions. The possible improvement, however, is already limited by the control design itself. To overcome this drawback, this paper presents an alternative approach: In a first step, we formulate the geometrically optimal solution for a Dubins car reaching a straight target path as a globally stabilizing sliding-mode feedback law. In a second step, we introduce a single tuning parameter that achieves robustness with respect to exemplary matched disturbances. The result is a sub-optimal but robust controller. We investigate the impact of tuning this parameter towards optimality or robustness with respect to the transient and stationary tracking behavior in simulation scenarios. Following the presented approach, it is possible to adapt the control law with respect to various disturbances facing and utilizing the trade-off between optimality and robustness. This is a significant advantage in the design of path tracking controllers due to the variety of vehicle actuation and measurement systems and operational design domains.

Deep Neural Networks (DNNs) are a critical component for self-driving vehicles. They achieve impressive performance by reaping information from high amounts of labeled data. Yet, the full complexity of the real world cannot be encapsulated in the training data, no matter how big the dataset, and DNNs can hardly generalize to unseen conditions. Robustness to various image corruptions, caused by changing weather conditions or sensor degradation and aging, is crucial for safety when such vehicles are deployed in the real world. We address this problem through a novel type of layer, dubbed StyleLess, which enables DNNs to learn robust and informative features that can cope with varying external conditions. We propose multiple variations of this layer that can be integrated in most of the architectures and trained jointly with the main task. We validate our contribution on typical autonomous-driving tasks (detection, semantic segmentation), showing that in most cases, this approach improves predictive performance on unseen conditions (fog, rain), while preserving performance on seen conditions and objects.

We propose a novel technique for guidance of buoyancy-controlled vehicles in uncertain under-ice ocean flows. In-situ melt rate measurements collected at the grounding zone of Antarctic ice shelves, where the ice shelf meets the underlying bedrock, are essential to constrain models of future sea level rise. Buoyancy-controlled vehicles, which control their vertical position in the water column through internal actuation but have no means of horizontal propulsion, offer an affordable and reliable platform for such in-situ data collection. However, reaching the grounding zone requires vehicles to traverse tens of kilometers under the ice shelf, with approximate position knowledge and no means of communication, in highly variable and uncertain ocean currents. To address this challenge, we propose a partially observable MDP approach that exploits model-based knowledge of the under-ice currents and, critically, of their uncertainty, to synthesize effective guidance policies. The approach uses approximate dynamic programming to model uncertainty in the currents, and QMDP to address localization uncertainty. Numerical experiments show that the policy can deliver up to 88.8% of underwater vehicles to the grounding zone – a 33% improvement compared to state-of-the-art guidance techniques, and a 262% improvement over uncontrolled drifters. Collectively, these results show that model-based under-ice guidance is a highly promising technique for exploration of under-ice cavities, and has the potential to enable cost-effective and scalable access to these challenging and rarely observed environments.

This paper proposes a stereovision-guided robotic laser system that can conduct laser ablation on targets selected by human operators in the color image, referred as StereoCNC. Two digital cameras are integrated into a previously developed robotic laser system to add a color sensing modality and formulate the stereovision. A calibration method is implemented to register the coordinate frames between stereo cameras and the laser system, modelled as a 3D-to-3D Least-squares problem. This problem is solved by a RANSAC-based 3D rigid transformation method and the calibration reprojection errors are used to characterize a 3D error field by Gaussian process regression. This regression error model is used to predict an error value for each data point of a stereo-reconstructed point cloud and an optimization problem is formulated to adjust the surgical site to a new position with minimum reprojection errors. Based on the calibrated system and the error model, a stereovision-guided laser-tissue removal pipeline is proposed to precisely locate, target, and ablate a surface region. The pipeline is validated by the experiments on phantoms with color texture and various geometric shapes. The overall targeting accuracy of the system achieves an average RMSE of 0.13±0.02 mm and maximum error of 0.34±0.06 mm, as measured by pre- and post-laser ablation images. The results show potential applications of using the developed stereovision-guided robotic system for superficial laser surgery, including dermatologic applications or removal of exposed tumorous tissue in neurosurgery.