ICRA 2020 Papers

This paper presents the development and results of a large 3 d.o.f cable-driven parallel robot (CDPR) that has been extensively used between June and August 2019 for an artistic exhibition. The purpose of the exhibition was to 3D print a wall of glass powder, which will slowly collapse after the deposit of each layer. Positioning control on the assigned trajectory was an issue because of the CDPR geometry imposed by the specific configuration of the exhibition place. We describe how this problem was solved using a combination of cable length estimation based on the winch rotation measured by encoder, together with 3 on-board lidars that were used to provide a measure of the robot position. To the best of our knowledge this is the first time that such method was used for controlling a large CDPR. This CDPR has run for 174 hours since 6/18/2019, averaging a run time of 4h15mn per day. The 3D printing of the wall started on 7/18/2019 and stops on 8/31/2019. During this period the robot was used for 32 days with an average of 2h18mn run-time per day. The robot has traveled on a total distance of 4757 meters, of which 3893 meters on the assigned trajectory. During the period 76 layers have been deposited, representing a mass of 1.5 tons of glass powder.

In this paper we present The Oxford Radar RobotCar Dataset, a new dataset for researching scene understanding using Millimetre-Wave FMCW scanning radar data. The target application is autonomous vehicles where this modality is robust to environmental conditions such as fog, rain, snow, or lens flare, which typically challenge other sensor modalities such as vision and LIDAR.(/P)(P)The data were gathered in January 2019 over thirty-two traversals of a central Oxford route spanning a total of 280 km of urban driving. It encompasses a variety of weather, traffic, and lighting conditions. This 4.7 TB dataset consists of over 240,000 scans from a Navtech CTS350-X radar and 2.4 million scans from two Velodyne HDL-32E 3D LIDARs; along with six cameras, two 2D LIDARs, and a GPS/INS receiver. In addition we release ground truth optimised radar odometry to provide an additional impetus to research in this domain. The full dataset is available for download at: ori.ox.ac.uk/datasets/radar-robotear-dataset.

Despite the utility and benefits of omnidirectional images in robotics and automotive applications, there are no datasets of omnidirectional images available with semantic segmentation, depth map, and dynamic properties. This is due to the time cost and human effort required to annotate ground truth images. This paper presents a framework for generating omnidirectional images using images that are acquired from a virtual environment. For this purpose, we demonstrate the relevance of the proposed framework on two well-known simulators: CARLA Simulator, which is an open-source simulator for autonomous driving research, and Grand Theft Auto V (GTA V), which is a very high quality video game. We explain in details the generated OmniScape dataset, which includes stereo fisheye and catadioptric images acquired from the two front sides of a motorcycle, including semantic segmentation, depth map, intrinsic parameters of the cameras and the dynamic parameters of the motorcycle. It is worth noting that the case of two-wheeled vehicles is more challenging than cars due to the specific dynamic of these vehicles.

This paper describes a new type of bending module inspired, in part, by the musculoskeletal structure of the lobster leg joint. The bending module proposed combines enhanced torque output, reconfigurability in assembling, safe compliant actuation, and accurate control on its mechanical performance. In this module, antagonistic soft chambers are enveloped by exoskeleton shells, and the bending angle and the stiffness can be independently adjusted by controlling the input pressure in the two chambers. Theoretical models are developed to characterize the relationships between the input pressure, bending angle, and stiffness, and a controller for angle control and stiffness tuning is constructed with experimental validation. The fabricated module can reach the maximum torque output of 109.7 N•mm under 40 kPa and the stiffness range from 40 to 220 N•mm / rad, demonstrating its capacity to fulfill both safe interactions and forceful tasks.

We propose to integrate text objects in man-made scenes tightly into the visual SLAM pipeline. The key idea of our novel text-based visual SLAM is to treat each detected text as a planar feature which is rich of textures and semantic meanings. The text feature is compactly represented by three parameters and integrated into visual SLAM by adopting the illumination-invariant photometric error. We also describe important details involved in implementing a full pipeline of text-based visual SLAM. To our best knowledge, this is the first visual SLAM method tightly coupled with the text features. We tested our method in both indoor and outdoor environments. The results show that with text features, the visual SLAM system becomes more robust and produces much more accurate 3D text maps that could be useful for navigation and scene understanding in robotic or augmented reality applications.

This paper presents an approach and introduces new open-source tools that can be used to evaluate robotic mapping algorithms. Also described is an extensive subterranean mine rescue dataset based upon the DARPA Subterranean (SubT) challenge including professionally surveyed ground truth. Finally, some commonly available approaches are evaluated using this metric.

The horizon line is an important geometric feature for many image processing and scene understanding tasks in computer vision. For instance, in navigation of autonomous vehicles or driver assistance, it can be used to improve 3D reconstruction as well as for semantic interpretation of dynamic environments. While both algorithms and datasets exist for single images, the problem of horizon line estimation from video sequences has not gained attention. In this paper, we show how convolutional neural networks are able to utilise the temporal consistency imposed by video sequences in order to increase the accuracy and reduce the variance of horizon line estimates. A novel CNN architecture with an improved residual convolutional LSTM is presented for temporally consistent horizon line estimation. We propose an adaptive loss function that ensures stable training as well as accurate results. Furthermore, we introduce an extension of the KITTI dataset which contains precise horizon line labels for 43699 images across 72 video sequences. A comprehensive evaluation shows that the proposed approach consistently achieves superior performance compared with existing methods.

We present a temporal Bayesian filter for semantic segmentation of a video sequence. Each pixel is a random variable following a discrete probabilistic distribution function representing possible semantic classes. Bayesian filtering consists in two main steps: 1) a prediction model and 2) an observation model (likelihood). We propose to use a datadriven prediction function derived from a dense optical flow between images t and t + 1 achieved by a deep neural network [1]. Moreover, the observation function uses a semantic segmentation network. The resulting approach is evaluated on the public dataset Cityscapes. We show that using the temporal filtering increases the accuracy of the semantic segmentation.

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.

Multi-robot systems are able to achieve common objectives exchanging information among each other. This is possible exploiting a communication structure, usually modeled as a graph, whose topological properties (such as connectivity) are very relevant in the overall performance of the multirobot system. When considering mobile robots, such properties can change over time: robots are then controlled to preserve them, thus guaranteeing the possibility, for the overall system, to achieve its goals. This, however, implies limitations on the possible motion patterns of the robots, thus reducing the flexibility of the overall multi-robot system. In this paper we introduce teleoperation as a means to reduce these limitations, allowing temporary violations of topological properties, with the aim of increasing the flexibility of the multi-robot system.

Deep-learning driven safety-critical autonomous systems, such as self-driving cars, must be able to detect situations where its trained model is not able to make a trustworthy prediction. This ability to determine the novelty of a new input with respect to a trained model is critical for such systems because novel inputs due to changes in the environment, adversarial attacks, or even unintentional noise can potentially lead to erroneous, perhaps life-threatening decisions. This paper proposes a learning framework that leverages information learned by the prediction model in a task-aware manner to detect novel scenarios. We use network saliency to provide the learning architecture with knowledge of the input areas that are most relevant to the decision-making and learn an association between the saliency map and the predicted output to determine the novelty of the input. We demonstrate the efficacy of this method through experiments on real-world driving datasets as well as through driving scenarios in our in-house indoor driving environment where the novel image can be sampled from another similar driving dataset with similar features or from adversarial attacked images from the training dataset. We find that our method is able to systematically detect novel inputs and quantify the deviation from the target prediction through this task-aware approach.

We investigate algorithmic approaches for targeted drug delivery in a complex, maze-like environment, such as a vascular system. The basic scenario is given by a large swarm of micro-scale particles ("agents") and a particular target region ("tumor") within a system of passageways. Agents are too small to contain on-board power or computation and are instead controlled by a global external force that acts uniformly on all particles, such as an applied fluidic flow or electromagnetic field. The challenge is to deliver all agents to the target region with a minimum number of actuation steps. We provide a number of results for this challenge. We show that the underlying problem is NP-hard, which explains why previous work did not provide provably efficient algorithms. We also develop a number of algorithmic approaches that greatly improve the worst-case guarantees for the number of required actuation steps. We evaluate our algorithmic approaches by a number of simulations, both for deterministic algorithms and searches supported by deep learning, which show that the performance is practically promising.

The ocean is a harsh and unstructured environment for robotic systems; high ambient pressures, saltwater corrosion and low-light conditions demand machines with robust electrical and mechanical parts that are able to sense and respond to the environment. Prior work shows that the addition of gentle suction flow to the hands of underwater robots can aid in the handling of objects during mobile manipulation tasks. The current paper explores using this suction flow mechanism as a new modality for tactile sensing; by monitoring orifice occlusion we can get a sense of how objects make contact in the hand. The electronics required for this sensor can be located remotely from the hand and the signal is insensitive to large changes in ambient pressure associated with diving depth. In this study, suction is applied to the fingertips of a two-fingered compliant gripper and suction-based tactile sensing is monitored while an object is pulled out of a pinch grasp. As a proof of concept, a recurrent neural network model was trained to predict external force trends using only the suction signals. This tactile sensing modality holds the potential to enable automated robotic behaviors or to provide operators of remotely operated vehicles with additional feedback in a robust fashion suitable for ocean deployment.

The sense of touch, which is essential for human dexterity, is virtually absent from today's robotic hands. In this work we present progress in creating a highly-dexterous bimanual tactile telerobot, and evaluate its performance compared to bare human hands. The system, consisting of anthropomorphic robot hands, biomimetic tactile sensors, and advanced haptic gloves, enables a human operator to intuitively control and feel what the robotic hands are touching. Through carefully tuned tactile and kinematic mapping it was possible to intuitively perform dexterous operations, including pick and place tasks and even in-hand manipulation, a challenge for most autonomous robotic hands. Performance of the system was evaluated in standard measures of human and robotic dexterity such as the Box and Block test and other YCB benchmarks. This first-generation telerobot was found to have promising performance with the pilot able to do the same tasks in the telerobot between 1/4th to 1/12th the speed of their bare hands depending on the task complexity.

This paper develops closed-loop tactile controllers for dexterous robotic manipulation with a dual-palm robotic system. Tactile dexterity is an approach to dexterous manipulation that plans for robot/object interactions that render interpretable tactile information for control. We divide the role of tactile control into two goals: 1) control the contact state between the end-effector and the object (contact/no-contact, stick/slip) by regulating the stability of planned contact configurations and monitoring undesired slip events; and 2) control the object state by tactile-based tracking and iterative replanning of the object and robot trajectories. Key to this formulation is the decomposition of manipulation plans into sequences of manipulation primitives with simple mechanics and efficient planners. We consider the scenario of manipulating an object from an initial pose to a target pose on a flat surface while correcting for external perturbations and uncertainty in the initial pose of the object. We experimentally validate the approach with an ABB YuMi dual-arm robot and demonstrate the ability of the tactile controller to react to external perturbations.

TUNERCAR is a toolchain that jointly optimizes racing strategy, planning methods, control algorithms, and vehicle parameters for an autonomous racecar. In this paper, we detail the target hardware, software, simulators, and systems infrastructure for this toolchain. Our methodology employs a parallel implementation of CMA-ES which enables simulations to proceed 6 times faster than real-world rollouts. We show our approach can reduce the lap times in autonomous racing, given a fixed computational budget. For all tested tracks, our method provides the lowest lap time, and relative improvements in lap time between 7-21%. We demonstrate improvements over a naive random search method with equivalent computational budget of over 15 seconds/lap, and improvements over expert solutions of over 2 seconds/lap. We further compare the performance of our method against hand-tuned solutions submitted by over 30 international teams, comprised of graduate students working in the field of autonomous vehicles. Finally, we discuss the effectiveness of utilizing an online planning mechanism to reduce the reality gap between our simulation and actual tests.

A longstanding challenge in robot learning for manipulation tasks has been the ability to generalize to varying initial conditions, diverse objects, and changing objectives. Learning based approaches have shown promise in producing robust policies, but require heavy supervision and large number of environment interactions, especially from visual inputs. We propose a novel self-supervision technique that uses time-reversal to provide high level supervision to reach goals. In particular, we introduce the time-reversal model (TRM), a self-supervised model which explores outward from a set of goal states and learns to predict these trajectories in reverse. This provides a high level plan towards goals, allowing us to learn complex manipulation tasks with no demonstrations or exploration at test time. We test our method on the domain of assembly, specifically the mating of tetris-style block pairs. Using our method operating atop visual model predictive control, we are able to assemble tetris blocks on a KuKa IIWA-7 using only uncalibrated RGB camera input, and generalize to unseen block pairs. Project's-page: https://sites.google.com/view/time-reversal.

In a physical Human-Robot Interaction for industrial scenarios is paramount to guarantee the safety of the user while keeping the robot's performance. Hierarchical task approaches are not sufficient since they tend to sacrifice the low priority tasks in order to guarantee the consistency of the main task. To handle this problem, we enhance the standard hierarchical fusion by introducing a novel interactive task-reconfiguring approach (TACTO-Selector) that uses the information of the tactile interaction to adapt the dimension of the tasks, therefore guaranteeing the execution of the safety task while performing the other task as good as possible. In this work, we hierarchically combine a 6 DOF Position-Based Visual Servoing (PBVS) task with a reactive skin control. This approach was evaluated on a 6 DOF industrial robot showing an improvement of 36.37% on average in tracking error reduction compared with a standard approach.

The fundamental path planning problem for a mobile robot involves generating a trajectory for point-to-point navigation while avoiding obstacles. Heuristic-based search algorithms like A* have been shown to be efficient in solving such planning problems. Recently, there has been an increased interest in specifying complex path planning problem using temporal logic. In the state-of-the-art algorithm, the temporal logic path planning problem is reduced to a graph search problem, and Dijkstra's shortest path algorithm is used to compute the optimal trajectory satisfying the specification.The A* algorithm, when used with an appropriate heuristic for the distance from the destination, can generate an optimal path in a graph more efficiently than Dijkstra's shortest path algorithm. The primary challenge for using A* algorithm in temporal logic path planning is that there is no notion of a single destination state for the robot. We present a novel path planning algorithm T* that uses the A* search procedure opportunistically to generate an optimal trajectory satisfying a temporal logic query. Our experimental results demonstrate that T* achieves an order of magnitude improvement over the state-of-the-art algorithm to solve many temporal logic path planning problems in 2-D as well as 3-D workspaces.