ICRA 2019 Papers

We present a novel method of integrating image-based measurements into a drone navigation system for the automated inspection of wind turbines. We take a model-based tracking approach, where a 3D skeleton representation of the turbine is matched to the image data. Matching is based on comparing the projection of the representation to that inferred from images using a convolutional neural network. This enables us to find image correspondences using a generic turbine model that can be applied to a wide range of turbine shapes and sizes. To estimate 3D pose of the drone, we fuse the network output with GPS and IMU measurements using a pose graph optimiser. Results illustrate that the use of the image measurements significantly improves the accuracy of the localisation over that obtained using GPS and IMU alone.

We investigate decentralised decision-making, in which a robot swarm is tasked with selecting the best-quality option among a set of alternatives. Individual robots are simplistic as they only perform diffusive search, make local noisy estimates of the options' quality, and exchange information with near neighbours. We propose a decentralised algorithm, inspired by house-hunting honeybees, to efficiently aggregate noisy estimations. Individual robots, by varying over time a single decentralised parameter that modulates the interaction strength, balance exploration and agreement. In this way, the swarm first identifies the options under consideration, then rapidly converges on the best available option, even when outnumbered by lower quality options. We present stochastic analyses and swarm robotics simulations to compare the novel strategy with previous methods and to quantify the performance improvement. The proposed strategy limits the spreading of errors within the population and allows swarms of simple noisy units with minimal communication capabilities to make highly accurate collective decisions in predictable time.

This paper presents two novel approaches for improving image-based underwater obstacle detection by combining sparse stereo point clouds with monocular semantic image segmentation. Generating accurate image-based obstacle maps in cluttered underwater environments, such as coral reefs, are essential for robust robotic path planning and navigation. However, these maps can be challenged by factors including visibility, lighting and dynamic objects (e.g. fish) that may lead to falsely identified free space or dynamic objects which trajectory planners may react to undesirably. We propose combining feature-based stereo matching with learning-based segmentation to produce a more robust obstacle map. This approach considers direct binary learning of the presence or absence of underwater obstacles, as well as a multiclass learning approach to classify their distance (near, mid and far) in the scene. An enhancement to the binary map is also shown by including depth information from sparse stereo matching to produce 3D obstacle maps of the scene. The performance is evaluated using field data collected in cluttered, and at times, visually degraded coral reef environments. The results show improved image-wide obstacle detection, rejection of transient objects (such as fish), and range estimation compared to feature-based sparse and dense stereo point clouds alone.

Motivated by the need to improve the performance of visual loop closure verification via multi-view geometry (MVG) under significant illumination and viewpoint changes, we propose a keypoint matching method that uses landmarks as an intermediate image representation in order to leverage the power of deep learning. In environments with various changes, the traditional verification method via MVG may encounter difficulty because of their inability to generate a sufficient number of correctly matched keypoints. Our method exploits the excellent invariance properties of convolutional neural network (ConvNet) features, which have shown outstanding performance for matching landmarks between images. By generating and matching landmarks first in the images and then matching the keypoints within the matched landmark pairs, we can significantly improve the quality of matched keypoints in terms of precision and recall measures. The proposed method is validated on challenging datasets that involve significant illumination and viewpoint changes, to establish its superior performance to the standard keypoint matching method.

Deployment of motion planning algorithms in practical applications has lagged due to their slow speed in reacting to disturbances. We believe that the best way to address this is to reuse learned planning and control information across queries. In previous work, we introduced Chekov, a reactive, integrated motion planning and execution system that reuses learned information in the form of an enhanced roadmap. We have previously shown how we can use Chekov to formulate trajectory optimization problems that result in superior performance in static environments. In this work, we show how incremental planning can be incorporated into the formulation of optimized trajectories from roadmap seed trajectories. Further, we show how an incremental planner can be adapted to reduce the overhead incurred for replanning when trajectories become invalid during execution.

Humans can form an impression of how a new object feels simply by touching its surfaces with the densely innervated skin of the fingertips. Many haptics researchers have recently been working to endow robots with similar levels of haptic intelligence, but these efforts almost always employ hand-crafted features, which are brittle, and concrete tasks, such as object recognition. We applied unsupervised feature learning methods, specifically K-SVD and Spatio-Temporal Hierarchical Matching Pursuit (ST-HMP), to rich multi-modal haptic data from a diverse dataset. We then tested the learned features on 19 more abstract binary classification tasks that center on haptic adjectives such as smooth and squishy. The learned features proved superior to traditional hand-crafted features by a large margin, almost doubling the average F1 score across all adjectives. Additionally, particular exploratory procedures (EPs) and sensor channels were found to support perception of certain haptic adjectives, underlining the need for diverse interactions and multi-modal haptic data.

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically-continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Given the task of learning robotic grasping solely based on a depth camera input and gripper force feedback, we derive a learning algorithm from an applied point of view to significantly reduce the amount of required training data. Major improvements in time and data efficiency are achieved by: Firstly, we exploit the geometric consistency between the undistorted depth images and the task space. Using a relative small, fully-convolutional neural network, we predict grasp and gripper parameters with great advantages in training as well as inference performance. Secondly, motivated by the small random grasp success rate of around 3 %, the grasp space was explored in a systematic manner. The final system was learned with 23 000 grasp attempts in around 60 h, improving current solutions by an order of magnitude. For typical bin picking scenarios, we measured a grasp success rate of (96.6 ± 1.0) %. Further experiments showed that the system is able to generalize and transfer knowledge to novel objects and environments.

We describe a single fingertip-mounted sensing system for robot manipulation that provides proximity (pre-touch), contact detection (touch), and force sensing (post-touch). The sensor system consists of optical time-of-flight range measurement modules covered in a clear elastomer. Because the elastomer is clear, the sensor can detect and range nearby objects, as well as measure deformations caused by objects that are in contact with the sensor and thereby estimate the applied force. We examine how this sensor design can be improved with respect to invariance to object reflectivity, signal-to-noise ratio, and continuous operation when switching between the distance and force measurement regimes. By harnessing time-of-flight technology and optimizing the elastomer-air boundary to control the emitted light's path, we develop a sensor that is able to seamlessly transition between measuring distances of up to 50 mm and contact forces of up to 10 newtons. We demonstrate that our sensor improves manipulation accuracy in a block unstacking task. Thorough instructions for manufacturing the sensor from inexpensive, commercially available components are provided, as well as all relevant hardware design files and software sources.

Gesture interaction is a natural way of communicating with a robot as an alternative to speech. Gesture recognition methods leverage optical flow in order to understand human motion. However, while accurate optical flow estimation (i.e., traditional) methods are costly in terms of runtime, fast estimation (i.e., deep learning) methods' accuracy can be improved. In this paper, we present a pipeline for gesture-based human-robot interaction that uses a novel optical flow estimation method in order to achieve an improved speed-accuracy trade-off. Our optical flow estimation method introduces four improvements to previous deep learning-based methods: strong feature extractors, attention to contours, midway features, and a combination of these three. This results in a better understanding of motion, and a finer representation of silhouettes. In order to evaluate our pipeline, we generated our own dataset, MIBURI, which contains gestures to command a house service robot. In our experiments, we show how our method improves not only optical flow estimation, but also gesture recognition, offering a speed-accuracy trade-off more realistic for practical robot applications.

In the task of Autonomous aerial filming of a moving actor (e.g. a person or a vehicle), it is crucial to have a good heading direction estimation for the actor from the visual input. However, the models obtained in other similar tasks, such as pedestrian collision risk analysis and human-robot interaction, are very difficult to generalize to the aerial filming task, because of the difference in data distributions. Towards improving generalization with less amount of labeled data, this paper presents a semi-supervised algorithm for heading direction estimation problem. We utilize temporal continuity as the unsupervised signal to regularize the model and achieve better generalization ability. This semi-supervised algorithm is applied to both training and testing phases, which increases the testing performance by a large margin. We show that by leveraging unlabeled sequences, the amount of labeled data required can be significantly reduced. We also discuss several important details on improving the performance by balancing labeled and unlabeled loss, and making good combinations. Experimental results show that our approach robustly outputs the heading direction for different types of actor. The aesthetic value of the video is also improved in the aerial filming task.

Coverage performance in a coverage path planning problem depends both on the path created and on the footprint of the sensor used. The footprint can be increased either by increasing the size of the sensor, or by mounting the sensor on a robotic arm to allow scanning over larger areas as the platform moves, effectively creating a virtual sensor with a larger footprint than the physical sensor's. However, the virtual footprint comes at a cost requiring formulating an optimization problem for the area of interest. In this work, three common strategies to use a metal detector on a platform are discussed, their time and energy performances are formulated and the corresponding optima are found.

With node selection being directed by a heuristic cost [1]-[3], A-search guided tree (AGT) is constructed on-the-fly and enables fast kinodynamic planning. This work presents two variants of AGT to improve computation efficiency. An improved AGT (i-AGT) biases node expansion through prioritizing control actions, an analogy of prioritizing nodes. Focusing on node selection, a bi-directional AGT (BAGT) introduces a second tree originated from the goal in order to offer a better heuristic cost of the first tree. Effectiveness of BAGT pivots on the fact that the second tree encodes obstacles information near the goal. Case study demonstrates that i-AGT consistently reduces the complexity of the tree and improves computation efficiency; and BAGT works largely but not always, particularly with no benefit observed for simple cases.

We present a Learning from Demonstration method for teaching robots to perform search strategies imitated from humans in scenarios where alignment tasks fail due to position uncertainty. The method utilizes human demonstrations to learn both a state invariant dynamics model and an exploration distribution that captures the search area covered by the demonstrator. We present two alternative algorithms for computing a search trajectory from the exploration distribution, one based on sampling and another based on deterministic ergodic control. We augment the search trajectory with forces learnt through the dynamics model to enable searching both in force and position domains. An impedance controller with superposed forces is used for reproducing the learnt strategy. We experimentally evaluate the method on a KUKA LWR4+ performing a 2D peg-in-hole and a 3D electricity socket task. Results show that the proposed method can, with only few human demonstrations, learn to complete the search task.

Vision-based localization and mapping in outdoor environments is still a challenging issue, which requests significant robustness against various unpredictable illumination changes. In this paper, an illumination-robust direct monocular SLAM system that focuses on modeling outdoor scenery is presented. To deal with global and local lighting changes, such as solar flares, the state-of-art illumination invariant photometric costs for RGB-D and stereo SLAM systems are revisited in the context of their monocular counterpart, where the camera motion and scene structure are jointly optimized with a reasonably poor initialization. Based on our analysis, a combined cost is proposed to achieve a high-precision motion estimation with an improved convergence radius. The proposed system is extensively evaluated on the synthetic and real-world datasets regarding accuracy, robustness, and processing time, where our approach outperforms systems with other costs and state-of-art DSO and ORBSLAM2 systems.

Successfully completing a complex manufacturing task requires finding a feasible placement of the workpiece in the robot workspace. The workpiece placement should be such that the task surfaces on the workpiece are reachable by the robot, the robot can apply the required forces, and the end-effector/tool can move with the desired velocity. This paper formulates the problem of identifying a feasible placement as a non-linear optimization problem over the constraint violation functions. This is a computationally challenging problem. We show that this problem can be solved by successively searching for the solution by incrementally applying different constraints. We demonstrate the feasibility of our approach using several complex workpieces.

Ability of autonomous vehicles to operate in complex dynamic environments requires, among other things, fast and accurate perception of surroundings, which includes recognition and tracking of traffic signs.For development and testing of modern sophisticated computer vision systems large and diverse datasets are of the major importance. To test the robustness of algorithms, image data with different moving speeds, camera settings, lighting and weather conditions are especially important.In this work we present a comprehensive, lifelike dataset of traffic sign images collected on the Russian winter roads in varying conditions, which include different weather, camera exposure, illumination and moving speeds. The dataset was annotated in accordance with the Russian traffic code. Annotation results and images are published under open CC BY 4.0 license and can be downloaded from the project website: http://oscar.skoltech.ru/.

Pneumatic actuators have gained huge popularity in the field of soft robotics. A class of this kind of devices exploits inflatable thin membranes which generate a desired displacement upon inflation, but often without providing sufficient force/torque to perform their task. In this paper, we propose a novel actuator combining a membrane and a rigid foldable structure. Experimental tests show that such INFlatable ORigami Actuator (INFORA) is characterized by relatively high stiffness compared to other actuators of the same class. We provide a mathematical model to be used for design purposes and we describe the fabrication process. In addition, we show how the INFORA can be used to build a tendril-like structure capable of performing grasping tasks.

In this paper, we introduce a probabilistic framework for INertial Lidar Localisation And MApping (IN2LAMA). Most of today's lidars are based on spinning mechanisms that do not capture snapshots of the environment. As a result, movement of the sensor can occur while scanning. Without a good estimation of this motion, the resulting point clouds might be distorted. In the lidar mapping literature, a constant velocity motion model is commonly assumed. This is an approximation that does not necessarily always hold. The key idea of the proposed framework is to exploit preintegrated measurements over upsampled inertial data to handle motion distortion without the need for any explicit motion-model. It tightly integrates inertial and lidar data in a batch on-manifold optimisation formulation. Using temporally precise upsampled preintegrated measurement allows frame-to-frame planar and edge features association. Moreover, features are re-computed when the estimate of the state changes, consolidating front-end and back-end interaction. We validate the effectiveness of the approach through simulated and real data.