ICRA 2019 Papers

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. We present results in simulation and on a real robot.

In this paper, we propose an approach to detect incipient slip, i.e. predict slip, by using a high-resolution vision-based tactile sensor, GelSlim. The sensor dynamically captures the tactile imprints of the grasped object and their changes with a soft gel pad. The method assumes the object is mostly rigid and expects the motion of object's imprint on the sensor surface to be a 2D rigid-body motion. We use the deviation of the true motion field from that of a 2D planar rigid transformation as a measure of slip. The output is a dense slip field which we monitor in real time to detect when small areas of the contact patch start to slip (incipient slip). The method can detect incipient slip in any direction without any prior knowledge of the object at 24 Hz. We test the method on 10 objects for 240 times and achieve 86.25% detection accuracy with the vast majority of failure cases occurring when grasping highly deformable objects. We further show how the slip feedback can be used to adjust the gripping force to avoid slip with a closed-loop bottle-cap screwing and unscrewing experiment. The method can be used to enable many manipulation tasks in both structured and unstructured environments.

In this paper, a magnetic-field-inspired robot navigation is used to navigate an under-actuated quad-copter towards the desired position amidst previously-unknown arbitrary-shaped convex obstacles. Taking inspiration from the phenomena of magnetic field interaction with charged particles observed in nature, the algorithm outperforms previous reactive navigation algorithms for flying robots found in the literature as it is able to reactively generate motion commands relying only on a local sensory information without prior knowledge of the obstacles' shape or location and without getting trapped in local minima configurations. The application of the algorithm in a dynamic model of quadcopter system and in the realistic model of the commercial AscTec Pelican micro-aerial vehicle confirm the superior performance of the algorithm.

Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six birds eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

Place recognition and loop closure detection are challenging for long-term visual navigation tasks. SeqSLAM is considered to be one of the most successful approaches to achieve long-term localization under varying environmental conditions and changing viewpoints. SeqSLAM uses a brute-force sequential matching method, which is computationally intensive. In this work, we introduce a multi-resolution sampling-based global visual place recognition method (MRS-VPR), which can significantly improve the matching efficiency and accuracy in sequential matching. The novelty of this method lies in the coarse-to-fine searching pipeline and a particle filter-based global sampling scheme, that can balance the matching efficiency and accuracy in the long-term navigation task. Moreover, our model works much better than SeqSLAM when the testing sequence is over a much smaller time scale than the reference sequence. Our experiments demonstrate that MRSVPR is efficient in locating short temporary trajectories within long-term reference ones without compromising on the accuracy compared to SeqSLAM.

We propose a new multi-instance dynamic RGB-D SLAM system using an object-level octree-based volumetric representation. It can provide robust camera tracking in dynamic environments and at the same time, continuously estimate geometric, semantic, and motion properties for arbitrary objects in the scene. For each incoming frame, we perform instance segmentation to detect objects and refine mask boundaries using geometric and motion information. Meanwhile, we estimate the pose of each existing moving object using an object-oriented tracking method and robustly track the camera pose against the static scene. Based on the estimated camera pose and object poses, we associate segmented masks with existing models and incrementally fuse corresponding colour, depth, semantic, and foreground object probabilities into each object model. In contrast to existing approaches, our system is the first system to generate an object-level dynamic volumetric map from a single RGB-D camera, which can be used directly for robotic tasks. Our method can run at 2-3 Hz on a CPU, excluding the instance segmentation part. We demonstrate its effectiveness by quantitatively and qualitatively testing it on both synthetic and real-world sequences.

A novel nonlinear incremental optimization algorithm MH-iSAM2 is developed to handle ambiguity in simultaneous localization and mapping (SLAM) problems in a multi-hypothesis fashion. It can output multiple possible solutions for each variable according to the ambiguous inputs, which is expected to greatly enhance the robustness of autonomous systems as a whole. The algorithm consists of two data structures: an extension of the original Bayes tree that allows efficient multi-hypothesis inference, and a Hypo-tree that is designed to explicitly track and associate the hypotheses of each variable as well as all the inference processes for optimization. With our proposed hypothesis pruning strategy, MH-iSAM2 enables fast optimization and avoids the exponential growth of hypotheses. We evaluate MH-iSAM2 using both simulated datasets and real-world experiments, demonstrating its improvements on the robustness and accuracy of SLAM systems.

A local map module is often implemented in modern VO/VSLAM systems to improve data association and pose estimation. Conventionally, the local map contents are determined by co-visibility. While co-visibility is cheap to establish, it utilizes the relatively-weak temporal prior (i.e. seen before, likely to be seen now), therefore admitting more features into the local map than necessary. This paper describes an enhancement to co-visibility local map building by incorporating a strong appearance prior, which leads to a more compact local map and latency reduction in downstream data association. The appearance prior collected from the current image influences the local map contents: only the map features visually similar to the current measurements are potentially useful for data association. To that end, mapped features are indexed and queried with Multi-index Hashing (MIH). An online hash table selection algorithm is developed to further reduce the query overhead of MIH and the local map size. The proposed appearance-based local map building method is integrated into a state-of-the-art VO/VSLAM system. When evaluated on two public benchmarks, the size of the local map, as well as the latency of real-time pose tracking in VO/VSLAM are significantly reduced. Meanwhile, the VO/VSLAM mean performance is preserved or improves.

A key capability for autonomous underground mining vehicles is real-time accurate localisation. While significant progress has been made, currently deployed systems have several limitations ranging from dependence on costly additional infrastructure to failure of both visual and range-sensor-based techniques in highly aliased or visually challenging environments. In our previous work, we presented a lightweight coarse vision-based localisation system that could map and then localise to within a few metres in an underground mining environment. However, this level of precision is insufficient for providing a cheaper, more reliable vision-based automation alternative to current range sensor-based systems. Here we present a new precision localisation system dubbed “LookUP”, which learns a neural-network-based pixel sampling strategy for estimating homographies based on ceiling-facing cameras without requiring any manual labelling. This new system runs in real time on limited computation resource and is demonstrated on two different underground mine sites, achieving real time performance at ~5 frames per second and a much improved average localisation error of ~1.2 metre.

Visual place recognition (VPR) - the act of recognizing a familiar visual place - becomes difficult when there is extreme environmental appearance change or viewpoint change. Particularly challenging is the scenario where both phenomena occur simultaneously, such as when returning for the first time along a road at night that was previously traversed during the day in the opposite direction. While such problems can be solved with panoramic sensors, humans solve this problem regularly with limited field-of-view vision and without needing to constantly turn around. In this paper, we present a new depth- and temporal-aware visual place recognition system that solves the opposing viewpoint, extreme appearance-change visual place recognition problem. Our system performs sequence-to-single frame matching by extracting depth-filtered keypoints using a state-of-the-art depth estimation pipeline, constructing a keypoint sequence over multiple frames from the reference dataset, and comparing these keypoints to the keypoints extracted from a single query image. We evaluate the system on a challenging benchmark dataset and show that it consistently outperforms state-of-the-art techniques. We also develop a range of diagnostic simulation experiments that characterize the contribution of depth-filtered keypoint sequences with respect to key domain parameters including the degree of appearance change and camera motion.

We tackle the long-term prediction of scene evolution in a complex downtown scenario for automated driving based on Lidar grid fusion and recurrent neural networks (RNNs). A bird's eye view of the scene, including occupancy and velocity, is fed as a sequence to a RNN which is trained to predict future occupancy. The nature of prediction allows generation of multiple hours of training data without the need of manual labeling. Thus, the training strategy and loss function are designed for long sequences of real-world data (unbalanced, continuously changing situations, false labels, etc.). The deep CNN architecture comprises convolutional long short-term memories (ConvLSTMs) to separate static from dynamic regions and to predict dynamic objects in future frames. Novel recurrent skip connections show the ability to predict small occluded objects, i.e. pedestrians, and occluded static regions. Spatio-temporal correlations between grid cells are exploited to predict multimodal future paths and interactions between objects. Experiments also quantity improvements to our previous network, a Monte Carlo approach, and literature.

Locomotion planning for legged systems requires reasoning about suitable contact schedules. The contact sequence and timings constitute a hybrid dynamical system and prescribe a subset of achievable motions. State-of-the-art approaches cast motion planning as an optimal control problem. In order to decrease computational complexity, one common strategy separates footstep planning from motion optimization and plans contacts using heuristics. In this paper, we propose to learn contact schedule selection from high-level task descriptors using Bayesian Optimization. A bi-level optimization is defined in which a Gaussian Process model predicts the performance of trajectories generated by a motion planning nonlinear program. The agent, therefore, retains the ability to reason about suitable contact schedules, while explicit computation of the corresponding gradients is avoided. We delineate the algorithm in its general form and provide results for planning single-legged hopping. Our method is capable of learning contact schedule transitions that align with human intuition. It performs competitively against a heuristic baseline in predicting task appropriate contact schedules.

In a recent study, we developed a minimally actuated wave-like robot and analyzed its kinematics. In this paper, we present the dynamic locomotion analysis of a miniature version of this wave robot. We examine different crawling environments, determine under which conditions it can advance, and evaluate its propulsion force. We first developed two locomotion models to characterize the cases where the robot is crawling between two straight surfaces or over a single flat surface. We specified the conditions in which the robot will advance and the advance time ratio as a function of the friction forces and weight of the robot. Next, we developed highly flexible tube-like shapes that we molded from silicone rubber to experimentally test the forces acting on the robot inside these tubes. Finally, we designed a miniature model of the robot and experimentally validated its crawling conditions (see video).

We address the problem of visual place recognition with perceptual changes. The fundamental problem of visual place recognition is generating robust image representations which are not only insensitive to environmental changes but also distinguishable to different places. Taking advantage of the feature extraction ability of Convolutional Neural Networks (CNNs), we further investigate how to localize discriminative visual landmarks that positively contribute to the similarity measurement, such as buildings and vegetations. In particular, a Landmark Localization Network (LLN) is designed to indicate which regions of an image are used for discrimination. Detailed experiments are conducted on open source datasets with varied appearance and viewpoint changes. The proposed approach achieves superior performance against state-of-the-art methods.

Localizing a vehicle in a map is essential for automated driving and various other robotic applications. This paper addresses the problem of vehicle localization in urban environments. Our approach performs a graph-based sliding window optimization over a set of recent landmark and odometry measurements for fast and accurate vehicle localization on third-party maps. Our work incorporates landmark priors from third-party maps into the estimation problem and shows how to exploit the sliding window formulation for revising data associations. We describe how to construct our factor graph and derive its necessary factors to model the information from the map as a prior over the landmark detections. We implemented our approach on an automated car and thoroughly tested it on real-world data. The experiments suggest that the approach provides highly accurate pose estimates, is fast enough for automated driving applications, and outperforms localization using particle filters.

Tracking of uncontrollable agents in a controlled environment is an important research question for the coordination of controllable and uncontrollable agents and bio-inspired multi-agent control. This paper presents a framework that approaches the multiagent tracking problem from a localization perspective, utilizing a combination of wearable sensors and stationary cameras. Specifically, this framework was applied to localize uncontrollable biological agents (dolphins) in a well defined environment. The biological agents were outfitted with wearable sensors (IMU, speed, depth) and were free to move in their three dimensional habitat. The dynamic data collected by the wearable sensors was supplemented with image data collected using a pair of cameras mounted above the habitat. The framework presented in this paper combines data from these sensor streams to calculate an accurate estimate of the animal's location during extended periods of free movement. The associations between camera detections and tagged agents are handled using a particle filter embedded with a fuzzy observation concept. The platform is readily implementable in similar water / land environments, and is able to handle nonlinear agent dynamics, non-Gaussian noise, and sparse camera observations while maintaining robust agent localization and tracking.

This paper presents the results of the German research project SYMPARTNER (4/2015 - 6/2018), which aimed at developing a functional-emotional, mobile domestic robot companion for elderly people. The paper gives an overview of the developed robot, its system architecture, and essential skills and behaviors required for being a friendly home companion. Based on this, in a long-term field study running from January to June 2018 both technical aspects regarding the practical suitability and robustness of the robot under domestic operating conditions and social scientific questions on usability and acceptance of the robot and the users’ familiarization with their new housemate were evaluated. In the field study, two of these autonomous companion robots were used in 20 senior households in Erfurt (Germany). All participants lived with their robot in their apartments for one week without the need for supervising or supporting persons being present on-site. The tests in 20 single-person households in the age group 62 to 94 years (average 74 years) provided important insights into the special challenges of domesticity from a technical, social scientific, and user-oriented point of view. The results of the study show how seniors can shape their everyday life with a companion robot and how quickly they get used to the new housemate.

Reconfiguring heterogeneous modular robots in which all modules are not identical is much more time consuming than reconfiguring homogeneous ones, because ordinary heterogeneous reconfiguration is a combination of homogeneous transformation and heterogeneous permutation. While linear homogeneous transformation has been accomplished in previous research, linear heterogeneous permutation has not. This paper studies a reconfiguration algorithm for heterogeneous lattice modular robots with linear operation time cost. The algorithm is based on simultaneous tunneling and permutation, where a robot transforms its configuration via tunneling motion while permutation of each module's position is performed simultaneously during the tunneling transformation. To achieve this, we introduce the idea of a transparent meta-module that allows modules belonging to a meta-module to pass through the spaces occupied by other meta-modules. We prove the correctness and completeness of the proposed algorithm for a 2×2×2 cubic meta-module-based connected robot structure. We also show examples of the reconfiguration simulations of heterogeneous modular robots by the proposed algorithm.

Robotic platforms dedicated to powerline inspection are often complex to operate, take several minutes to cross any obstacle, and must be operated by highly trained specialists. To further its goal of massive inspection of its power grid, Hydro-Québec developed an innovative robot that is simple to operate and can be used directly by line maintenance technicians. LineRanger was developed following the field deployment of LineROVer and LineScout but aims at surpassing them in terms of inspection efficiency. With an ingenious and passive obstacle-crossing system, this new robot allows large-scale inspection of bundled-type powerlines, since the obstacle crossing time is considerably reduced. In this paper, details on the robot's key features are presented along with its mathematical analysis, which guarantees its stability on flexible bundles and directly influenced the design. Finally, the LineRanger prototype is presented, with insights about its first field deployments.