ICRA 2019 Papers

Bundle adjustment plays a vital role in feature-based monocular SLAM. In many modern SLAM pipelines, bundle adjustment is performed to estimate the 6DOF camera trajectory and 3D map (3D point cloud) from the input feature tracks. However, two fundamental weaknesses plague SLAM systems based on bundle adjustment. First, the need to carefully initialise bundle adjustment means that all variables, in particular the map, must be estimated as accurately as possible and maintained over time, which makes the overall algorithm cumbersome. Second, since estimating the 3D structure (which requires sufficient baseline) is inherent in bundle adjustment, the SLAM algorithm will encounter difficulties during periods of slow motion or pure rotational motion. We propose a different SLAM optimisation core: instead of bundle adjustment, we conduct rotation averaging to incrementally optimise only camera orientations. Given the orientations, we estimate the camera positions and 3D points via a quasi-convex formulation that can be solved efficiently and globally optimally. Our approach not only obviates the need to estimate and maintain the positions and 3D map at keyframe rate (which enables simpler SLAM systems), it is also more capable of handling slow motions or pure rotational motions.

Prospection is key to solving challenging problems in new environments, but it has not been deeply explored as applied to task planning for perception-driven robotics. We propose visual robot task planning, where we take in an input image and must generate a sequence of high-level actions and associated observations that achieve some task. In this paper, we describe a neural network architecture and associated planning algorithm that (1) learns a representation of the world that can generate prospective futures, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) evaluates these actions via a variant of Monte Carlo Tree Search to find a viable solution to a particular problem. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information, and used this to generate and visualize task plans on held-out examples of a block-stacking simulation.

What is a good visual representation for navigation? We study this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a previously unseen environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues. We propose to use semantic segmentation and detection masks as observations obtained by state-of-the-art computer vision algorithms and use a deep network to learn the navigation policy. The availability of equitable representations in simulated environments enables joint training using real and simulated data and alleviates the need for domain adaptation or domain randomization commonly used to tackle the sim-to-real transfer of the learned policies. Both the representation and the navigation policy can be readily applied to real non-synthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach successfully gets to the target in 54% of the cases in unexplored environments, compared to 46% for a non-learning based approach, and 28% for a learning-based baseline.

One of the main challenges in sampling-based motion planners is to find an efficient sampling strategy. While methods such as Rapidly-exploring Random Tree (RRT) have shown to be more reliable in complex environments than optimization-based methods, they often require longer planning times, which reduces their usability for real-time applications. Recently, biased sampling methods have shown to remedy this issue. For example Gaussian Mixture Models (GMMs) have been used to sample more efficiently in feasible regions of the configuration space. Once the GMM is learned, however, this approach does not adapt its biases to individual planning scene during inference. Hence, we propose in this work a more efficient sampling strategy to further bias the GMM based on visual input upon query. We employ an autoencoder trained entirely in simulation to extract features from depth images and use the latent representation to adjust the weights of each mixture components in the GMM. We show empirically that this improves the sampling efficiency of an RRT motion planner in both real and simulated scenes.

This paper deals with the task of ego-vehicle localization at intersections, a significant task in autonomous road driving. We propose an online vision-based method that can hence be applied if the intersection is visible. It relies on stereo images and on a coarse street-level pose estimate, used to retrieve intersection data from a digital map service. Pixel-level semantic segmentation, and 3D reconstruction from state-of-the art Deep Neural Networks are coupled with an intersection model; this allows good positioning accuracy compared to the state-of-the-art in this task. To demonstrate the effectiveness of the method and make it possible to compare it with other methods, an extensive activity has been conducted in order to set up a dataset of approaches to an intersection, which has then been used to benchmark the proposed method. The dataset is made available to the community, and it currently includes more than forty intersection approaches, from KITTI. Another important contribution of the paper is the definition of criteria for the comparison of different methods, on recorded datasets. The proposed method achieves nearly sub-meter accuracy in difficult real conditions.

Personalized stent graft is designed to treat Abdominal Aortic Aneurysms (AAA). Due to the individual difference in arterial structures, stent graft has to be custom made for each AAA patient. Robotic platforms for autonomous personalized stent graft manufacturing have been proposed in recently which rely upon stereo vision systems for coordinating multiple robots for fabricating customized stent grafts. This paper proposes a novel hybrid vision system for real-time visual-sevoing for personalized stent-graft manufacturing. To coordinate the robotic arms, this system is based on projecting a dynamic stereo microscope coordinate system onto a static wide angle view stereo webcam coordinate system. The multiple stereo camera configuration enables accurate localization of the needle in 3D during the sewing process. The scale-invariant feature transform (SIFT) method and color filtering are implemented for stereo matching and feature identifications for object localization. To maintain the clear view of the sewing process, a visual-servoing system is developed for guiding the stereo microscopes for tracking the needle movements. The deep deterministic policy gradient (DDPG) reinforcement learning algorithm is developed for real-time intelligent robotic control. Experimental results have shown that the robotic arm can learn to reach the desired targets autonomously.

This paper presents an approach for autonomous underwater robots to visually detect and identify divers. The proposed approach enables an autonomous underwater robot to detect multiple divers in a visual scene and distinguish between them. Such methods are useful for robots to identify a human leader, for example, in multi-human/robot teams where only designated individuals are allowed to command or lead a team of robots. Initial diver identification is performed using the Faster R-CNN algorithm with a region proposal network which produces bounding boxes around the divers' locations. Subsequently, a suite of spatial and frequency domain descriptors are extracted from the bounding boxes to create a feature vector. A K-Means clustering algorithm, with k set to the number of detected bounding boxes, thereafter identifies the detected divers based on these feature vectors. We evaluate the performance of the proposed approach on video footage of divers swimming in front of a mobile robot and demonstrate its accuracy.

This paper presents a coverage control strategy for teams of quadcopters that ensures that no area is left unsurveyed in between the fields of view of the visual sensors mounted on the quadcopters. We present a locational cost that quantifies the team's coverage performance according to the sensors' performance function. Moreover, the cost function penalizes overlaps between the fields of view of the different sensors, with the objective of increasing the area covered by the team. A distributed control law is derived for the quadcopters so that they adjust their position and zoom according to the direction of ascent of the cost. Control barrier functions are implemented to ensure that, while executing the gradient ascent control law, no holes appear in between the fields of view of neighboring robots. The performance of the algorithm is evaluated in simulated experiments.

Monocular simultaneous localisation and mapping (SLAM) is a cheap and energy efficient way to enable Unmanned Aerial Vehicles (UAVs) to safely navigate managed forests and gather data crucial for monitoring tree health. SLAM research, however, has mostly been conducted in structured human environments, and as such is poorly adapted to unstructured forests. In this paper, we compare the performance of state of the art monocular SLAM systems on forest data and use visual appearance statistics to characterise the differences between forests and other environments, including a photorealistic simulated forest. We find that SLAM systems struggle with all but the most straightforward forest terrain and identify key attributes (lighting changes and in-scene motion) which distinguish forest scenes from “classic” urban datasets. These differences offer an insight into what makes forests harder to map and open the way for targeted improvements. We also demonstrate that even simulations that look impressive to the human eye can fail to properly reflect the difficult attributes of the environment they simulate, and provide suggestions for more closely mimicking natural scenes.

In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.

We consider the task of controlling a quadrotor to hover in front of a freely moving user, using input data from an onboard camera. On this specific task we compare two widespread learning paradigms: a mediated approach, which learns a high-level state from the input and then uses it for deriving control signals; and an end-to-end approach, which skips high-level state estimation altogether. We show that despite their fundamental difference, both approaches yield equivalent performance on this task. We finally qualitatively analyze the behavior of a quadrotor implementing such approaches.

Most existing controllers for Car-Like Mobile Robots (CLMR) are designed to handle dynamic effects by decoupling speed and steering controls, also assume that full states are accessible, which are unrealistic for real-world applications. This paper presents a combined speed and steering control system for CLMR. To provide the essential state for the controller, a newly developed visual algorithm is adopted for estimating the high-update rate longitudinal and lateral velocities of the robot which cannot be accurately measured by wheel encoders due to the skidding and slipping effects. The stability of the proposed system can be guaranteed by Lyapunov method since the velocity estimation error, the speed tracking error and the lateral deviation converging to zero simultaneously. Real-world experiments are conducted on an electric autonomous tractor with online estimation to demonstrate the feasibility of the approach.

This paper reports a vision-based microfluidic system for automated, high-speed sorting of the nematode worm C. elegans. Exceeding the capabilities of conventional worm sorting microfluidic devices purely relying on passive sorting mechanisms, our system is capable of accurate measurement of the worm body length/width and active sorting of worms with the desired sizes from a mixture of worms at different developmental stages. This feature is enabled by the combination of vision-based worm detection and sizing algorithms and automated on-chip worm manipulation. A double-layer microfluidic device with computer-controlled pneumatic valves is developed for sequential loading, trapping, imaging, and sorting of single worms based on vision-based worm size measurement results. To keep the system operation robust, vision-based algorithms for detecting multi-worm loading and worm size measurement failure have also been developed. We conducted sorting experiments on 319 worms and achieve an average sorting speed of 10.4 worms per minute (5.8 s/worm) with an operation success rate of 90.3%. This system will facilitate worm biology studies where body size measurement and size-based sorting of many worms are needed.

Among the currently available grasp-type selection techniques for hand prostheses, there is a distinct lack of intuitive, robust, low-latency solutions. In this paper we investigate the use of a portable, forearm-mounted, video-based technique for the prediction of hand-grasp preshaping for arbitrary objects. The purpose of this system is to automatically select the grasp-type for the user of the prosthesis, potentially increasing ease-of-use and functionality. This system can be used to supplement and improve existing control strategies, such as surface electromyography (sEMG) pattern recognition, for prosthetic and orthotic devices. We designed and created a suitable dataset consisting of RGB-D video data for 2212 grasp examples split evenly across 7 classes; 6 grasps commonly used in activities of daily living, and an additional no-grasp category. We processed and analyzed the dataset using several state-of-the-art deep learning architectures. Our selected model shows promising results for realistic, intuitive, real-world use, reaching per-frame accuracies on video sequences of up to 95.90% on the validation set. Such a system could be integrated into the palm of a hand prosthesis, allowing an automatic prediction of the grasp-type without requiring any special movements or aiming by the user.

Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance “teacher” network provides pseudo-labels to adapt an appearance “student” network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Our proposed adaptation method outperforms the state-of-theart on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.

In this contribution the problem of vibration control is studied on the basis of a fundamental oscillatory system consisting of a mass spring system and an additional mass. The proposed control strategy couples the orbits of the two masses such that both masses stop, while simultaneously stabilizing the second mass to a desired equilibrium. Using a coordinate and input transformation, the control strategy is directly transferred to an n-link manipulator mounted on a base with linear translational stiffness. Using semidefinite Lyapunov functions and a conditional stability argument, it is shown that the proposed control strategy damps out base vibrations, while additionally achieving a desired configuration in the task-space. Finally, the proposed method is compared to a state-of-the-art approach using numerical simulations.

When experiencing disturbances during locomotion, human beings use several strategies to maintain balance, e.g. changing posture, modulating step frequency and location. However, when it comes to the gait generation for humanoid robots, modifying step time or body posture in real time introduces nonlinearities in the walking dynamics, thus increases the complexity of the planning. In this paper, we propose a two-layer hierarchical optimization framework to address this issue and provide the humanoids with the abilities of step time and step location adjustment, Center of Mass (CoM) height variation and angular momentum adaptation. In the first layer, times and locations of consecutive two steps are modulated online based on the current CoM state using the Linear Inverted Pendulum Model. By introducing new optimization variables to substitute the hyperbolic functions of step time, the derivatives of the objective function and feasibility constraints are analytically derived, thus reduces the computational cost. Then, taking the generated horizontal CoM trajectory, step times and step locations as inputs, CoM height and angular momentum changes are optimized by the second layer nonlinear model predictive control. This whole procedure will be repeated until the termination condition is met. The improved recovery capability under external disturbances is validated in simulation studies.

While much research has been conducted into the generation of smooth trajectories for underactuated unstable aerial vehicles such as quadrotors, less attention has been paid to the application of the same techniques to ground based omnidirectional dynamically balancing robots. These systems have more control authority over their linear accelerations than aerial vehicles, meaning trajectory smoothness is less of a critical design parameter. However, when operating in indoor environments these systems must often adhere to relatively low velocity constraints, resulting in very conservative trajectories when enforced using existing trajectory optimisation methods. This paper makes two contributions; this gap is bridged by the extension of these existing methods to create a fast velocity constrained trajectory planner, with trajectory timing characteristics derived from the optimal minimum-time solution of a simplified acceleration and velocity constrained model. Next, a differentially flat model of an omnidirectional balancing robot utilizing a collinear Mecanum drive is derived, which is used to allow an experimental prototype of this configuration to smoothly follow these velocity constrained trajectories.

Deep learning has revolutionized the ability to learn “end-to-end” autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-topoint navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.