IROS 2018 Papers

We address the problem of learning vision-based, collision-avoiding, and target-selecting controllers in 3D, specifically in underwater environments densely populated with coral reefs. Using a highly maneuverable, dynamic, six-legged (or flippered) vehicle to swim underwater, we exploit real time visual feedback to make close-range navigation decisions that would be hard to achieve with other sensors. Our approach uses computer vision as the sole mechanism for both collision avoidance and visual target selection. In particular, we seek to swim close to the reef to make observations while avoiding both collisions and barren, coral-deprived regions. To carry out path selection while avoiding collisions, we use monocular image data processed in real time. The proposed system uses a convolutional neural network that takes an image from a forward-facing camera as input and predicts unscaled and relative path changes. The network is trained to encode our desired obstacle-avoidance and reef-exploration objectives via supervised learning from human-labeled data. The predictions from the network are transformed into absolute path changes via a combination of a temporally-smoothed proportional controller for heading targets and a low-level motor controller. This system enables safe and autonomous coral reef navigation in underwater environments. We validate our approach using an untethered and fully autonomous robot swimming through coral reef in the open ocean. Our robot successfully traverses 1000 m of the ocean floor collision-free while collecting close-up footage of coral reefs.

Adstract- We present an unsupervised deep neural network approach to the fusion of RGB-D imagery with inertial measurements for absolute trajectory estimation. Our network, dubbed the Visual-Inertial-Odometry Learner (VIOLearner), learns to perform visual-inertial odometry (VIO) without inertial measurement unit (IMU) intrinsic parameters (corresponding to gyroscope and accelerometer bias or white noise) or the extrinsic calibration between an IMU and camera. The network learns to integrate IMU measurements and generate hypothesis trajectories which are then corrected online according to the Jacobians of scaled image projection errors with respect to a spatial grid of pixel coordinates. We evaluate our network against state-of-the-art (SOA) visual-inertial odometry, visual odometry, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI Odometry dataset [1] and demonstrate competitive odometry performance.

Deployment of aerial robotic vehicles for real world tasks such as home deliveries, close range aerial inspection, etc., require robotic vehicles to fly through complex and cluttered 3D environments such as forests, shrubbery or into balconies, garages, or sheds. Dense high-speed optical flow can provide real-time motion cues for obstacle avoidance that does not require 3D full reconstruction of the environment. However, classical reactive control does not `look ahead' and tends to bounce off obstacles rather than generating a smooth trajectory that anticipates and avoids upcoming obstacles. In this paper, we consider deriving a fully image based control criteria that forward predicts a cylinder of free space into the image flow representation of the environment and steers the vehicle by manoeuvering this cylinder through the upcoming environment. The length and radius of the cylinder provide a guarantee that the vehicle can indeed fly through the space identified and the fact that it is predicted forward into the environment leads to smooth anticipation of upcoming obstacles. Results are obtained for a quadrotor flying autonomously through a forest environment.

We address the problem of visually monitoring a terrain path using ground and aerial robots. This is a coupled problem that involves computation of a guard set for the environment and route planning for a heterogeneous group of robots through the points in the guard set. A terrain path that needs to be monitored can be transformed to generate a 1.5D terrain and robot paths can be modeled as chain visible curves to the terrain to ensure visibility. To efficiently monitor this 1.5D terrain, we present two solutions - a dynamic programming approach that finds the optimal solution but is slower and a integer linear programming solution that is faster in practice and that can take more constraints into account. We perform extensive simulations and do a comparative analysis of the two solution techniques.

Robots hold promise in many scenarios involving outdoor use, such as search-and-rescue, wildlife management, and collecting data to improve environment, climate, and weather forecasting. However, autonomous navigation of outdoor trails remains a challenging problem. Recent work has sought to address this issue using deep learning. Although this approach has achieved state-of-the-art results, the deep learning paradigm may be limited due to a reliance on large amounts of annotated training data. Collecting and curating training datasets may not be feasible or practical in many situations, especially as trail conditions may change due to seasonal weather variations, storms, and natural erosion. In this paper, we explore an approach to address this issue through virtual-to-real-world transfer learning using a variety of deep learning models trained to classify the direction of a trail in an image. Our approach utilizes synthetic data gathered from virtual environments for model training, bypassing the need to collect a large amount of real images of the outdoors. We validate our approach in three main ways. First, we demonstrate that our models achieve classification accuracies upwards of 95% on our synthetic data set. Next, we utilize our classification models in the control system of a simulated robot to demonstrate feasibility. Finally, we evaluate our models on real-world trail data and demonstrate the potential of virtual-to-real-world transfer learning.

In this paper, we propose a mapping approach that constructs a globally deformable virtual occupancy grid map (VOG-map) based on local submaps. Such a representation allows pose graph SLAM systems to correct globally accumulated drift via loop closures while maintaining free space information for the purpose of path planning. We demonstrate use of such a representation for implementing an underwater SLAM system in which the robot actively plans paths to generate accurate 3D scene reconstructions. We evaluate performance on simulated as well as real-world experiments. Our work furthers capabilities of mobile robots actively mapping and exploring unstructured, three dimensional environments.

We address the problem of interactively controlling the workspace of a mobile robot to ensure a human-aware navigation. This is especially of relevance for non-expert users living in human-robot shared spaces, e.g. home environments, since they want to keep the control of their mobile robots, such as vacuum cleaning or companion robots. Therefore, we introduce virtual borders that are respected by a robot while performing its tasks. For this purpose, we employ a RGB-D Google Tango tablet as human-robot interface in combination with an augmented reality application to flexibly define virtual borders. We evaluated our system with 15 non-expert users concerning accuracy, teaching time and correctness and compared the results with other baseline methods based on visual markers and a laser pointer. The experimental results show that our method features an equally high accuracy while reducing the teaching time significantly compared to the baseline methods. This holds for different border lengths, shapes and variations in the teaching process. Finally, we demonstrated the correctness of the approach, i.e. the mobile robot changes its navigational behavior according to the user-defined virtual borders.

This paper discusses video motion capture, namely, 3D reconstruction of human motion from multi-camera images. After the Part Confidence Maps are computed from each camera image, the proposed spatiotemporal filter is applied to deliver the human motion data with accuracy and smoothness for human motion analysis. The spatiotemporal filter uses the human skeleton and mixes temporal smoothing in two-time inverse kinematics computations. The experimental results show that the mean per joint position error was 26.1mm for regular motions and 38.8mm for inverted motions.

During percutaneous interventions in the brain, puncturing a vessel can cause life threatening complications. To avoid such a risk, current research has been directed towards the development of steerable needles. However, there is a risk that vessels of a size which is close to or smaller than the resolution of commonly used preoperative imaging modalities (0.59 × 0.59 × 1 mm) would not be detected during procedure planning, with a consequent increase in risk to the patient. In this work, we present a novel ensemble of forward looking sensors based on laser Doppler flowmetry, which are embedded within a biologically inspired steerable needle to enable vessel detection during the insertion process. Four Doppler signals are used to classify the pose of a vessel in front of the advancing needle with a high degree of accuracy (2° and 0.1 mm RMS errors), where relative measurements between sensors are used to correct for ambiguity. By using a robotic assisted needle insertion process, and thus a precisely controlled insertion speed, we also demonstrate how the setup can be used to discriminate between tissue bulk motion and vessel motion. In doing so, we describe a sensing apparatus applicable to a variety of needle steering systems, with the potential to eliminate the risk of hemorrhage during percutaneous procedures.

Walking ability is critically important for pediatric health, well-being, and independence. Children with cerebral palsy (CP), the most prevalent cause of pediatric physical disability, often present pathological gait patterns that negatively impact walking capacity. Reduced function of the muscles surrounding the ankle joint in those with CP also greatly increases the energy cost of transport leading to reduce mobility. Ankle-foot-orthoses show limited effectiveness for clinically relevant improvement in gait mechanics, while orthopedic surgery, muscle injections and physical therapy are unable to completely restore gait function. While wearable exoskeletons hold promise for gait rehabilitation, appropriately controlling the timing and magnitude of powered assistance across individuals and conditions remains a considerable challenge. This work seeks to address this challenge through the design and initial clinical verification of a simple ankle exoskeleton control scheme designed to reduce the metabolic cost of transport during walking in an individual with CP. Preliminary experimental results from instrumented gait analysis following 5 training visits demonstrated a 45% increase in positive ankle power and a 16% reduction in net metabolic rate during walking with the exoskeleton providing powered plantar-flexion assistance compared to walking without the exoskeleton. Future work will expand this investigation to a larger cohort of individuals with CP and across additional modes of locomotion.

Recent developments in Mobility-on-Demand (MoD) systems have demonstrated the potential of road vehicles as an efficient mode of urban transportation Newly developed algorithms can compute vehicle routes in real-time for batches of requests and allow for multiple requests to share vehicles. These algorithms have primarily focused on optimally producing vehicle schedules to pick up and drop off requests. The redistribution of idle vehicles to areas of high demand, known as rebalancing, on the contrary has received little attention in the context of ride-sharing. In this paper, we present a method to rebalance idle vehicles in a ride-sharing enabled MoD fleet. This method consists of an algorithm to optimally partition the fleet operating area into rebalancing regions, an algorithm to determine a real-time demand estimate for every region using incoming requests, and an algorithm to optimize the assignment of idle vehicles to these rebalancing regions using an integer linear program. Evaluation with historical taxi data from Manhattan shows that we can service 99.8% of taxi requests in Manhattan using 3000 vehicles with an average waiting time of 57.4 seconds and an average in-car delay of 13.7 seconds. Moreover, we can achieve a higher service rate using 2000 vehicles than prior work achieved with 3000. Furthermore, with a fleet of 3000 vehicles, we reduce the average travel delay by 86%, the average waiting time by 37%, and the amount of ignored requests by 95% compared to earlier work at the expense of an increased distance travelled by the fleet.

Adapting the perceptual capabilities of mobile robots to new objects or new environments can be a time consuming task. In this paper we focus on specializing perceptual capabilities of mobile robots to new objects through a knowledge based, virtual scene rendering approach. Episodic memories of a robotic agent, gathered during the execution of a task are considered to be the main "theme". Variations of this theme are then generated based on background knowledge about the objects and data gathered with the purpose of learning new models for detection and recognition. We demonstrate the applicability of our approach by adapting the perceptual capabilities of a mobile robot performing pick and place tasks, to recognize new sets of objects.

This paper introduces a new method for end-to-end training of deep neural networks (DNNs) and evaluates it in the context of autonomous driving. DNN training has been shown to result in high accuracy for perception to action learning given sufficient training data. However, the trained models may fail without warning in situations with insufficient or biased training data. In this paper, we propose and evaluate a novel architecture for self-supervised learning of latent variables to detect the insufficiently trained situations. Our method also addresses training data imbalance, by learning a set of underlying latent variables that characterize the training data and evaluate potential biases. We show how these latent distributions can be leveraged to adapt and accelerate the training pipeline by training on only a fraction of the total dataset. We evaluate our approach on a challenging dataset for driving. The data is collected from a full-scale autonomous vehicle. Our method provides qualitative explanation for the latent variables learned in the model. Finally, we show how our model can be additionally trained as an end-to-end controller, directly outputting a steering control command for an autonomous vehicle.

In this paper, a method for variable admittance control in human-robot cooperation is proposed. A multilayer feedforward neural network is designed using the Cartesian velocity of the robot and the applied force by the operator as its inputs to modify online the virtual damping of the admittance controller. The neural network is trained online using the error backpropagation algorithm based on the error between the velocity of the minimum jerk trajectory model and the measured velocity of the robot. The performance of the proposed controller and the NN generalization ability are evaluated by conducting a point-to-point cooperative motion with multiple subjects using the KUKA LWR robot.

Unsupervised video prediction is a very challenging task due to the complexity and diversity in natural scenes. Prior works directly predicting pixels or optical flows either have the blurring problem or require additional assumptions. We highlight that the crux for video frame prediction lies in precisely capturing the inter-frame variations which encompass the movement of objects and the evolution of the surrounding environment. We then present an unsupervised video prediction framework - Variation Network (VarNet) to directly predict the variations between adjacent frames which are then fused with current frame to generate the future frame. In addition, we propose an adaptively re-weighting mechanism for loss function to offer each pixel a fair weight according to the amplitude of its variation. Extensive experiments for both short-term and long-term video prediction are implemented on two advanced datasets - KTH and KITTI with two evaluating metrics - PSNR and SSIM. For the KTH dataset, the VarNet outperforms the state-of-the-art works up to 11.9% on PSNR and 9.5% on SSIM. As for the KITTI dataset, the performance boosts are up to 55.1% on PSNR and 15.9% on SSIM. Moreover, we verify that the generalization ability of our model excels other state-of-the-art methods by testing on the unseen CalTech Pedestrian dataset after being trained on the KITTI dataset. Source code and video are available at https://github.com/jinbeibei/VarNet.

We propose VLASE, a framework to use semantic edge features from images to achieve on-road localization. Semantic edge features denote edge contours that separate pairs of distinct objects such as building-sky, road-sidewalk, and building-ground. While prior work has shown promising results by utilizing the boundary between prominent classes such as sky and building using skylines, we generalize this to consider 19 semantic classes. We extract semantic edge features using CASENet architecture and utilize VLAD framework to perform image retrieval. We achieve improvement over state-of-the-art localization algorithms such as SIFT-VLAD and its deep variant NetVLAD. Ablation study shows the importance of different semantic classes, and our unified approach achieves better performance compared to individual prominent features such as skylines. We also introduce SLC Marathon dataset, a challenging dataset covering most of Salt Lake City with sufficient lighting variations.

This paper proposes a novel gripper or end-effector named VARO-fi (VARiable Orientable fingers with translation), with the aim of obtaining human like prehensile manoeuvre such as, in-hand manipulation. The 4 fingered VARO-fi consists of 9 degrees of freedom and it can perform several in-hand manipulation tasks which have been described in this paper. Moreover, the gripper is a simplification of previously proposed gripper platform called Dexclar. The derivation of VARO-fi has been presented and its capabilities have been demonstrated by experiments. Although a generic convex payload is considered as a primitive in the design of VARO-fi however, it is capable to address manipulation for other regular shaped payloads, which has been proven by experiments. A comparison is also illustrated in order to underline the strength of the novel gripper with respect to the state of the art.

This paper describes a re-descriptive approach to the efficient acquisition of ever higher level and more precise utility models within the motivational system (MotivEn) of a cognitive architecture. The approach is based on a two-step process whereby, as a first step, simple imprecise sensor correlation related utility models are obtained from the interaction traces of the robot. These utility models allow the robot to increase the frequency of achieving goals, and thus, provide lots of traces that can be used to try to train precise value functions implemented as artificial neural networks. The approach is tested experimentally on a real robotic setup that involves the coordination of two robots.

We present a human-subjects study approach that supports the analysis of the manipulation performance of robotic hands that have the same morphology but different actuation and compliance. Specifically, we use this approach to analyze three different types of hands (one underactuated, one fully actuated, one fully actuated with compliant distal joints) as they are used to perform two manipulation tasks. The first task uses a power grasp (spraying with a spray bottle), the second a precision grasp (tracing a line on a bowl with a pen). We show that compliance in the distal joints significantly improves performance and task completion. We also show that humans choose significantly different poses for the same task when using a fully-actuated versus underactuated hand, which also results in superior task performance. Our results suggest that humans use a combination of under-actuated and fully-actuated techniques, which when used on robotic systems would also improve their performance on manipulation tasks.