ICRA 2018 Papers

We would like robots to be able to safely navigate at high speed, efficiently use local 3D information, and robustly plan motions that consider pose uncertainty of measurements in a local map structure. This is hard to do with previously existing mapping approaches, like occupancy grids, that are focused on incrementally fusing 3D data into a common world frame. In particular, both their fragile sensitivity to state estimation errors and computational cost can be limiting. We develop an alternative framework, NanoMap, which alleviates the need for global map fusion and enables a motion planner to efficiently query pose-uncertainty-aware local 3D geometric information. The key idea of NanoMap is to store a history of noisy relative pose transforms and search over a corresponding set of depth sensor measurements for the minimum-uncertainty view of a queried point in space. This approach affords a variety of capabilities not offered by traditional mapping techniques: (a) the pose uncertainty associated with 3D data can be incorporated in motion planning, (b) poses can be updated (i.e., from loop closures) with minimal computational effort, and (c) 3D data can be fused lazily for the purpose of planning. We provide an open-source implementation of NanoMap, and analyze its capabilities and computational efficiency in simulation experiments. Finally, we demonstrate in hardware its effectiveness for fast 3D obstacle avoidance onboard a quadrotor flying up to 10 m/s.

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance-a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.

Many fields of application could benefit from an accurate registration of measurements of different modalities over a known 3D model. However, aligning a 2D image to a 3D model is a challenging task and is even more complex when the two have a different modality. Most of the 2D/3D registration methods are based on either geometric or dense visual features. Both have their own advantages and their own drawbacks. We propose, in this paper, to mutually exploit the advantages of one feature type to reduce the drawbacks of the other one. For this, an hybrid registration framework has been designed to mutually align geometrical and dense visual features in order to obtain an accurate final 2D/3D alignment. We evaluate and compare the proposed registration method on real data acquired by a robot equipped with several visual sensors. The results highlights the robustness of the method and its ability to produce wide convergence domain and a high registration accuracy.

This study generates and analyzes unsafe human motions that cannot be measured experimentally in laboratories with dynamic consistency. Detailed whole-body motions are generated by a multilayered kinodynamics simulation (MLKD Sim) that uses a detailed digital whole-body human model and a simple motion-representation model that parametrically represents human motion mechanisms. First, we develop the simple motion-representation model that represents human motion and contact force data that are experimentally measured in a laboratory, and we identify this model's parameters based on these experimental data. Forward dynamics computation of this motion-representation model with changing model and/or environmental parameters simulates motion modification as well as a contact force with dynamic consistency. Finally, the mapping function from the motion-representation model's motion to the detailed motion identified from the experimental data is used to reconstruct the detailed whole-body motion. MLKD Sim reconstructs a vertical contact force with average error of 2.18E+02 N, center of mass trajectory with average error of 3.31E-02 m, ankle joint angle with average error of 1.11E-01 rad (2.95E+00%), and ankle joint torque with average error of 6.13E+01 Nm (1.93E+01%). Unsafe motion simulation results show that the physical load on the hip, knee, and ankle joints increases by 9.23E+01%, 5.42E+02%, and 1.45E+02% respectively with 0.5-m level difference in a running surface. These results imply that when sprinting in an unknown environment, we need to protect, in order, the knee ankle, and hip joints. This study conducts detailed dynamics and kinematics analysis of unsafe human motions that cannot be measured experimentally in laboratories to prevent injuries, falls, and fatigue, and these results should find applications in the fields of medicine and welfare.

Child-robot interaction is an interdisciplinary research area that has been attracting growing interest, primarily focusing on edutainment applications. A crucial factor to the successful deployment and wide adoption of such applications remains the robust perception of the child's multi-modal actions, when interacting with the robot in a natural and untethered fashion. Since robotic sensory and perception capabilities are platform-dependent and most often rather limited, we propose a multiple Kinect-based system to perceive the child-robot interaction scene that is robot-independent and suitable for indoors interaction scenarios. The audio-visual input from the Kinect sensors is fed into speech, gesture, and action recognition modules, appropriately developed in this paper to address the challenging nature of child-robot interaction. For this purpose, data from multiple children are collected and used for module training or adaptation. Further, information from the multiple sensors is fused to enhance module performance. The perception system is integrated in a modular multi-robot architecture demonstrating its flexibility and scalability with different robotic platforms. The whole system, called Multi3, is evaluated, both objectively at the module level and subjectively in its entirety, under appropriate child-robot interaction scenarios containing several carefully designed games between children and robots.

In large scale coverage operations, such as marine exploration or aerial monitoring, single robot approaches are not ideal, as they may take too long to cover a large area. In such scenarios, multi-robot approaches are preferable. Furthermore, several real world vehicles are non-holonomic, but can be modeled using Dubins vehicle kinematics. This paper focuses on environmental monitoring of aquatic environments using Autonomous Surface Vehicles (ASVs). In particular, we propose a novel approach for solving the problem of complete coverage of a known environment by a multi-robot team consisting of Dubins vehicles. It is worth noting that both multi-robot coverage and Dubins vehicle coverage are NP-complete problems. As such, we present two heuristics methods based on a variant of the traveling salesman problem-k-TSP-formulation and clustering algorithms that efficiently solve the problem. The proposed methods are tested both in simulations to assess their scalability and with a team of ASVs operating on a 200 km2 lake to ensure their applicability in real world.

Applications that provide location related services need to understand the environment in which humans live such that verbal references and human interaction are possible. We formulate this semantic labelling task as the problem of learning the semantic labels from the perceived 3D structure. In this contribution we propose a batch approach and a novel multi-view frame fusion technique to exploit multiple views for improving the semantic labelling results. The batch approach works offline and is the direct application of an existing single-view method to scene reconstructions with multiple views. The multi-view frame fusion works in an incremental fashion accumulating the single-view results, hence allowing the online multi-view semantic segmentation of single frames and the offline reconstruction of semantic maps. Our experiments show the superiority of the approaches based on our fusion scheme, which leads to a more accurate semantic labelling.

In this paper we consider a fleet of self-driving cars operating in a road network governed by rules of the road, such as the Vienna Convention on Road Traffic, providing rides to customers to serve their demands with desired deadlines. We focus on the associated motion planning problem that trades-off the demands' delays and level of violation of the rules of the road to achieve social optimum among the vehicles. Due to operating in the same environment, the interaction between the cars must be taken into account, and can induce further delays. We propose an integrated route and motion planning approach that achieves scalability with respect to the number of cars by resolving potential collision situations locally within so-called bubble spaces enclosing the conflict. The algorithms leverage the road geometries, and perform joint planning only for lead vehicles in the conflict and use queue scheduling for the remaining cars. Furthermore, a framework for storing previously resolved conflict situations is proposed, which can be use for quick querying of joint motion plans. We show the mobility-on-demand setup and effectiveness of the proposed approach in simulated case studies involving up to 10 self-driving vehicles.

Learning-based approaches to robotic manipulation are limited by the scalability of data collection and accessibility of labels. In this paper, we present a multi-task domain adaptation framework for instance grasping in cluttered scenes by utilizing simulated robot experiments. Our neural network takes monocular RGB images and the instance segmentation mask of a specified target object as inputs, and predicts the probability of successfully grasping the specified object for each candidate motor command. The proposed transfer learning framework trains a model for instance grasping in simulation and uses a domain-adversarial loss to transfer the trained model to real robots using indiscriminate grasping data, which is available both in simulation and the real world. We evaluate our model in real-world robot experiments, comparing it with alternative model architectures as well as an indiscriminate grasping baseline.

The technique of robust suture detection is vital in many applications including trainee suturing skill evaluation, suture augmentation in robotic-assisted surgery and suture recognition for automatic suturing. Due to the complicated environment of surgery, the detection of a suture is challenged by high deformation and frequent occlusion. In this paper, we propose a deep multi-stage framework for suture detection. The fully convolutional neural networks are firstly used to predict a gradient map which not only serves as a segmentation mask, but also provides useful structure information for the following thread centerline reconstruction. An overlapping map is also predicted to improve the quality of the gradient map in self-intersection area. Based on the gradient map, multiple segments of the thread are extracted and linked to form the whole thread using a curvilinear structure detector. Experiments on two types of threads demonstrate that the proposed method is able to detect the thread with human level performance when the thread is no occlusion or under finite self-intersection.

This paper considers the problem of multi-robot realization. A realization is a set of robot positions where pairwise distances are either bounded from above or from below by a given adjacency threshold - depending on whether the respective robot pairs are to be adjacent or not. In the realization problem, unlike the related coordinated navigation or formation control problems, exact goal positions or relative distances need not be specified. Rather, only pairwise adjacency constraints are given and the robots' positions are required to satisfy these constraints. Applications of realization problem include multi-robots involved in team games (playing soccer, etc), patrolling and area coverage. We present a novel solution to this problem in which the robots simultaneously navigate to find a realization of a given adjacency matrix without colliding with each other along the way. In this solution, complete information about pairwise distances and free configuration space are encoded using an artificial potential function over the cross product space of the robots' simultaneous positions and proximity variables. The closed-loop dynamics governing the motion of each velocity-controlled robot take the form of the appropriate projection of the gradient of this function while pairwise distances are adjusted accordingly. Our extensive simulations demonstrate that the proposed approach has considerably higher realization percentage and shorter movement distances in comparison to a standard 2-stage approach.

This work addresses multi-robot coordination in social human-populated environments using a market-based framework for solving the Multi-Robot Task Allocation (MRTA) problem. Humans are considered in the proposed coordination mechanism by means of accounting for social costs in bid evaluations and requesting collaboration in socially blocking situations. Initially, the effect of a realistic environment with varying number of static/moving humans on the behavior and performance of our method is studied through an extensive suite of experiments in a high-fidelity simulator. Results show that the total traveled distance and time are increased when humans are present in the environments. Localization noise is also increased particularly in the case of static people. In the second series of experiments, a number of problematic cases resulting in longer modified paths, blocked passages, and long waits have been investigated. A comparative study targeting human-agnostic navigation and planning, human-aware navigation and human-agnostic planning, and human-aware navigation and planning has been conducted. Both simulated and real robot experiments confirm the effectiveness of accounting for humans at both team and individual levels. This leads to respecting social constraints as well as achieving a better performance based on MRTA metrics.

In this work we introduced a prioritized Cartesian impedance control under the framework of the Quadratic Programming (QP) optimization. In particular, we present a formulation which is simpler than full inverse dynamics, avoids any matrix pseudo-inversion, inverse kinematics computation and considers strict priorities among tasks. Our formulation is based on QP optimization permitting to take into account also explicit inequality constraints. We compare in simulation the tracking results obtained with a classical algebraic implementation against those derived from the proposed QP implementation taking into account joint torque limits. We consider the classical Cartesian impedance controller and a simplified version, also known as Virtual Model Control. Finally the proposed method was implemented and validated on a humanoid upper-body torque controlled robot. Experimental trials involving various physical interaction conditions were executed to demonstrate the performance of the proposed method.

Many robotic applications, such as search-and-rescue, require multiple agents to search for and perform actions on targets. However, such missions present several challenges, including cooperative exploration, task selection and allocation, time limitations, and computational complexity. To address this, we propose a decentralized multi-agent decision-making framework for the search and action problem with time constraints. The main idea is to treat time as an allocated budget in a setting where each agent action incurs a time cost and yields a certain reward. Our approach leverages probabilistic reasoning to make near-optimal decisions leading to maximized reward. We evaluate our method in the search, pick, and place scenario of the Mohamed Bin Zayed International Robotics Challenge (MBZIRC), by using a probability density map and reward prediction function to assess actions. Extensive simulations show that our algorithm outperforms benchmark strategies, and we demonstrate system integration in a Gazebo-based environment, validating the framework's readiness for field application.

Morphological computation is a concept relevant to robots made of soft and elastic materials. It states that robot's rich dynamics can be exploited to generate desirable behaviors, which can be altered when their morphology is adapted accordingly. This paper presents a low-cost robot made of elastic curved beam driven by a motor, with morphological computation and adaptation ability. Simply by changing robot's shape and the rotating frequency of the motor that vibrates the robot's body, the robot is able to shift its behavior from showing a tendency to slide when it needs to perform tasks like going under confined space, to have more tendency to hop diagonally forward when the robot stands upright. It will also be shown that based on the proposed mechanism, the energy efficiency of the robot locomotion can be maximized.

Scale recovery is one of the essential problems for monocular visual odometry. The camera height is usually used as an absolute reference to recover the scale. In this case, the precision of scale recovery depends on the accuracy of the road region detection and road geometrical model calculation. In previous works, road detection and road geometrical model calculation are solved sequentially: the road geometrical model calculation is based on the road detection and the road region detection is based on the color information. However, the color information of a road is not stable enough. In the proposed method, the estimated road geometrical model is taken into consideration to detect the road region as a feedback. Therefore, the road region detection and road geometrical model estimation can benefit each other. Delaunay Triangulation method is used to segment an input image to many triangles with the matched feature points as vertices. Every triangle region is classified as a road region or not by comparing their geometrical model with that of the road and the road geometrical model is updated online. We evaluate our visual odometry scale recovery method on the KITTI dataset and the results show that our method is achieving the best performance among all existing monocular visual odometry scale recovery methods without additional sensors.

In this paper we present mono-stixels, a compact environment representation specially designed for dynamic street scenes. Mono-stixels are a novel approach to estimate stixels from a monocular camera sequence instead of the traditionally used stereo depth measurements. Our approach jointly infers the depth, motion and semantic information of the dynamic scene as a 1D energy minimization problem based on optical flow estimates, pixel-wise semantic segmentation and camera motion. The optical flow of a stixel is described by a homography. By applying the mono-stixel model the degrees of freedom of a stixel-homography are reduced to only up to two degrees of freedom. Furthermore, we exploit a scene model and semantic information to handle moving objects. In our experiments we use the public available DeepFlow for optical flow estimation and FCN8s for the semantic information as inputs and show on the KITTI 2015 dataset that mono-stixels provide a compact and reliable depth reconstruction of both the static and moving parts of the scene. Thereby, mono-stixels overcome the limitation to static scenes of previous structure-from-motion approaches.

Soft robots have recently evoked extensive attention due to their abilities to work effectively in unstructured environments. As an actuation technology of soft robots, dielectric elastomers exhibit many intriguing attributes such as large strain and high energy density. This work presents a novel dielectric elastomer based soft crawling robot inspired by inchworms. To fill the need of control of the soft robot, a model describing the interaction between the dielectric elastomer actuator and the environment is proposed, which takes inertia, viscoelasticity and friction into consideration. The model can well describe the robot's dynamic performances and the modelling approach used here can be extended to other dielectric elastomer actuators with complicated geometries for control purposes. The obtained model allows us to design a feedforward plus feedback control scheme for the robot to achieve desired motion. Simulation shows fast response and good tracking performances which are further confirmed by the experiments.

This paper proposes an analytical framework for modelling resource contention in multi-robot systems, where the travel times and task durations are uncertain. It uses several approximation methods to quickly and accurately calculate the probability distributions describing the times at which the tasks start and finish. Specific contributions include exact and fast approximation methods for calculating the probability of a set of independent normally distributed random events occurring in a given order, a method for calculating the most likely and n-th most likely orders of occurrence for a set of independent normally distributed random events that have equal standard deviations, and a method for approximating the conditional probability distributions of the events given a specific order of the events. The complete framework is shown to be faster than a Monte Carlo approach for the same accuracy in two multi-robot task allocation problems. In addition, the importance of incorporating uncertainty is demonstrated through a comparison with a deterministic method. This is a general framework that is agnostic to the optimisation method and objective function used, and is applicable to a wide range of problems.