ICRA 2019 Papers

Interactive driving is challenging but essential for autonomous cars in dense traffic or urban areas. Proper interaction requires understanding and prediction of future trajectories of all neighboring cars around a target vehicle. Current solutions typically assume a certain distribution or stochastic process to approximate human-driven cars' behaviors. To relax this assumption, a Recurrent Meta Induction Network (RMIN) framework is developed. The original Conditional Neural Process (CNP) on which this is based does not consider the sequence of the conditions, due to the permutation invariance requirements for stochastic processes. However, the sequential information is important for the driving behavior estimation. Therefore, in the proposed method, a recurrent neural cell replaces the original demonstration sub-net. The behavior estimation is conditioned on the historical observations for all related cars, including the target car and its surrounding cars. The method is applied to predict the lane change trajectory of a target car in dense traffic areas. The proposed method achieves better results than previous methods and thanks to the meta-learning framework, it can use a smaller dataset, putting fewer demands on autonomous driving data collection.

Service robots are expected to autonomously and efficiently work in human-centric environments. For this type of robots, object perception and manipulation are challenging tasks due to need for accurate and real-time response. This paper presents an interactive open-ended learning approach to recognize multiple objects and their grasp affordances concurrently. This is an important contribution in the field of service robots since no matter how extensive the training data used for batch learning, a robot might always be confronted with an unknown object when operating in human-centric environments. The paper describes the system architecture and the learning and recognition capabilities. Grasp learning associates grasp configurations (i.e., end-effector positions and orientations) to grasp affordance categories. The grasp affordance category and the grasp configuration are taught through verbal and kinesthetic teaching, respectively. A Bayesian approach is adopted for learning and recognition of object categories and an instance-based approach is used for learning and recognition of affordance categories. An extensive set of experiments has been performed to assess the performance of the proposed approach regarding recognition accuracy, scalability and grasp success rate on challenging datasets and real-world scenarios.

In order to enable high-quality decision making and motion planning of intelligent systems such as robotics and autonomous vehicles, accurate probabilistic predictions for surrounding interactive objects is a crucial prerequisite. Although many research studies have been devoted to making predictions on a single entity, it remains an open challenge to forecast future behaviors for multiple interactive agents simultaneously. In this work, we take advantage of the Generative Adversarial Network (GAN) due to its capability of distribution learning and propose a generic multi-agent probabilistic prediction and tracking framework which takes the interactions among multiple entities into account, in which all the entities are treated as a whole. However, since GAN is very hard to train, we make an empirical research and present the relationship between training performance and hyperparameter values with a numerical case study. The results imply that the proposed model can capture both the mean, variance and multi-modalities of the groundtruth distribution. Moreover, we apply the proposed approach to a real-world task of vehicle behavior prediction to demonstrate its effectiveness and accuracy. The results illustrate that the proposed model trained by adversarial learning can achieve a better prediction performance than other state-of-the-art models trained by traditional supervised learning which maximizes the data likelihood. The well-trained model can also be utilized as an implicit proposal distribution for particle filtered based Bayesian state estimation.

In a multi-agent setting, the optimal policy of a single agent is largely dependent on the behavior of other agents. We investigate the problem of multi-agent reinforcement learning, focusing on decentralized learning in non-stationary domains for mobile robot navigation. We identify a cause for the difficulty in training non-stationary policies: mutual adaptation to sub-optimal behaviors, and we use this to motivate a curriculum-based strategy for learning interactive policies. The curriculum has two stages. First, the agent leverages policy gradient algorithms to learn a policy that is capable of achieving multiple goals. Second, the agent learns a modifier policy to learn how to interact with other agents in a multi-agent setting. We evaluated our approach on both an autonomous driving lane-change domain and a robot navigation domain.

Promoting a robot agent's autonomy level, which allows it to understand the human operator's intent and provide motion assistance to achieve it, has demonstrated great advantages to the operator's intent in teleoperation. However, the research has been limited to the target approaching process. We advance the shared control technique one step further to deal with the more challenging object manipulation task. Appropriately manipulating an object is challenging as it requires fine motion constraints for a certain manipulation task. Although these motion constraints are critical for task success, they are subtle to observe from ambiguous human motion. The disembodiment problem and physical discrepancy between the human and robot hands bring additional uncertainty, make the object manipulation task more challenging. Moreover, there is a lack of modeling and planning techniques that can effectively combine the human motion input and robot agent's motion input while accounting for the ambiguity of the human intent. To overcome this challenge, we built a multi-task robot grasping model and developed an intent-uncertainty-aware grasp planner to generate robust grasp poses given the ambiguous human intent inference inputs. With this validated modeling and planning techniques, it is expected to extend teleoperated robots' functionality and adoption in practical telemanipulation scenarios.

This paper presents a Model Predictive Controller (MPC) that uses navigation integrity risk as a constraint. Navigation integrity risk accounts for the presence of faults in localization sensors and algorithms, an increasingly important consideration as the number of robots operating in life and mission-critical situations is expected to increase dramatically in near future (e.g. a potential influx of self-driving cars). Specifically, the work uses a local nearest neighbor integrity risk evaluation methodology that accounts for data association faults as a constraint in order to guarantee localization safety over a receding horizon. Moreover, state and control-input constraints have also been enforced in this work. The proposed MPC design is tested using real-world mapped environments, showing that a robot is capable of maintaining a predefined minimum level of localization safety while operating in an urban environment.

Though vision-based techniques have become quite popular for autonomous docking of Unmanned Aerial Vehicles (UAVs), due to limited field of view (FOV), the UAV must rely on other methods to detect and approach the target before vision can be used. In this paper we propose a method combining Ultra-wideband (UWB) ranging sensor with vision-based techniques to achieve both autonomous approaching and landing capabilities in GPS-denied environments. In the approaching phase, a robust and efficient recursive least-square optimization algorithm is proposed to estimate the position of the UAV relative to the target by using the distance and relative displacement measurements. Using this estimate, UAV is able to approach the target until the landing pad is detected by an onboard vision system, then UWB measurements and vision-derived poses are fused with onboard sensor of UAV to facilitate an accurate landing maneuver. Real-world experiments are conducted to demonstrate the efficiency of our method.

This paper develops a decentralized approach for mapping and information-driven path planning for Very Large Scale Robotic (VLSR) systems. In this approach, obstacle mapping is performed using a continuous probabilistic representation known as a Hilbert map, which formulates the mapping problem as a binary classification task and uses kernel logistic regression to train a discriminative classifier online. A novel Hilbert map fusion method is presented that quickly and efficiently combines the information from individual robot maps. An integrated mapping and path planning algorithm is presented to determine paths of maximum information value, while simultaneously performing obstacle avoidance. Furthermore, the effect of how percentage communication failure effects the overall performance of the system is investigated. The approach is demonstrated on a VLSR system with hundreds of robots that must map obstacles collaboratively over a large region of interest using onboard range sensors and no prior obstacle information. The results show that, through fusion and decentralized processing, the entropy of the map decreases over time and robot paths remain collision-free.

There are many potential applications that require flying robots to navigate through tunnel-like environments, such as inspections of small railway culverts and mineral mappings of mining tunnels. Nevertheless, those environments present many challenges for quadrotors to navigate through. The aerodynamic disturbances created from the fluid interaction between the propellers' downwash and the surrounding surfaces of the environment, as well as longitudinal wind gusts, add hardship in stabilising the vehicle while the restricted narrow space increases the risk of collision. Furthermore, poor visibility and dust blown by the downwash make vision-based localisation extremely difficult. This paper presents a cross-sectional localisation system using Hough Scan Matching and a simple kinematic Kalman filter. Using the estimated state information, an integral backstepping controller is implemented which enables quadrotors to robustly fly in tunnel-like confined environments. A semi-autonomous system is proposed with self-stabilisation in the vertical and lateral axes while a pilot provides commands in the longitudinal direction. The results of a series of experiments in a simulated tunnel show that the proposed system successfully hovered itself and tracked various trajectories in a cross-sectional area without the aid of any external sensing or computing system.

Soft-bodied robots are getting attention from researchers as their potential in designing compliant and adaptive robots. However, soft-bodied robots also pose many challenges not only in non-linear controlling but also in design and fabrication. Especially, the non-compatibility between soft materials and rigid sensors/actuators makes it more difficult to design a fully compliant soft-bodied robot. In this paper, we propose an all-printed sensor and actuator for designing soft-bodied robots by printing silver nano-particle ink on top of a flexible plastic film. We can print bending sensors and thermal based actuators instantly with home-commodity inkjet printers without any pre/post-processing. We exemplify the application of this fabrication method with an all-printed paper caterpillar robots which can inch forward and sense its body bending angle.

The problem of minimizing cost in nonlinear control systems with uncertainties or disturbances remains a major challenge. Model predictive control (MPC), and in particular sampling-based MPC has recently shown great success in complex domains such as aggressive driving with highly nonlinear dynamics. Sampling-based methods rely on a prior distribution to generate samples in the first place. Obviously, the choice of this distribution highly influences efficiency of the controller. Existing approaches such as sampling around the control trajectory of the previous time step perform suboptimally, especially in multi-modal or highly dynamic settings. In this work, we therefore propose to learn models that generate samples in low-cost areas of the state-space, conditioned on the environment and on contextual information of the task to solve. By using generative models as an informed sampling distribution, our approach exploits guidance from the learned models and at the same time maintains robustness properties of the MPC methods. We use Conditional Variational Autoencoders (CVAE) to learn distributions that imitate samples from a training dataset containing optimized controls. An extensive evaluation in the autonomous navigation domain suggests that replacing previous sampling schemes with our learned models considerably improves performance in terms of path quality and planning efficiency.

Task sharing between heterogeneous robots currently requires a priori capability knowledge, a shared communication protocol, or a centralized planner. However, in practice, when two robots are brought together, the effort required to construct shared action and structure models can be significant. In this paper, we describe our approach to determining the kinematic model of a robot based purely on observation of unscripted movement. We describe construction of large-scale data simulating low-cost RGB-D camera output, and application of two different RNN-based methods to the learning problem. Our results suggest that this is an efficient and effective way to determine a robot's morphological structure without requiring communication or pre-existing knowledge of its capabilities.

The speed and accuracy with which robots are able to interpret natural language is fundamental to realizing effective human-robot interaction. A great deal of attention has been paid to developing models and approximate inference algorithms that improve the efficiency of language understanding. However, existing methods still attempt to reason over a representation of the environment that is flat and unnecessarily detailed, which limits scalability. An open problem is then to develop methods capable of producing the most compact environment model sufficient for accurate and efficient natural language understanding. We propose a model that leverages environment-related information encoded within instructions to identify the subset of observations and perceptual classifiers necessary to perceive a succinct, instruction-specific environment representation. The framework uses three probabilistic graphical models trained from a corpus of annotated instructions to infer salient scene semantics, perceptual classifiers, and grounded symbols. Experimental results on two robots operating in different environments demonstrate that by exploiting the content and the structure of the instructions, our method learns compact environment representations that significantly improve the efficiency of natural language symbol grounding.

Visual-Inertial Odometry (VIO) algorithms typically rely on a point cloud representation of the scene that does not model the topology of the environment. A 3D mesh instead offers a richer, yet lightweight, model. Nevertheless, building a 3D mesh out of the sparse and noisy 3D landmarks triangulated by a VIO algorithm often results in a mesh that does not fit the real scene. In order to regularize the mesh, previous approaches decouple state estimation from the 3D mesh regularization step, and either limit the 3D mesh to the current frame [1], [2] or let the mesh grow indefinitely [3], [4]. We propose instead to tightly couple mesh regularization and state estimation by detecting and enforcing structural regularities in a novel factor-graph formulation. We also propose to incrementally build the mesh by restricting its extent to the time-horizon of the VIO optimization; the resulting 3D mesh covers a larger portion of the scene than a per-frame approach while its memory usage and computational complexity remain bounded. We show that our approach successfully regularizes the mesh, while improving localization accuracy, when structural regularities are present, and remains operational in scenes without regularities.

Collaborative robots are expected to work in cooperation with humans to improve productivity and maintain the quality of products. In the previous study, we have proposed an incremental learning system for adaptively scheduling a motion of the collaborative robot based on a worker's behavior. Although this system could model the worker's motion pattern precisely and robustly without collecting the worker's data in advance, it required two different models for modeling the worker's spatial and temporal features respectively and was not well considered for generalization. In this paper, we extend the previous incremental learning system by integrating the spatial and temporal models using a mixture model. In addition, we install a new incremental learning algorithm which improves a generalization capability of the mixture model and avoids overfitting in the situation where the prior information is limited. Implementing the proposed algorithm, we evaluate the effectiveness of the proposed system by experiments for several workers and for several assembly processes.

Previous work on emotion recognition demonstrated a synergistic effect of combining several modalities such as auditory, visual, and transcribed text to estimate the affective state of a speaker. Among these, the linguistic modality is crucial for the evaluation of an expressed emotion. However, manually transcribed spoken text cannot be given as input to a system practically. We argue that using ground-truth transcriptions during training and evaluation phases leads to a significant discrepancy in performance compared to real-world conditions, as the spoken text has to be recognized on the fly and can contain speech recognition mistakes. In this paper, we propose a method of integrating an automatic speech recognition (ASR) output with a character-level recurrent neural network for sentiment recognition. In addition, we conduct several experiments investigating sentiment recognition for human-robot interaction in a noise-realistic scenario which is challenging for the ASR systems. We quantify the improvement compared to using only the acoustic modality in sentiment recognition. We demonstrate the effectiveness of this approach on the Multimodal Corpus of Sentiment Intensity (MOSI) by achieving 73,6% accuracy in a binary sentiment classification task, exceeding previously reported results that use only acoustic input. In addition, we set a new state-of-the-art performance on the MOSI dataset (80.4% accuracy, 2% absolute improvement).

This paper proposes a technique for 3D object scanning via in-hand manipulation, in which an object reoriented in front of a video camera with multiple grasps and regrasps. In-hand object tracking is a significant challenge under fast movement, rapid appearance changes, and occlusions. This paper proposes a novel video-segmentation-based object tracking algorithm that tracks arbitrary in-hand objects more effectively than existing techniques. It also describes a novel RGB-D in-hand object manipulation dataset consisting of several common household objects. Experiments show that the new method achieves 6% increase in accuracy compared to top performing video tracking algorithms and results in noticeably higher quality reconstructed models. Moreover, testing with a novice user on a set of 200 objects demonstrates relatively rapid construction of complete 3D object models.

Visual-inertial navigation methods have been shown to be an effective, low-cost way to operate autonomously without GPS or other global measurements, however most filtering approaches to VI suffer from observability and consistency problems. To increase robustness of the state-of-the-art methods, we propose a three-fold improvement. First, we propose the addition of a linear drag term in the velocity dynamics which improves estimation accuracy. Second, we propose the use of a partial-update formulation which limits the effect of linearization errors in partially-observable states, such as sensor biases. Finally, we propose the use of a keyframe reset step to enforce observability and consistency of the normally unobservable position and heading states. While all of these concepts have been used independently in the past, our experiments demonstrate additional strength when they are used simultaneously in a visual-inertial state estimation problem. In this paper, we derive the proposed filter and use a Monte Carlo simulation experiment to analyze the response of visual-inertial Kalman filters with the above described additions. The results of this study show that the combination of all of these features significantly improves estimation accuracy and consistency.

Kinematic redundancy enhances the dexterity and flexibility of robot manipulators. By exploiting the redundant degrees of freedom, auxiliary null space tasks can be carried out in addition to the primary task. Such auxiliary tasks are often formulated in terms of a performance or safety criterion that shall be minimized. If the optimization criterion, however, is defined in global terms, then it is directly affected by the primary task. As a consequence, the task achievement of the auxiliary task may be unnecessarily detrimented by the main task. In addition to modifying the primary task via constraint relaxation, a possible solution for improving the performance of the auxiliary task is to relax the primary task temporarily via time scaling. This gives the null space task more time for achieving its objective. In this paper, we propose several such time scaling schemes and verify their performance for a DLR/KUKA Lightweight Robot with one redundant degree of freedom. Finally, we extend the concept to multiple prioritized tasks and provide a simulation example.