IROS 2017 Papers

Aerial manipulation tasks necessitate a reliable trajectory planning algorithms to perform complicated tasks. This paper provides a method of developing the locally optimal trajectory for aerial manipulation in constrained environments. We first show differential flatness of the aerial manipulation system when the inertial effect due to equipped robotic arm is compensated with the aid of a robust controller. To find the locally optimal path, we parameterize flat outputs as polynomials of time, and formulate a sequential quadratic programming (SQP) problem. Given a convex mesh representation of environment, we obtain a collision-free trajectory in almost real time, by imposing constraints based on a signed distance metric. We also conduct an experiment of operating an object located in a confined space, which validates effectiveness of the proposed algorithm.

Rotary-Wing Air Vehicles (RW-UAVs), also referred to as drones, have gained in popularity over the last few years. Intrusions over secured areas have become common and authorities are actively looking for solutions to detect and localize undesired drones. The sound generated by the propellers of the RW-UAVs is powerful enough to be perceived by a human observer nearby. In this paper, we examine the use of particle filtering to detect and localize in 3D the position of a RW-UAV based on sound source localization (SSL) over distributed microphone arrays (MAs). Results show that the proposed method is able to detect and track a drone with precision, as long as the noise emitted by the RW-UAVs dominates the background noise.

Most Unmanned Aerial Vehicle (UAV) controllers require linear velocities as input. An effective method to obtain linear velocity is to place a downward facing camera and to estimate the velocity from the optical flow. However, this technique fails in outdoor environments when the ground is covered with grass or other objects which move due to winds such as those caused by the propellers. We present a novel method to estimate the linear velocities from stereo images even in the presence of disorderly motion of image features. We validate the approach using imagery obtained from a UAV flying through orchard rows.

We consider a class of robotic systems composed of high elongation linear actuators connected at universal joints. We derive the differential kinematics of such robots, and formalize concepts of controllability based on graph rigidity. Control methods are then developed for two separate applications: locomotion and shape morphing. The control algorithm in both cases solves a series of linearly constrained quadratic programs at each time step to minimize an objective function while ensuring physical feasibility. We present simulation results for locomotion along a prescribed path, and morphing to a target shape.

In this paper, we present an advanced real-time Visual Inertial Navigation System (VINS) based on Multi-State Constraint Kalman Filter (MSCKF). This filter uses Cubature Kalman Filter (CKF) for nonlinear measurement update and Maximum Likelihood Estimate (MLE) to optimize the estimate, which in turn provides better system accuracy and stability. The measurement model is developed basing Trifocal Tensor Geometry (TTG), which allows replacing the 3D feature-point reconstruction step as in traditional VINS systems. Alternatively the available Unscented MSCKF [1] based on Unscented Kalman Filter has an implementation issue of executing the square-root operation of the covariance matrix due to the negatively-weighted sigma points, and this may halt the filter operation or even causes the system to fail. The proposed CKF structure has the ability to carry the highly-nonlinear TTG-based measurement model as well as overcome the issue associated with the covariance square-root operation. The MLE based iteration is applied to optimize the visual measurement update where it performs multiple corrections on a single measurement. This procedure helps to minimize the error accumulation allowing the filter to operate for longer durations. The proposed Iterated Cubature MSCKF is tested using KITTI datasets [2] and compared against the Unscented MSCKF and non-iterated Cubature MSCKF.

Probabilistic graphical models, such as partially observable Markov decision processes (POMDPs), have been used in stochastic spoken dialog systems to handle the inherent uncertainty in speech recognition and language understanding. Such dialog systems suffer from the fact that only a relatively small number of domain variables are allowed in the model, so as to ensure the generation of good-quality dialog policies. At the same time, the non-language perception modalities on robots, such as vision-based facial expression recognition and Lidar-based distance detection, can hardly be integrated into this process. In this paper, we use a probabilistic commonsense reasoner to “guide” our POMDP-based dialog manager, and present a principled, multimodal dialog management (MDM) framework that allows the robot's dialog belief state to be seamlessly updated by both observations of human spoken language, and exogenous events such as the change of human facial expressions. The MDM approach has been implemented and evaluated both in simulation and on a real mobile robot using guidance tasks.

The ability to hand objects to users is a key skill for service robots. While the main purpose of a handover action is to successfully transfer an object to the user, it is also relevant that the robot takes the user's preferences into consideration when deciding which handover strategy to use. In a recent online study, we found evidence that suggests two important facts: (1) Users find value in a robot capable of performing handover tasks in more than one manner, and (2) Different users display different preferences on how they would like to be handed a requested object. Therefore, in this work we are proposing a novel system for learning handover preferences. Our system is evaluated in both simulation and on a real robot using computational perception to estimate human activity and environment type.

Pouring is a simple task people perform daily. It is the second most frequently executed motion in cooking scenarios, after pick-and-place. We present a pouring trajectory generation approach, which uses force feedback from the cup to determine the future velocity of pouring. The approach uses recurrent neural networks as its building blocks. We collected the pouring demonstrations which we used for training. To test our approach in simulation, we also created and trained a force estimation system. The simulated experiments show that the system is able to generalize to single unseen element of the pouring characteristics.

We present a controller that allows an armlike manipulator to navigate deformable cloth garments in simulation through the use of haptic information. The main challenge of such a controller is to avoid getting tangled in, tearing or punching through the deforming cloth. Our controller aggregates force information from a number of haptic-sensing spheres all along the manipulator for guidance. Based on haptic forces, each individual sphere updates its target location, and the conflicts that arise between this set of desired positions is resolved by solving an inverse kinematic problem with constraints. Reinforcement learning is used to train the controller for a single haptic-sensing sphere, where a training run is terminated (and thus penalized) when large forces are detected due to contact between the sphere and a simplified model of the cloth. In simulation, we demonstrate successful navigation of a robotic arm through a variety of garments, including an isolated sleeve, a jacket, a shirt, and shorts. Our controller out-performs two baseline controllers: one without haptics and another that was trained based on large forces between the sphere and cloth, but without early termination.

How do you learn to navigate an Unmanned Aerial Vehicle (UAV) and avoid obstacles? One approach is to use a small dataset collected by human experts: however, high capacity learning algorithms tend to overfit when trained with little data. An alternative is to use simulation. But the gap between simulation and real world remains large especially for perception problems. The reason most research avoids using large-scale real data is the fear of crashes! In this paper, we propose to bite the bullet and collect a dataset of crashes itself! We build a drone whose sole purpose is to crash into objects: it samples naive trajectories and crashes into random objects. We crash our drone 11,500 times to create one of the biggest UAV crash dataset. This dataset captures the different ways in which a UAV can crash. We use all this negative flying data in conjunction with positive data sampled from the same trajectories to learn a simple yet powerful policy for UAV navigation. We show that this simple self-supervised model is quite effective in navigating the UAV even in extremely cluttered environments with dynamic obstacles including humans. For supplementary video see:.

A new class of simple, adaptive, under-actuated and compliant robot hands has recently attracted the interest of the robotics community. The under-actuated mechanisms and the structural compliance used in these hands facilitate and robustify not only grasping but also the execution of dexterous, in-hand manipulation tasks. Another significant characteristic of the particular hands is that they are able to efficiently grasp a wide range of everyday life objects even under significant object pose uncertainties. However, these hands, are difficult to model due to kinematic constraints introduced by the underactuation and the use of complex flexure joints. Moreover, adaptive hands tend to reconfigure upon contact with the object surface, imposing certain parasitic object motions. In this paper, we propose a learning scheme that uses the contact force measurements collected from tactile sensors to estimate the post-contact reconfiguration of the hand-object system and the imposed parasitic object motion. The learning scheme's estimates are compared with “ground truth” data that describe the actual motion of the object and that are collected using a vision based motion capture system. The proposed learning scheme can be used with any type of adaptive robot hand and its efficiency is experimentally validated using extensive paradigms involving different hand designs and various everyday life objects.

In the context of robotic control, synergies can form elementary units of behavior. By specifying task-dependent coordination behaviors at a low control level, one can achieve task-specific disturbance rejection. In this work we present an approach to learn the parameters of such low-level controllers by demonstration. We identify a synergy by extracting covariance information from demonstration data. The extracted synergy is used to derive a time-invariant state feedback controller through optimal control. To cope with the non-Euclidean nature of robot poses, we utilize Riemannian geometry, where both estimation of the covariance and the associated controller take into account the geometry of the pose manifold. We demonstrate the efficacy of the approach experimentally in a bimanual manipulation task.

We design a new approach that allows robot learning of new activities from unlabeled human example videos. Given videos of humans executing the same activity from a human's viewpoint (i.e., first-person videos), our objective is to make the robot learn the temporal structure of the activity as its future regression network, and learn to transfer such model for its own motor execution. We present a new deep learning model: We extend the state-of-the-art convolutional object detection network for the representation/estimation of human hands in training videos, and newly introduce the concept of using a fully convolutional network to regress (i.e., predict) the intermediate scene representation corresponding to the future frame (e.g., 1-2 seconds later). Combining these allows direct prediction of future locations of human hands and objects, which enables the robot to infer the motor control plan using our manipulation network. We experimentally confirm that our approach makes learning of robot activities from unlabeled human interaction videos possible, and demonstrate that our robot is able to execute the learned collaborative activities in real-time directly based on its camera input.

Motion planning with predictable timing and velocity will enable a number of interesting applications such as autonomous intersection management (AIM). These planning algorithms depend on an accurate model of the performance of the vehicular controllers, which can be highly non-linear. Au et al. proposed a motion planning algorithm to satisfy the arrival requirements in AIM. However, they assumed that the performance models are given for every road and did not discuss how to learn these models. In this paper, we propose an instance-based learning approach to learn the performance models automatically, and argue that instance-based learning is suitable for this learning task because performance models for different roads can have a high correlation with each other. Moreover, an exploration strategy based on the principle of least effort is given to speed up the learning process. Our experiments showed that the instance-based learning method with distance-based exploration strategy offers a faster learning rate than the artificial neural network methods.

Over the past years learning from demonstration has become a popular method to intuitively teach new skills to service robots without explicit programming. However, most teaching approaches in literature use kinesthetic training and do not include mobile platforms. Here, we present a novel approach to learn joint robot base and gripper action models from observing demonstrations carried out by a human teacher. To achieve this we adapt RGBD observations of the human teacher to the capabilities of the robot. We formulate a graph optimization problem that links observations with robot grasping capabilities and kinematic constraints between co-occurring base and gripper poses. In real world experiments we show that the robot is able to learn complex mobile manipulation tasks such as opening and driving through a door.

Posture body variation is one of the ways in which humans skillfully and naturally augment their motion and strength capabilities along specific task-space directions in order to successfully perform complex manipulation tasks. Posture variation also has a significant role in robot manipulation, where manipulability arises as a useful criterion to analyze and control the robot dexterity as a function of its joint configuration. In this context, this paper introduces the promising idea of manipulability transfer, a method that allows robots to learn and reproduce desired manipulability ellipsoids from expert demonstrations. The proposed framework is built on a tensor-based formulation of Gaussian mixture model that takes into account that manipulability ellipsoids lie on the manifold of symmetric positive definite matrices. This geometry-aware method is used to design a manipulability-based redundancy resolution that allows the robot to modify its posture so that its manipulability ellipsoid coincides with the desired one. Experiments in simulation validate the functionality of the proposed approach, which extends the robot learning capability beyond trajectory, force and impedance learning approaches.

The work presented in this paper describes the use and evaluation of machine learning techniques like neural networks and support vector regression to learn a model of magnetic field distortions often induced in inertial measurement units using magnetometers by changing currents, postures or configurations of a robotic system. Such a model is needed in order to compensate the local dynamic distortions, especially for complex and confined robotic systems, and to achieve more robust and accurate ambient magnetic field measurements. This is crucial for a wide variety of autonomous navigation purposes from simple heading estimation over standard SLAM approaches to sophisticated magnetic field based localization techniques. The approach was evaluated in a laboratory setup and with a complex robotic system in an outdoor environment.

Robots performing everyday tasks such as cooking in a kitchen need to be able to deal with variations in the household tools that may be available. Given a particular task and a set of tools available, the robot needs to be able to assess which would be the best tool for the task, and also where to grasp that tool and how to orient it. This requires an understanding of what is important in a tool for a given task, and how the grasping and orientation relate to performance in the task. A robot can learn this by trying out many examples. This learning can be faster if these trials are done in simulation using tool models acquired from the Web. We provide a semi-automatic pipeline to process 3D models from the Web, allowing us to train from many different tools and their uses in simulation. We represent a tool object and its grasp and orientation using 21 parameters which capture the shapes and sizes of principal parts and the relationships among them. We then learn a `task function' that maps this 21 parameter vector to a value describing how effective it is for a particular task. Our trained system can then process the unsegmented point cloud of a new tool and output a score and a way of using the tool for a particular task. We compare our approach with the closest one in the literature and show that we achieve significantly better results.

In many situations, users walk on typical paths between specific destinations at which the service of a mobile robot is needed. Depending on the environment and the paths, step-by-step following of the human might not be the optimal solution since better paths for the robot exist. We propose to perform a prediction about the human's future movements and use this information in a reinforcement learning framework to generate foresighted navigation actions for the robot. Since frequent occlusions of the human will occur due to obstacles and the robot's constrained field of view, the estimate about the humans's position and the prediction of the next destination are affected by uncertainty. Our approach deals with such situations by explicitly considering occlusions in the reward function such that the robot automatically considers to execute actions to get the human in its field of view. We show in simulated and real-world experiments that our technique leads to significantly shorter paths compared to an approach in which the robot always tries to closely follow the user and, additionally, can handle occlusions.