ICRA 2019 Papers

Nowadays robotic systems are expected to share workspaces and collaborate with humans. In such collaborative environments, an important challenge is to ground or establish the correct semantic interpretation of a human request. Once such an interpretation is available, the request must be translated into robot motion commands in order to complete the desired task. It is not unusual that a human request cannot be grounded to a unique interpretation, thus leading to an ambiguous request. A simple example is to ask a robot to “put a cup on the table,” when there are multiple cups available. In order to deal with this kind of ambiguous request, we propose a delayed or lazy variable grounding. The focus of this paper is a motion planning algorithm that, given goal regions that represent different valid groundings, lazily finds a feasible path to any one valid grounding. This algorithm includes a reward-penalty strategy, which attempts to prioritize those goal regions that seem more promising to provide a solution. We validate our approach by solving requests with multiple valid alternatives in both simulation and real-world experiments.

This paper addresses the problem of object discovery from unlabeled driving videos captured in a realistic automotive setting. Identifying recurring object categories in such raw video streams is a very challenging problem. Not only do object candidates first have to be localized in the input images, but many interesting object categories occur relatively infrequently. Object discovery will therefore have to deal with the difficulties of operating in the long tail of the object distribution. We demonstrate the feasibility of performing fully automatic object discovery in such a setting by mining object tracks using a generic object tracker. In order to facilitate further research in objet discovery, we release a collection of more than 360,000 automatically mined object tracks from 10 + hours of video data (560,000 frames). We use this dataset to evaluate the suitability of different feature representations and clustering strategies for object discovery.

This paper describes a new robotic tabletop rearrangement system, and presents experimental results. The tasks involve rearranging as many as 30 to 100 blocks, sometimes packed with a density of up to 40%. The high packing factor forces the system to push several objects at a time, making accurate simulation difficult, if not impossible. Nonetheless, the system achieves goals specifying the pose of every object, with an average precision of ± 1 mm and ± 2°. The system searches through policy rollouts of simulated pushing actions, using an Iterated Local Search technique to escape local minima. In real world execution, the system executes just one action from a policy, then uses a vision system to update the estimated task state, and replans. The system accepts a fully general description of task goals, which means it can solve the singulation and separation problems addressed in prior work, but can also solve sorting problems and spell out words, among other things. The paper includes examples of several solved problems, statistical analysis of the system's behavior on different types of problems, and some discussion of limitations, insights, and future work.

Assessment of minimally invasive surgical skills is a non-trivial task, usually requiring the presence and time of expert observers, including subjectivity and requiring special and expensive equipment and software. This study develops an algorithm for tracking laparoscopy instruments in the video cues of a standard laparoscopy training box with a single webcam camera and proposes new criteria to assess skill level using the extracted tool trajectories. Instrument tracking and assessment criteria together constitute a significant step towards developing a low cost, automated, and widely applicable laparoscopy training and assessment system using a standard physical training box equipped with a webcam. The developed visual tracking algorithm recovers the 3D positions of the laparoscopic instruments tips to which simple colored tapes (markers) are attached. The new assessment criteria are based on frequency analysis and linear discriminant analysis of the 3D reconstructed trajectories of the instruments. The performance of these proposed criteria are compared to the conventional criteria for laparoscopy training and demonstrated to be superior on the data we have recorded from six professional laparoscopy surgeons and ten novice subjects.

Guided policy search is a popular approach for training controllers for high-dimensional systems, but it has a number of pitfalls. Non-convex trajectory optimization has local minima, and non-uniqueness in the optimal policy itself can mean that independently-optimized samples do not describe a coherent policy from which to train. We introduce LVIS, which circumvents the issue of local minima through global mixed-integer optimization and the issue of non-uniqueness through learning the optimal value function rather than the optimal policy. To avoid the expense of solving the mixed-integer programs to full global optimality, we instead solve them only partially, extracting intervals containing the true cost-to-go from early termination of the branch-and-bound algorithm. These interval samples are used to weakly supervise the training of a neural net which approximates the true cost-to-go. Online, we use that learned cost-to-go as the terminal cost of a one-step model-predictive controller, which we solve via a small mixed-integer optimization. We demonstrate LVIS on piecewise affine models of a cart-pole system with walls and a planar humanoid robot and show that it can be applied to a fundamentally hard problem in feedback control-control through contact.

In this work, we present a novel framework for on-line human gait stability prediction of the elderly users of an intelligent robotic rollator using Long Short Term Memory (LSTM) networks, fusing multimodal RGB-D and Laser Range Finder (LRF) data from non-wearable sensors. A Deep Learning (DL) based approach is used for the upper body pose estimation. The detected pose is used for estimating the body Center of Mass (CoM) using Unscented Kalman Filter (UKF). An Augmented Gait State Estimation framework exploits the LRF data to estimate the legs' positions and the respective gait phase. These estimates are the inputs of an encoder-decoder sequence to sequence model which predicts the gait stability state as Safe or Fall Risk walking. It is validated with data from real patients, by exploring different network architectures, hyperparameter settings and by comparing the proposed method with other baselines. The presented LSTM-based human gait stability predictor is shown to provide robust predictions of the human stability state, and thus has the potential to be integrated into a general user-adaptive control architecture as a fall-risk alarm.

While today's robots are able to perform sophisticated tasks, they can only act on objects they have been trained to recognize. This is a severe limitation: any robot will inevitably see new objects in unconstrained settings, and thus will always have visual knowledge gaps. However, standard visual modules are usually built on a limited set of classes and are based on the strong prior that an object must belong to one of those classes. Identifying whether an instance does not belong to the set of known categories (i.e. open set recognition), only partially tackles this problem, as a truly autonomous agent should be able not only to detect what it does not know, but also to extend dynamically its knowledge about the world. We contribute to this challenge with a deep learning architecture that can dynamically update its known classes in an end-to-end fashion. The proposed deep network, based on a deep extension of a non-parametric model, detects whether a perceived object belongs to the set of categories known by the system and learns it without the need to retrain the whole system from scratch. Annotated images about the new category can be provided by an `oracle' (i.e. human supervision), or by autonomous mining of the Web. Experiments on two different databases and on a robot platform demonstrate the promise of our approach.

A novel 3-[R(RR-RRR)SR] kinematically redundant 6+3-degree-of-freedom (dof)spatial hybrid parallel robot with revolute actuators is proposed. The kinematic model is developed based on the constraint conditions of the robot. It is shown that the type II (parallel) singularities can be completely avoided, thereby greatly extending the orientational workspace. Mechanisms are then introduced to use the redundant degrees of freedom of the robot to operate a gripper with the robot actuators, which are mounted on or close to the base. A CAD model of the robot is shown and a computer animation is provided to demonstrate the resulting architecture, which has full 6-dof capabilities and a large orientational workspace.

Optimal path planning and smooth trajectory planning are critical for effective navigation of mobile robots working towards accomplishing complex missions. For autonomous, real time and extended operations of mobile robots, the navigation capability needs to be executed at the edge. Thus, efficient compute, minimum memory utilization and smooth trajectory are the key parameters that drive the successful operation of autonomous mobile robots. Traditionally, navigation solutions focus on developing robust path planning algorithms which are complex and compute/memory intensive. Bidirectional-RRT(Bi-RRT) based path planning algorithms have gained increased attention due to their effectiveness and computational efficiency in generating feasible paths. However, these algorithms neither optimize memory nor guarantee smooth trajectories. To this end, we propose a kinematically constrained Bi-RRT (KB-RRT) algorithm, which restricts the number of nodes generated without compromising on the accuracy and incorporates kinodynamic constraints for generating smooth trajectories, together resulting in efficient navigation of autonomous mobile robots. The proposed algorithm is tested in a highly cluttered environment on an Ackermannsteering vehicle model with severe kinematic constraints. The experimental results demonstrate that KB-RRT achieves three times (3 X) better performance in terms of convergence rate and memory utilization compared to a standard Bi-RRT algorithm.

This paper introduces a novel four-degree-of-freedom (4-DOF) parallel mechanism having 3 translational DOFs and 1 rotational DOF. The mechanism comprises 2 sets of parallelogram linkages, which constrain two of the rotational DOFs of the mechanism. An interesting feature of the mechanism is that it can be driven using 4 parallel sliders mounted on its base. As a result, one of the translational DOFs can be infinitely large. Also, the architecture of the mechanism provides a large rotational DOF in one direction. The kinematic equations of the mechanism are derived and the Jacobian matrices are obtained. The mathematical conditions that lead to singularities are also found. Moreover, a geometric description of the boundaries of the workspace is given, which can be expressed using simple equations. Finally, some design examples are proposed and a prototype is presented.

This paper proposes an approach for fusing direct radiometric data from a thermal camera with inertial measurements to extend the robotic capabilities of aerial robots for navigation in GPS-denied and visually degraded environments in the conditions of darkness and in the presence of airborne obscurants such as dust, fog and smoke. An optimization based approach is developed that jointly minimizes the re-projection error of 3D landmarks and inertial measurement errors. The developed solution is extensively verified against both ground-truth in an indoor laboratory setting, as well as inside an underground mine under severely visually degraded conditions.

Dense visual SLAM methods are able to estimate the 3D structure of an environment and locate the observer within them. They estimate the motion of a camera by matching visual information between consecutive frames, and are thus prone to failure under extreme motion conditions or when observing texture-poor regions. The integration of additional sensor modalities has shown great promise in improving the robustness and accuracy of such SLAM systems. In contrast to the popular use of inertial measurements we propose to tightly-couple a dense RGB-D SLAM system with kinematic and odometry measurements from a wheeled robot equipped with a manipulator. The system has real-time capability while running on GPU. It optimizes the camera pose by considering the geometric alignment of the map as well as kinematic and odometric data from the robot. Through experimentation in the real-world, we show that the system is more robust to challenging trajectories featuring fast and loopy motion than the equivalent system without the additional kinematic and odometric knowledge, whilst retaining comparable performance to the equivalent RGB-D only system on easy trajectories.

Robotics applications often suffer from the `two-language problem', requiring a low-level language for performance-sensitive components and a high-level language for interactivity and experimentation, which tends to increase software complexity. We demonstrate the use of the Julia programming language to solve this problem by being fast enough for online control of a humanoid robot and flexible enough for prototyping. We present several Julia packages developed by the authors, which together enable roughly 2× realtime simulation of the Boston Dynamics Atlas humanoid robot balancing on flat ground using a quadratic-programming-based controller. Benchmarks show a sufficiently low variation in control frequency to make deployment on the physical robot feasible. We also show that Julia's naturally generic programming style results in versatile packages that are easy to compose and adapt to a wide variety of computational tasks in robotics.

The physical design of a robot and the policy that controls its motion are inherently coupled, and should be determined according to the task and environment. In an increasing number of applications, data-driven and learning-based approaches, such as deep reinforcement learning, have proven effective at designing control policies. For most tasks, the only way to evaluate a physical design with respect to such control policies is empirical-i.e., by picking a design and training a control policy for it. Since training these policies is time-consuming, it is computationally infeasible to train separate policies for all possible designs as a means to identify the best one. In this work, we address this limitation by introducing a method that jointly optimizes over the physical design and control network. Our approach maintains a distribution over designs and uses reinforcement learning to optimize a control policy to maximize expected reward over the design distribution. We give the controller access to design parameters to allow it to tailor its policy to each design in the distribution. Throughout training, we shift the distribution towards higher-performing designs, eventually converging to a design and control policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel designs and walking gaits, outperforming baselines across different settings.

Although the design of legged robots may be dependent on the application, all of them share the need to safely deal with physical interaction with the environment in every step they take. Impedance controllers have been applied with success to handle contact, and some authors applied the concept of passivity to guarantee stability during the interaction. Whereas previous studies on the passivity of legged robots considered aspects such as inner force loop gains and actuation bandwidth influence on the Z-width (i.e. the range of renderable passive impedances), they did not take into account the role of the kinematic configuration of the leg on the stability of the interaction. Thus, in this work we present a systematic analysis of the effects of joint positions on the passivity conditions of a robotic leg and show that this is a very relevant aspect that may seriously affect the stability and passivity of an impedance controller. By analyzing a linearized model of the leg via its Nyquist plots and the respective Z-width diagrams, we were able to determine what joint configurations within the leg workspace are more suitable to physically interact with the environment or people.

We present joint learning of instance and semantic segmentation for visible and occluded region masks. Sharing the feature extractor with instance occlusion segmentation, we introduce semantic occlusion segmentation into the instance segmentation model. This joint learning fuses the instance-and image-level reasoning of the mask prediction on the different segmentation tasks, which was missing in the previous work of learning instance segmentation only (instance-only). In the experiments, we evaluated the proposed joint learning comparing the instance-only learning on the test dataset. We also applied the joint learning model to 2 different types of robotic pick-and-place tasks (random and target picking) and evaluated its effectiveness to achieve real-world robotic tasks.

To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model tracker, and either a biomimetic tactile sensor or a force-torque sensor. We show how to fuse raw measurements from the tracker and tactile sensors to jointly estimate the trajectory of the kinematic states and the forces in the system via probabilistic inference on factor graphs, in both batch and incremental settings. We perform several benchmarks with our framework and show how performance is affected by incorporating various geometric and physics based constraints, occluding vision sensors, or injecting noise in tactile sensors. We also compare with prior work on multiple datasets and demonstrate that our approach can effectively optimize over multi-modal sensor data and reduce uncertainty to find better state estimates.

Older adults are vulnerable to symptoms of depression. The degree of depression is particularly high among older adults who live alone. To address this issue, various companion robots, which are capable of psychologically communicating with users, have been proposed. However, older adults' preferences on the appearance of these robots have not been systematically investigated; this forms the focus of the present study. We interviewed 191 older adults; investigated their preferences on the design elements of robots including type, weight, and material; and analyzed the data by age, gender, and living arrangement. Our primary goal was to determine how companion robots should be designed, paying special attention to older adults who live alone. Our findings indicated that those living alone prefer a bear-like robot and negative to the heavy robot. Our results suggest that companion robots need to be designed with careful consideration of older adults' physical and psychological preferences.

This paper presents a framework to learn the reward function underlying high-level sequential tasks from demonstrations. The purpose of reward learning, in the context of learning from demonstration (LfD), is to generate policies that mimic the demonstrator's policies, thereby enabling imitation learning. We focus on a human-robot interaction (HRI) domain where the goal is to learn and model structured interactions between a human and a robot. Such interactions can be modeled as a partially observable Markov decision process (POMDP) where the partial observability is caused by uncertainties associated with the ways humans respond to different stimuli. The key challenge in finding a good policy in such a POMDP is determining the reward function that was observed by the demonstrator. Existing inverse reinforcement learning (IRL) methods for POMDPs are computationally very expensive and the problem is not well understood. In comparison, IRL algorithms for Markov decision process (MDP) are well defined and computationally efficient. We propose an approach of reward function learning for high-level sequential tasks from human demonstrations where the core idea is to reduce the underlying POMDP to an MDP and apply any efficient MDPIRL algorithm. Our extensive experiments suggest that the reward function learned this way generates POMDP policies that mimic the policies of the demonstrator well.