ICRA 2019 Papers

We present a method for improving segmentation tasks on images affected by adherent rain drops and streaks. We introduce a novel stereo dataset recorded using a system that allows one lens to be affected by real water droplets while keeping the other lens clear. We train a denoising generator using this dataset and show that it is effective at removing the effect of real water droplets, in the context of image reconstruction and road marking segmentation. To further test our de-noising approach, we describe a method of adding computer-generated adherent water droplets and streaks to any images, and use this technique as a proxy to demonstrate the effectiveness of our model in the context of general semantic segmentation. We benchmark our results using the CamVid road marking segmentation dataset, Cityscapes semantic segmentation datasets and our own realrain dataset, and show significant improvement on all tasks.

In this paper, an active compliant revolute joint actuated by hydraulic energy is developed. The joint is made of polymer for integration in medical robotic systems, even in a challenging environment such as Magnetic Resonance Imaging (MRI). The use of multimaterial additive manufacturing allows us to develop two original aspects. First, a new seal design is proposed to build miniature hydraulic cylinders embedded in the active joint, with low level of friction. Second, a rack-and-pinion mechanism is being integrated to a compliant revolute joint to obtain a high level of compactness. Design and experimental assessment of the hydraulic cylinder and the compliant joint with embedded rack-and-pinion are presented, as well as an illustration in the context of needle manipulation with passive teleoperation.

Unauthorized drone flights near aircraft, airports, and emergency operations compromise the safety of passengers and bystanders. A detection system that can quickly find and track drones could help mitigate the risk of unauthorized drone flights. In this work, we show how a consumer drone outfitted with antennas and commodity radios can autonomously localize another drone by its telemetry radio emissions. We show how a non-myopic planner improves tracking performance over traditionally used greedy, one-step planners. Improved tracking is validated with simulations and the system is demonstrated with real drones in flight tests.

The locomotion and balance capabilities of bipedal robots have greatly improved in recent years. However, maintaining balance on difficult terrain still poses a significant challenge. In this paper, we examined how humans maintain mediolateral balance when standing on a narrow beam with bare feet and wearing rigid soles. Our results show that foot-beam interaction dynamics critically influence balancing behavior. Importantly, this suggests that differences in human balancing behavior across different support surfaces may not solely result from changes in their neural control strategy. They may also result from changes in foot-ground interaction. Thus, the altered foot-ground interaction dynamics must be considered to accurately capture changes in the human controller across different support surfaces. A simplified model of foot-beam interaction was added to a double inverted pendulum model for human balancing. This extended model could replicate the change in human behavior across different foot contact conditions (bare feet vs. rigid feet). A better understanding of how humans coordinate whole-body behavior across a range of conditions may inform the development of balance controllers for bipedal robots.

This paper describes a human measurement system for autonomous rounds robot aiming in physical support of care staffs working in nursing and medical facilities. We propose a novel scheme of vision-based contactless breathing measurement system that integrates three-dimensional shape and thermal information of the target person. We developed the human-care rounds robot Lucia and implemented the measurement system on the robot. We then evaluated the human detection and breathing measurement based on the conditions of real incident cases that occurred in the nursing facility for the physically handicapped. In the experiments, we examined that the breathing measurement system successfully measured volume variation of human subjects with different configurations of body postures. The robot is also capable of discrimination between the breathing and non-breathing states of targets based on the difference in the power spectrum patterns in the frequency domain. The experimental results showed that the proposed system detected the presence of breathing within the accuracy of about 90% or more, and moreover, the ability of anomaly detection in breathing was suggested.

While intelligence of autonomous vehicles (AVs) has significantly advanced in recent years, accidents involving AVs suggest that these autonomous systems lack gracefulness in driving when interacting with human drivers. In the setting of a two-player game, we propose model predictive control based on social gracefulness, which is measured by the discrepancy between the actions taken by the AV and those that could have been taken in favor of the human driver. We define social awareness as the ability of an agent to infer such favorable actions based on knowledge about the other agent's intent, and further show that empathy, i.e., the ability to understand others' intent by simultaneously inferring others' understanding of the agent's self intent, is critical to successful intent inference. Lastly, through an intersection case, we show that the proposed gracefulness objective allows an AV to learn more sophisticated behavior, such as passive-aggressive motions that gently force the other agent to yield.

This study proposes a dynamic observable contact (DOC) hand as a new multifingered hand to ensure high- speed insertion in an assembly process with a small clearance between objects. To achieve insertion with a small clearance at high speed, a robot hand must realize both impact reduction and position-error compensation when the two objects contact each other. The DOC hand, with its features of 6-degrees-of- freedom dynamic passivity and object-pose observability, can realize both impact reduction and position-error compensation. To evaluate the effectiveness of the DOC hand, we construct a robot system using the DOC hand. We evaluate the performance of the system in the task of ring insertion with a small clearance (0-36um). The results indicate that the robot system performs with a higher speed than a human. In fact, the average cycle time is 2.42 s for the robot, whereas it is 2.58 s for a human. The DOC hand has opened up the possibility for achieving high-speed precision assembly using robots.

Location-aware applications play an increasingly critical role in everyday life. However, satellite-based localization (e.g., GPS) has limited accuracy and can be unusable in dense urban areas and indoors. We introduce an image-based global localization system that is accurate to a few millimeters and performs reliable localization both indoors and outside. The key idea is to capture and index distinctive local keypoints in ground textures. This is based on the observation that ground textures including wood, carpet, tile, concrete, and asphalt may look random and homogeneous, but all contain cracks, scratches, or unique arrangements of fibers. These imperfections are persistent, and can serve as local features. Our system incorporates a downward-facing camera to capture the fine texture of the ground, together with an image processing pipeline that locates the captured texture patch in a compact database constructed offline. We demonstrate the capability of our system to robustly, accurately, and quickly locate test images on various types of outdoor and indoor ground surfaces. This paper contains a supplementary video. All datasets and code are available online at microgps.cs.princeton.edu.

This paper presents a design that jointly provides hand pose sensing, hand localization, and haptic feedback to facilitate real-time stable grasps in Virtual Reality (VR). The design is based on an easy-to-replicate glove-based system that can reliably perform (i) a high-fidelity hand pose sensing in real time through a network of 15 IMUs, and (ii) the hand localization using a Vive Tracker. The supported physics-based simulation in VR is capable of detecting collisions and contact points for virtual object manipulation, which drives the collision event to trigger the physical vibration motors on the glove to signal the user, providing a better realism inside virtual environments. A caging-based approach using collision geometry is integrated to determine whether a grasp is stable. In the experiment, we showcase successful grasps of virtual objects with large geometry variations. Comparing to the popular LeapMotion sensor, we demonstrate the proposed glove-based design yields a higher success rate in various tasks in VR. We hope such a glove-based system can simplify the data collection of human manipulations with VR.

Twisted string actuators are an emerging type of transmission systems that may benefit various applications of robotics and mechatronics. However, control of TSAs in applications that require high bandwidth has attracted comparatively little interest from research community, mainly due to complexity of twisted string behavior. This paper proposes a new adaptive control methodology that allows to sufficiently increase bandwidth of TSA-based systems. We reformulate mathematical model of the TSA into a suitable form for online parameter estimation, outline adaptive estimation methods and propose a method to design variable controller gain that rectifies nonlinearities in the system. we present experimental comparison of proposed adaptive control strategies with two conventional tsa control techniques. experimental results demonstrated that the proposed adaptive control architecture with feedforward speed term was nearly insensitive to increase in input signal frequency while reducing position tracking error by 80%. proposed algorithm can be applied in any tsa control system that has input and output signal measurements.

In this paper, we present a whole-body control framework for Wheeled Inverted Pendulum (WIP) Humanoids. WIP Humanoids are redundant manipulators dynamically balancing themselves on wheels. Characterized by several degrees of freedom, they have the ability to perform several tasks simultaneously, such as balancing, maintaining a body pose, controlling the gaze, lifting a load or maintaining end-effector configuration in operation space. The problem of whole-body control is to enable simultaneous performance of these tasks with optimal participation of all degrees of freedom at specified priorities for each objective. The control also has to obey constraint of angle and torque limits on each joint. The proposed approach is hierarchical with a low level controller for body joints manipulation and a high-level controller that defines center of mass (CoM) targets for the low-level controller to control zero dynamics of the system driving the wheels. The low-level controller plans for shorter horizons while considering more complete dynamics of the system, while the high-level controller plans for longer horizon based on an approximate model of the robot for computational efficiency.

The actions of an autonomous vehicle on the road affect and are affected by those of other drivers, whether overtaking, negotiating a merge, or avoiding an accident. This mutual dependence, best captured by dynamic game theory, creates a strong coupling between the vehicle's planning and its predictions of other drivers' behavior, and constitutes an open problem with direct implications on the safety and viability of autonomous driving technology. Unfortunately, dynamic games are too computationally demanding to meet the real-time constraints of autonomous driving in its continuous state and action space. In this paper, we introduce a novel game-theoretic trajectory planning algorithm for autonomous driving, that enables real-time performance by hierarchically decomposing the underlying dynamic game into a long-horizon “strategic” game with simplified dynamics and full information structure, and a short-horizon “tactical” game with full dynamics and a simplified information structure. The value of the strategic game is used to guide the tactical planning, implicitly extending the planning horizon, pushing the local trajectory optimization closer to global solutions, and, most importantly, quantitatively accounting for the autonomous vehicle and the human driver's ability and incentives to influence each other. In addition, our approach admits non-deterministic models of human decision-making, rather than relying on perfectly rational predictions. Our results showcase richer, safer, and more effective autonomous behavior in comparison to existing techniques.

This paper proposes a hierarchical depthwise graph convolutional neural network (HDGCN) for point cloud semantic segmentation. The main chanllenge for learning on point clouds is to capture local structures or relationships. Graph convolution has the strong ability to extract local shape information from neighbors. Inspired by depthwise convolution, we propose a depthwise graph convolution which requires less memory consumption compared with the previous graph convolution. While depthwise graph convolution aggregates features channel-wisely, pointwise convolution is used to learn features across different channels. A customized block called DGConv is specially designed for local feature extraction based on depthwise graph convolution and pointwise convolution. The DGConv block can extract features from points and transfer features to neighbors while being invariant to different point orders. HDGCN is constructed by a series of DGConv blocks using a hierarchical structure which can extract both local and global features of point clouds. Experiments show that HDGCN achieves the state-of-the-art performance in the indoor dataset S3DIS and the outdoor dataset Paris-Lille-3D.

Set-Based Multi-Task Priority is a recent framework to handle inverse kinematics for redundant structures. Both equality tasks, i.e., control objectives to be driven to a desired value, and set-bases tasks, i.e., control objectives to be satisfied with a set/range of values can be addressed in a rigorous manner within a priority framework. In addition, optimization tasks, driven by the gradient of a proper function, may be considered as well, usually as lower priority tasks. In this paper the proper design of the tasks, their priority and the use of a Set-Based Multi-Task Priority framework is proposed in order to handle several constraints simultaneously in real-time. It is shown that safety related tasks such as, e.g., joint limits or kinematic singularity, may be properly handled by consider them both at an higher priority as set-based task and at a lower within a proper optimization functional. Experimental results on a 7DOF Jaco2 arm with and without the proposed approach show the effectiveness of the proposed method.

Imitation learning has proven to be useful for many real-world problems, but approaches such as behavioral cloning suffer from data mismatch and compounding error issues. One attempt to address these limitations is the DAgger algorithm, which uses the state distribution induced by the novice to sample corrective actions from the expert. Such sampling schemes, however, require the expert to provide action labels without being fully in control of the system. This can decrease safety and, when using humans as experts, is likely to degrade the quality of the collected labels due to perceived actuator lag. In this work, we propose HG-DAgger, a variant of DAgger that is more suitable for interactive imitation learning from human experts in real-world systems. In addition to training a novice policy, HG-DAgger also learns a safety threshold for a model-uncertainty-based risk metric that can be used to predict the performance of the fully trained novice in different regions of the state space. We evaluate our method on both a simulated and real-world autonomous driving task, and demonstrate improved performance over both DAgger and behavioral cloning.

In this paper, we present a change detection algorithm that can run in real time as part of a backend-based stream processing pipeline. It can process the floating car data collected by series-production vehicles to detect changes in an automotive high definition digital (HD) map used for automated driving. The algorithm uses a particle filter approach with odometry, GNSS and landmark readings to localize the vehicle within the digital map. While all particles together represent the probability distribution for the vehicle's position at a given time, each individual particle also serves as a hypothesis about the vehicle's position. This is used to compute various metrics for how well the current sensor readings match the world model encoded in the HD map. The different metrics are evaluated by a number of weak classifiers that are used as input for a trained Adaboost classifier. The achievable detection rate of a single vehicle is then compared to that of a simple crowd-based approach, where each vehicle votes on whether or not the current section of the road has changed.

Autonomous navigation requires an accurate model or map of the environment. While dramatic progress in the prior two decades has enabled large-scale simultaneous localization and mapping (SLAM), the majority of existing methods rely on non-linear optimization techniques to find the maximum likelihood estimate (MLE) of the robot trajectory and surrounding environment. These methods are prone to local minima and are thus sensitive to initialization. Several recent papers have developed optimization algorithms for the Pose-Graph SLAM problem that can certify the optimality of a computed solution. Though this does not guarantee a priori that this approach generates an optimal solution, a recent extension has shown that when the noise lies within a critical threshold that the solution to the optimization algorithm is guaranteed to be optimal. To address the limitations of existing approaches, this paper illustrates that the Pose-Graph SLAM and Landmark SLAM can be formulated as polynomial optimization programs that are sum-of-squares (SOS) convex. This paper then describes how the Pose-Graph and Landmark SLAM problems can be solved to a global minimum without initialization regardless of noise level using the sparse bounded degree sum-of-squares (Sparse-BSOS) optimization method. Finally, the superior performance of the proposed approach when compared to existing SLAM methods is illustrated on graphs with several hundred nodes.

In this work, a passive physical human-robot interaction (pHRI) controller is proposed to intraoperatively ensure that sensitive tissues will not be damaged by the robot's tool. The proposed scheme uses the point cloud of the restricted region's surface as constraint definition and Artificial Potential fields for constraint enforcement. The controller is proven to be passive with respect to the interaction force and to guarantee constraint satisfaction in all cases. The proposed methodology is experimentally validated by the kinesthetic guidance of a KUKA LWR4+ robot's end-effector driving a virtual slave KUKA in the vicinity of a 3D point-cloud of a kidney and its adjacent vessels.

Sequential Convex Programming (SCP) has recently seen a surge of interest as a tool for trajectory optimization. However, most available methods lack rigorous performance guarantees and they are often tailored to specific optimal control setups. In this paper, we present GuSTO (Guaranteed Sequential Trajectory optimization), an algorithmic framework to solve trajectory optimization problems for control-affine systems with drift. GuSTO generalizes earlier SCP-based methods for trajectory optimization (by addressing, for example, goal-set constraints and problems with either fixed or free final time) and enjoys theoretical convergence guarantees in terms of convergence to, at least, a stationary point. The theoretical analysis is further leveraged to devise an accelerated implementation of GuSTO, which originally infuses ideas from indirect optimal control into an SCP context. Numerical experiments on a variety of trajectory optimization setups show that GuSTO generally outperforms current state-of-the-art approaches in terms of success rates, solution quality, and computation times.