IROS 2019 Papers

Event-based cameras are a novel asynchronous sensing modality that provide exciting benefits, such as the ability to track fast moving objects with no motion blur and low latency, high dynamic range, and low power consumption. Given the low latency of the cameras, as well as their ability to work in challenging lighting conditions, these cameras are a natural fit for reactive problems such as fast local structure estimation. In this work, we propose a fast method to perform structure estimation for vehicles traveling in a roughly 2D environment (e.g. in an environment with a ground plane). Our method transfers the method of plane and parallax to events, which, given the homography to a ground plane and the pose of the camera, generates a warping of the events which removes the optical flow for events on the ground plane, while inducing flow for events above the ground plane. We then estimate dense flow in this warped space using a self-supervised neural network, which provides the height of all points in the scene. We evaluate our method on the Multi Vehicle Stereo Event Camera dataset, and show its ability to rapidly estimate the scene structure both at high speeds and in low lighting conditions.

In this research, we tackle the challenge of picking only one object from a randomly stacked pile where the objects can potentially be tangled. No solution has been proposed to solve this challenge due to the complexity of picking one and only one object from the bin of tangled objects. Therefore, we propose a method for avoiding the situation where a robot picks multiple objects. In our proposed method, first, grasping pose candidates are computed by using the graspability index. Then, a Convolutional Neural Network (CNN) is trained to predict whether or not the robot can pick one and only one object from the bin. Additionally, since a physics simulator is used to collect data to train the CNN, an automatic picking system can be built. The effectiveness of the proposed method is confirmed through experiments on robot Nextage and compare with previous bin-picking methods.

Stable dynamical systems are a flexible tool to plan robotic motions in real-time. In the robotic literature, dynamical system motions are typically planned without considering possible limitations in the robot's workspace. This work presents a novel approach to learn workspace constraints from human demonstrations and to generate motion trajectories for the robot that lie in the constrained workspace. Training data are incrementally clustered into different linear subspaces and used to fit a low dimensional representation of each subspace. By considering the learned constraint subspaces as zeroing barrier functions, we are able to design a control input that keeps the system trajectory within the learned bounds. This control input is effectively combined with the original system dynamics preserving eventual asymptotic properties of the unconstrained system. Simulations and experiments on a real robot show the effectiveness of the proposed approach.

Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video captioning network (CNet). GNet performs two functions: providing grasp solutions and extracting the local features for the object of interests in robotic manipulation. CNet outputs the captioning results by fusing the features of both full frames and local objects. Experimental results on UR5 robotic arm show that our method could produce more accurate command from video demonstration than state-of-the-art work, thereby leading to more robust grasping performance.

We address the problem of 3D object detection from 2D monocular images in autonomous driving scenarios. We propose to lift the 2D images to 3D representations using learned neural networks and leverage existing networks working directly on 3D data to perform 3D object detection and localization. We show that, with carefully designed training mechanism and automatically selected minimally noisy data, such a method is not only feasible, but gives higher results than many methods working on actual 3D inputs acquired from physical sensors. On the challenging KITTI benchmark, we show that our 2D to 3D lifted method outperforms many recent competitive 3D networks while significantly outperforming previous state-of-the-art for 3D detection from monocular images. We also show that a late fusion of the output of the network trained on generated 3D images, with that trained on real 3D images, improves performance. We find the results very interesting and argue that such a method could serve as a highly reliable backup in case of malfunction of expensive 3D sensors, if not potentially making them redundant, at least in the case of low human injury risk autonomous navigation scenarios like warehouse automation.

We address variants of multi-robot path planning in graphs (MRPP). We assume robots placed in vertices of an undirected graph with at most one robot per vertex. Robots can move across edges while various problem specific constraints must be satisfied. We introduce a general problem formulation that encompasses known types of robot relocation problems such as multi-robot path planning (MRPP), token swapping (TSWAP), token rotation (TROT), and token permutation (TPERM). We generalize SMT-CBS, a recent solving approach for MRPP based on satisfiability modulo theories (SMT). SMT- CBS compiles MRPP lazily within the SMT framework, starting with the basic model that is refined with a collision resolution constraints whenever collisions between robots occur in the current solution. We show modifications the SMT-CBS algorithm for variants of MRPP and evaluate them experimentally.

In this paper, we introduce a new public dataset for 6D object pose estimation and instance segmentation for industrial bin-picking. The dataset comprises both synthetic and real-world scenes. For both, point clouds, depth images, and annotations comprising the 6D pose (position and orientation), a visibility score, and a segmentation mask for each object are provided. Along with the raw data, a method for precisely annotating real-world scenes is proposed.To the best of our knowledge, this is the first public dataset for 6D object pose estimation and instance segmentation for bin-picking containing sufficiently annotated data for learning-based approaches. Furthermore, it is one of the largest public datasets for object pose estimation in general. The dataset is publicly available at http://www.bin-picking.ai/en/ dataset.html.

In this paper, we propose a new algorithm that derives lane marking maps from crowdsourced data. We process the data in four steps: (i) We make use of a point landmark map and, if available, an existent lane marking map for trajectory optimization and alignment, (ii) use a custom DBSCAN variant to cluster observations that belong to the same lane marking, (iii) apply a novel graph fitting approach to extract lane marking dashes, lines and even complex structures such as splits and merges and (iv) optimize the graph geometry with domain knowledge. The process of point-landmark- and lane marking-based trajectory alignment and the lane marking derivation is repeated iteratively to improve the results. Evaluation is carried out on a 9km highway section by comparison with high accuracy aerial photographs and manually labeled ground truth lane markings.

We describe the concept, design and implementation of a unique manipulator system for a multirotor aerial vehicle. The proposed manipulator system consists of three robotic arms attached to a multirotor airframe with the objective to provide the ability to manipulate single or multiple objects as well as aid in complex navigation tasks by doing contact based obstacle avoidance and acting as adaptive landing gear in uneven terrain. In this paper we primarily focus on the description and experimentation of one of the tasks achievable by the proposed aerial multi-manipulator system: landing a multirotor aerial vehicle on an uneven surface. Deploying the on-board manipulator as landing gears reduces the hardware carried by the vehicle while additionally providing the ability to land in unstructured and commonly difficult to land terrain.

Recently, in the search for alternatives to conventional robots, various types of soft actuators and their applications have been studied. In particular, pneumatic soft actuators have the advantage of being lightweight and high power. One of the uses of these soft actuators is as a substitute for an electric motor to actuate the joint of a structure, such as a link mechanism. Another use involves their fusion with a flexible structure. The features of this concept are a simplified system, in which almost all interfaces can be configured as flexible structures. However, because these actuatable flexible structures are made with rubber, silicone, or flexible resin, it is difficult for them to support their own weight owing to the effect of the square-cube law in the case of increasing size. Hence, these structures are limited to a size of approximately 1 × 10-2 1 × 10-1 m. If an actuatable flexible structure with a size of 1 × 100 m can be realized, the concept of soft actuator-flexible structure fusion is expected to provide novel solutions and applications. Herein, as a feasibility study, large actuatable flexible structures were developed. The proposed structure, LayerCAKE, is a laminated open-cell and closed-cell foam structure; cell foam is a lightweight and flexible material that can be used to realize large actuatable flexible structures. LayerCAKE is actuated by using the concept that openand closed-cell foams contract differently when they are vacuumed. The bending-motion model and was experimentally verified, and different types of LayerCAKE models that could exhibit various types of motion were developed and tested. Furthermore, a complex-shaped LayerCAKE model (in the shape of a human hand) was developed. A large actuatable flexible structure of approximately 900 mm was realized. Furthermore, it was confirmed that the bending motion could be controlled by pressure.

In a context of autonomous robots, one of the most important tasks is to ensure the safety of the robot and its surrounding. The risk of navigation is usually said to be the probability of collision. This notion of risk is not well defined in the literature, especially when dealing with occupancy grids. The Bayesian occupancy grid is the most used method to deal with complex environments. However, this is not fitted to compute the risk along a path by its discrete nature. In this article, we present a new way to store the occupancy of the environment that allows the computation of risk along a given path. We then define the risk as the force of collision that would occur for a given obstacle. Using this framework, we are able to generate navigation paths ensuring the safety of the robot.

This paper presents a tightly-coupled multi-sensor fusion algorithm termed LiDAR-inertial-camera fusion (LIC-Fusion), which efficiently fuses IMU measurements, sparse visual features, and extracted LiDAR points. In particular, the proposed LIC-Fusion performs online spatial and temporal sensor calibration between all three asynchronous sensors, in order to compensate for possible calibration variations. The key contribution is the optimal (up to linearization errors) multi-modal sensor fusion of detected and tracked sparse edge/surf feature points from LiDAR scans within an efficient MSCKF-based framework, alongside sparse visual feature observations and IMU readings. We perform extensive experiments in both indoor and outdoor environments, showing that the proposed LIC-Fusion outperforms the state-of-the-art visual-inertial odometry (VIO) and LiDAR odometry methods in terms of estimation accuracy and robustness to aggressive motions.

We consider the problem of leveraging prior experience to generate roadmaps in sampling-based motion planning. A desirable roadmap is one that is sparse, allowing for fast search, with nodes spread out at key locations such that a low- cost feasible path exists. An increasingly popular approach is to learn a distribution of nodes that would produce such a roadmap. State-of-the-art is to train a conditional variational auto-encoder (CVAE) on the prior dataset with the shortest paths as target input. While this is quite effective on many problems, we show it can fail in the face of complex obstacle configurations or mismatch between training and testing. We present an algorithm LEGO that addresses these issues by training the CVAE with target samples that satisfy two important criteria. Firstly, these samples belong only to bottleneck regions along near-optimal paths that are otherwise difficult- to-sample with a uniform sampler. Secondly, these samples are spread out across diverse regions to maximize the likelihood of a feasible path existing. We formally define these properties and prove performance guarantees for LEGO. We extensively evaluate LEGO on a range of planning problems, including robot arm planning, and report significant gains over heuristics as well as learned baselines.

Two-fingered manipulation robotic systems are widely used in many application sand notably at small scales. The commonly employed solution is based on the attachment of a gripper on a robot. This paper introduces a new eight degrees of freedom (DoF) mechanism intended for two-fingered dexterous manipulation. The eight DoFs are obtained via a parallel architecture moved by eight actuators that are attached to the base of the robot. The novelty lies in the use of a 2-DoF configurable platform that insures two relative rotations of the gripper jaws (opening and twisting) in addition to the 6-DoF Cartesian movements. The paper presents the kinematics and the design of a proof-of-concept able to grasp, roll, translate and rotate objects in a single compact design. A prototype is operated to manipulate and insert a 2-mm screw.

Soft material robotic systems provide increased adaptability and flexibility compared to conventional rigid metal robots. The soft systems benefit from their inherent compliance, which enables them to be used in applications that require safe interaction between humans and robots or manipulation in cluttered environment. Despite advancements in recent years research on soft material robots still needs to make progress in terms of modeling for model based control or path planning. The high nonlinearity of soft material robots makes efficient and accurate modeling difficult. In this work we introduce a kinematic modeling approach based on cubic hermite splines. The method is applied to a soft pneumatic actuator and evaluated against the widely used constant curvature approach. The hermite spline offers the possibility of accurate shape reconstruction from simulated or measured deformation data. Both the shape of a robot's segment and its orientation can be approximated this way. In this paper a machine learning approach is used to train the kinematic relation between actuating pressure and configuration parameters.

The motion planners used in self-driving vehicles need to generate trajectories that are safe, comfortable, and obey the traffic rules. This is usually achieved by two modules: behavior planner, which handles high-level decisions and produces a coarse trajectory, and trajectory planner that generates a smooth, feasible trajectory for the duration of the planning horizon. These planners, however, are typically developed separately, and changes in the behavior planner might affect the trajectory planner in unexpected ways. Furthermore, the final trajectory outputted by the trajectory planner might differ significantly from the one generated by the behavior planner, as they do not share the same objective. In this paper, we propose a jointly learnable behavior and trajectory planner. Unlike most existing learnable motion planners that address either only behavior planning, or use an uninterpretable neural network to represent the entire logic from sensors to driving commands, our approach features an interpretable cost function on top of perception, prediction and vehicle dynamics, and a joint learning algorithm that learns a shared cost function employed by our behavior and trajectory components. Experiments on real-world self-driving data demonstrate that jointly learned planner performs significantly better in terms of both similarity to human driving and other safety metrics, compared to baselines that do not adopt joint behavior and trajectory learning.

This paper deals with the problem of accurately computing and estimating joint velocity and acceleration in robotic manipulators. Generally, it is well known that numerical differentiation of noisy position signals even with significant filtering is no viable solution. This is especially true for computing joint acceleration. Specifically, our solution to this problem fuses joint position measurement with link accelerometers, which are affordable and easy to install. Since the sensor readings are affected by noise, drift and bias, suitable data fusion and filtering methods are proposed for improving the estimation for practical use. Simulation results based on a realistic dynamics model of a 7-DoF robot including various parasitic effects and experimental results with a 7-DoF robot demonstrate the effectiveness of our approach. This method would have multiple use, e.g., in monitoring external joint torques and handle possibly unforeseen collisions. Furthermore, other applications such as load identification and compensation as well as state feedback linearization for flexible joint robots could finally become possible also practical.

This paper investigates dynamic and compliant control based on joint output torque estimation for electrically actuated quadruped robots with large-reduction-ratio harmonic gear. Compared with position control, force control exhibits better performance of dynamics and compliance for the robot's interactions with complex environments. However, force control without direct feedbacks from torque sensors may come with poor tracking performance of joint compliance when the robot equipped with gears of high reduction. To solve this problem, we propose a new method to estimate joint torque from motor current and rotation velocity detected on each joint, using a more precise friction model of the harmonic gear. We also introduce a pre-stance phase to the whole cycle of leg alternating swing/stance based on hybrid force and position control to dynamically absorb feet impacts on the ground. Our controller performance is validated by standing experiment and walking experiment.

In Robot-Assisted Surgery, predicting surgeon attributes such as task workload, operation performance, and expertise levels is important in providing tailored assistance. This paper proposes Joint Inference for Surgeon Attributes Prediction (JISAP), a computational framework to jointly infer surgeon attributes (i.e., task workload, operation performance, and expertise level) from multimodal physiological signals (heart rate variability, wrist motion, electrodermal, electromyography, and electroencephalogram activity). JISAP was evaluated with a dataset of twelve surgeons operating on the da Vinci Skills Simulator. It was found that JISAP can simultaneously predict surgeon attributes with a percentage error of 11.05%. Additionally, joint inference was found to outperform isolated inference with a boost of 10%.