ICRA 2020 Papers

We present an approach for the autonomous, robust execution of peg-in-hole assembly tasks. We build on a sampling-based state estimation framework, in which samples are weighted according to their consistency with the position and joint torque measurements. The key idea is to reuse these samples in a motion generation step, where they are assigned a second task-specific weight. The algorithm thereby guides the peg towards the goal along the configuration space. An advantage of the approach is that the user only needs to provide: the geometry of the objects as mesh data, as well as a rough estimate of the object poses in the workspace, and a desired goal state. Another advantage is that the local, online nature of our algorithm leads to robust behavior under uncertainty. The approach is validated in the case of our robotic setup and under varying uncertainties for the classical peg-in-hole problem subject to two different geometries.

This article presents a path tracking control strategy for a quadcopter to follow a time varying curve. The control is based on artificial vector fields. The construction of the field is based on a well known technique in the literature. Next, control laws are developed to impose the behavior of the vector field to a second order integrator model. Finally, control laws are developed to impose the dynamics of the controlled second order integrator to a quadcopter model, which assumes the thrust and the angular rates as input commands. Asymptotic convergence of the whole system is proved by showing that the individual systems in cascade connection are input-to-state stable. We also analyze the influence of norm-bounded disturbances in the control inputs to evaluate the robustness of the controller. We show that bounded disturbances originate limited deviations from the target curve. Simulations and a real robot experiment exemplify and validate the developed theory.

Capturing large objects of unknown shape and orientation remains a challenge for most robotic grippers. We present a highly under-actuated gripper well suited for this task. Prior work shows two primary limitations to these grippers: the grip force of each link tends to decrease as the number of links increases, and the stability of an under-actuated linkage depends on the configuration of the links so grippers with many links are unlikely to be stable for arbitrary surfaces. We address these concerns by implementing two complementary methods of stabilization: using high-friction materials and scaling forces into the surface. We show that gecko-inspired adhesives provide an adhesion-controlled friction that can stabilize the gripper and improve grasp performance without the need of large normal forces. The under-actuated linkages also conform around arbitrary shapes and provide capability beyond prior adhesion-based grippers. With these high-friction interfaces, we show highly under-actuated linkages successfully grasp in many configurations without strict stability. The gripper is capable of holding over 30 N and consists of two tendon driven linkages that are each 65 cm long. This type of gripper is well suited for tasks without a predefined target geometry or orientation such as satellite servicing.

Given a set of 3D-to-2D point correspondences corrupted by outliers, we aim to robustly estimate the absolute camera pose. Existing methods robust to outliers either fail to guarantee high robustness and efficiency simultaneously, or require an appropriate initial pose and thus lack generality. In contrast, we propose a novel approach based on the robust "L2-minimizing estimate" (L2E) loss. We first define a novel cost function by integrating the projection constraint into the L2E loss. Then to efficiently obtain the global minimum of this function, we propose a hybrid strategy of a local optimizer and branch-and-bound. For branch-and-bound, we derive effective function bounds. Our approach can handle high outlier ratios, leading to high robustness. It can run reliably regardless of whether the initial pose is appropriate, providing high generality. Moreover, given a decent initial pose, it is suitable for real-time applications. Experiments on synthetic and real-world datasets showed that our approach outperforms state-of-the-art methods in terms of robustness and/or efficiency.

This paper presents a robust and accurate approach for the rigid registration of pre-operative and intraoperative point sets in image-guided surgery (IGS). Three challenges are identified in the pre-to-intraoperative registration: the intra-operative 3D data (usually forms a 3D curve in space) (1) is often contaminated with noise and outliers; (2) usually only covers a partial region of the whole pre-operative model; (3) is usually sparse. To tackle those challenges, we utilize the tangent vectors extracted from the sparse intraoperative data points and the normal vectors extracted from the pre-operative model points. Our first contribution is to formulate a novel probabilistic distribution of the error between a pair of corresponding tangent and normal vectors. The second contribution is, based on the novel distribution, we formulate the registration of two multi-dimensional (6D) point sets as a maximum likelihood (ML) problem and solve it under the expectation maximization (EM) framework. Our last contribution is, in order to facilitate the computation process, the derivatives of the objective function with respect to desired parameters are presented. We conduct extensive experiments to demonstrate that our approach outperforms the state-of-the-art methods. Importantly, in the context of anteriro cruciate ligament (ACL) reconstruction, our method can achieve as low as 0.6795 mm mean target registration error (TRE) value with considerable noises and very limited overlapping ratios.

In this paper, we present an on-board vision-based approach for avoidance of moving obstacles in dynamic environments. Our approach relies on an efficient obstacle detection and tracking algorithm based on depth image pairs, which provides the estimated position, velocity and size of the obstacles. Robust collision avoidance is achieved by formulating a chance-constrained model predictive controller (CC-MPC) to ensure that the collision probability between the micro aerial vehicle (MAV) and each moving obstacle is below a specified threshold. The method takes into account MAV dynamics, state estimation and obstacle sensing uncertainties. The proposed approach is implemented on a quadrotor equipped with a stereo camera and is tested in a variety of environments, showing effective on-line collision avoidance of moving obstacles.

We present a novel, robust sound source localization algorithm considering back-propagation signals. Sound propagation paths are estimated by generating direct and reflection acoustic rays based on ray tracing in a backward manner. We then compute the back-propagation signals by designing and using the impulse response of the backward sound propagation based on the acoustic ray paths. For identifying the 3D source position, we use a well-established Monte Carlo localization method. Candidates for a source position are determined by identifying convergence regions of acoustic ray paths. Those candidates are validated by measuring similarities between back-propagation signals, under the assumption that the back-propagation signals of different acoustic ray paths should be similar near the ground-truth sound source position. Thanks to considering similarities of back-propagation signals, our approach can localize a source position with an averaged error of 0.55 m in a room of 7 m by 7 m area with 3 m height in tested environments. We also place additional 67 dB and 77 dB white noise at the background, to test the robustness of our approach. Overall, we observe a 7 % to 100 % improvement in accuracy over the state-of-the-art method.

Gradient-based trajectory optimization (GTO) has gained wide popularity for quadrotor trajectory replanning. However, it suffers from local minima, which is not only fatal to safety but also unfavorable for smooth navigation. In this paper, we propose a replanning method based on GTO addressing this issue systematically. A path-guided optimization (PGO) approach is devised to tackle infeasible local minima, which improves the replanning success rate significantly. A topological path searching algorithm is developed to capture a collection of distinct useful paths in 3-D environments, each of which then guides an independent trajectory optimization. It activates a more comprehensive exploration of the solution space and output superior replanned trajectories. Benchmark evaluation shows that our method outplays state-of-the-art methods regarding replanning success rate and optimality. Challenging experiments of aggressive autonomous flight are presented to demonstrate the robustness of our method. We will release our implementation as an open-source package1.

We propose a novel RGB-D camera tracking system that robustly reconstructs hand-held RGB-D camera sequences. The robustness of our system is achieved by two independent features of our method: adaptive visual odometry (VO) and integer programming-based key-frame selection. Our VO method adaptively interpolates the camera motion results of the direct VO (DVO) and the iterative closed point (ICP) to yield more optimal results than existing methods such as Elastic-Fusion. Moreover, our key-frame selection method locates globally optimum key-frames using a comprehensive objective function in a deterministic manner rather than heuristic or experience-based rules that prior methods mostly rely on. As a result, our method can complete reconstruction even if the camera fails to be tracked due to discontinuous camera motions, such as kidnap events, when conventional systems need to backtrack the scene. We validated our tracking system on 25 TUM benchmark sequences against state-of-the-art works, such as ORBSLAM2, Elastic-Fusion, and DVO SLAM, and experimentally showed that our method has smaller and more robust camera trajectory errors than these systems.

In reinforcement learning (RL), an autonomous agent learns to perform complex tasks by maximizing an exogenous reward signal while interacting with its environment. In real world applications, test conditions may differ substantially from the training scenario and, therefore, focusing on pure reward maximization during training may lead to poor results at test time. In these cases, it is important to trade-off between performance and robustness while learning a policy. While several results exist for robust, model-based RL, the model-free case has not been widely investigated. In this paper, we cast the robust, model-free RL problem as a multi-objective optimization problem. To quantify the robustness of a policy, we use delay margin and gain margin, two robustness indicators that are common in control theory. We show how these metrics can be estimated from data in the model-free setting. We use multi-objective Bayesian optimization (MOBO) to solve efficiently this expensive-to-evaluate, multi-objective optimization problem. We show the benefits of our robust formulation both in sim-to-real and pure hardware experiments to balance a Furuta pendulum.

We propose a framework for safe reinforcement learning that can handle stochastic nonlinear dynamical systems. We focus on the setting where the nominal dynamics are known, and are subject to additive stochastic disturbances with known distribution. Our goal is to ensure the safety of a control policy trained using reinforcement learning, e.g., in a simulated environment. We build on the idea of model predictive shielding (MPS), where a backup controller is used to override the learned policy as needed to ensure safety. The key challenge is how to compute a backup policy in the context of stochastic dynamics. We propose to use a tube-based robust nonlinear model predictive controller (NMPC) as the backup controller. We estimate the tubes using sampled trajectories, leveraging ideas from statistical learning theory to obtain high-probability guarantees. We empirically demonstrate that our approach can ensure safety in stochastic systems, including cart-pole and a non-holonomic particle with random obstacles.

3D point cloud maps are an accumulation of laser scans obtained at different positions and times. Since laser scans represent a snapshot of the surrounding at the time of capture, they often contain moving objects which may not be observed at all times. Dynamic objects in point cloud maps decrease the quality of maps and affect localization accuracy, hence it is important to remove the dynamic objects from 3D point cloud maps. In this paper, we present a robust method to remove dynamic objects from 3D point cloud maps. Given a registered set of 3D point clouds, we build an occupancy map in which the voxels represent the occupancy state of the volume of space over an extended time period. After building the occupancy map, we use it as a filter to remove dynamic points in lidar scans before adding the points to the map. Furthermore, we accelerate the process of building occupancy maps using object detection and a novel voxel traversal method. Once the occupancy map is built, dynamic object removal can run in real-time. Our approach works well on wide urban roads with stopped or moving traffic and the occupancy maps get better with the inclusion of more lidar scans from the same scene.

Formula Student Driverless (FSD) is a competition where student teams compete to build an autonomous racecar. The main dynamic event in FSD is trackdrive, where the racecar traverses an unknown track with lanes demarcated by cones. One major challenge of the event is to determine the boundaries of the lane from cones perceived online despite false positive cone detections and sharp turns. We present a binary integer optimization to address this problem by leveraging a priori knowledge from competition rules on parameters such as average cone spacings and minimum track width. In this paper, we describe our approach, and analyze its latency, accuracy, and robustness to false positive cone detections. This approach is used on-board to solve the lane detection problem during the competition in real-time.

Accurate 6D object pose estimation is fundamental to robotic manipulation and grasping. Previous methods follow a local optimization approach which minimizes the distance between closest point pairs to handle the rotation ambiguity of symmetric objects. In this work, we propose a novel discrete- continuous formulation for rotation regression to resolve this local-optimum problem. We uniformly sample rotation anchors in SO(3), and predict a constrained deviation from each anchor to the target, as well as uncertainty scores for selecting the best prediction. Additionally, the object location is detected by aggregating point-wise vectors pointing to the 3D center. Experiments on two benchmarks: LINEMOD and YCB-Video, show that the proposed method outperforms state-of-the-art approaches. Our code is available at https://github.com/mentian/object-posenet.

This paper introduces a method for robotic steering of a flexible needle inside moving and deformable tissues. The method relies on a set of objective functions allowing to automatically steer the needle along a predefined path. In order to follow the desired trajectory, an inverse problem linking the motion of the robot end effector with the objective functions is solved using a Finite Element simulation. The main contribution of the article is the new constraint-based formulation of the objective functions allowing to: 1) significantly reduce the computation time; 2) increase the accuracy and stability of the simulation-guided needle insertion. The method is illustrated, and its performances are characterized in a realistic framework, using a direct simulation of the respiratory motion generated from in vivo data of a pig. Despite the highly non-linear behavior of the numerical simulation and the significant deformations occurring during the insertion, the obtained performances enable the possibility to follow the trajectory with the desired accuracy for medical purpose.

Existing approaches for robotic control of magnetic swarms are not capable of generating magnetic aggregates precisely in an arbitrarily specified target region in a fluidic flow environment. Such a swarm control capability is demanded by medical applications such as clinical embolization (i.e., localized clogging of blood vessels). This paper presents a new magnetic swarm control strategy to generate aggregates only in a specified target region under fluidic flow. Within the target region, the magnetic field generates sufficiently large magnetic forces among magnetic particles to maintain the aggregates’ integrity at the junctions of blood vessels. In contrast, unintended aggregates outside the target region are disassembled by fluidic shear. The aggregation control approach achieved a mean absolute error of 0.15 mm in positioning a target region and a mean absolute error of 0.30 mm in controlling the target region’s radius. With thrombin coating, 1 µm magnetic particles were controlled to perform embolization both in vitro (using microfluidic channel networks) and ex vivo (using porcine tissue). Experiments proved the effectiveness of the swarm control technique for on-demand, targeted embolization.

The automatic parts feeding of multiple objects is an unsolved problem in the manufacturing industry. In this paper, we tackle the problem by proposing a multi-robot system. The system comprises three sub-components which perform bin-picking, regrasping, and kitting. The three subcomponents divide and conquer the automatic multiple parts feeding problem by considering a coarse-to-fine manipulation process. Multiple robot arms are connected in series as a pipeline. The robots are separated into three groups to perform the roles of each sub-component. The accuracy of the state and manipulation are getting higher along with the changes of the sub-components in the pipeline. In the experimental section, the performance of the system is evaluated by using the Mean Picks Per Hour (MPPH) metric and success rate, which are compared to traditional parts feeder and manual labor. The results show that the Mean Picks Per Hour (MPPH) of the proposed system is 351 with eleven various-shaped industrial parts, which is faster than the state-of-the-art robotic bin-picking system. The lead time of the proposed system for new parts is less than that of a traditional parts feeders and/or manual labor.

In biology, fluorescent dyes are routinely used for biochemical measurements such as pH and ion concentrations. They, especially when used for detecting a low concentration of ions, suffer from low signal-to-noise ratios (SNR); and increasing the concentration of fluorescent dyes causes more sever cytotoxicity. We invented a new approach that uses a low amount of fluorescent dye-coated magnetic nanoparticles for on-demand, accurately aggregating the nanoparticles and thus fluorescent dyes in a local region inside a cell for intracellular measurement. Experiments proved this approach is capable of achieving a significantly higher SNR and lower cytotoxicity. Different from existing magnetic micromanipulation systems that generate large swarms (several microns and above) or cannot move the generated swarm to an arbitrary position, we developed a five-pole magnetic micromanipulation system and technique for generating a small swarm (e.g., 1 µm; capable of generating a magnetic swarm from 0.52 µm to 52.7 µm with an error <7.5 %) and accurately positioning the small swarm (position control accuracy: 0.76 µm). As an example, the system performed intracellular pH mapping using a 1 µm swarm of pH sensitive fluorescent dye-coated magnetic nanoparticles. The swarm had an SNR inside a cell 10 times that by the traditional method, i.e., global dye treatment, with both cases using the same fluorescent dye concentration. Our intracellular measurement results, for the first time, quantitatively revealed the existence of pH gradient and polarized pH distribution in live migrating cells.

To be effective in unstructured and changing environments, robots must learn to recognize new objects. Deep learning has enabled rapid progress for object detection and segmentation in computer vision; however, this progress comes at the price of human annotators labeling many training examples. This paper addresses the problem of extending learning-based segmentation methods to robotics applications where annotated training data is not available. Our method enables pixelwise segmentation of grasped objects. We factor the problem of segmenting the object from the background into two sub-problems: (1) segmenting the robot manipulator and object from the background and (2) segmenting the object from the manipulator. We propose a kinematics-based foreground segmentation technique to solve (1). To solve (2), we train a self-recognition network that segments the robot manipulator. We train this network without human supervision, leveraging our foreground segmentation technique from (1) to label a training set of images containing the robot manipulator without a grasped object. We demonstrate experimentally that our method outperforms state-of-the-art adaptable in-hand object segmentation. We also show that a training set composed of automatically labelled images of grasped objects improves segmentation performance on a test set of images of the same objects in the environment.