ICRA 2021 Papers

Microinjection technology is applied widely in biomedical research for the purposes of gene manipulation and microinsemination. Generally, microinjection is performed under an optical microscope environment through image presentation of the targets. To perform the microinjection process, it is necessary to place multiple cells in the same droplet and perform multiple injections. This process requires observation at different magnifications for the injection and embryo transfer processes, with the operator required to change the magnification and light intensity each time. The complexity of the process can lead to variations in the accuracy, reproducibility, and productivity of the course of multiple microinjections. To simplify the microinjection process and reduce the workload on the operator, we propose a micromanipulation system that enables both wide-range and high-resolution video shooting with free viewpoint selection. The effectiveness of the proposed system is verified through microinjection experiments using porcine embryos.

We propose a learning-based navigation system for reaching visually indicated goals and demonstrate this system on a real mobile robot platform. Learning provides an appealing alternative to conventional methods for robotic navigation: instead of reasoning about environments in terms of geometry and maps, learning can enable a robot to learn about navigational affordances, understand what types of obstacles are traversable (e.g., tall grass) or not (e.g., walls), and generalize over patterns in the environment. However, unlike conventional planning algorithms, it is harder to change the goal for a learned policy during deployment. We propose a method for learning to navigate towards a goal image of the desired destination. By combining a learned policy with a topological graph constructed out of previously observed data, our system can determine how to reach this visually indicated goal even in the presence of variable appearance and lighting. Three key insights, waypoint proposal, graph pruning and negative mining, enable our method to learn to navigate in real-world environments using only offline data, a setting where prior methods struggle. We instantiate our method on a real outdoor ground robot and show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning, including other methods that incorporate reinforcement learning and search. We also study how ViNG generalizes to unseen environments and evaluate its ability to adapt to such an environment with growing experience. Finally, we demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection. We encourage the reader to visit the project website for videos of our experiments and demonstrations 1.

Stable walking in real-world environments is a challenging task for humanoid robots, especially when considering the dynamic disturbances, e.g., caused by external perturbations that may be encountered during locomotion. The varying nature of disturbance necessitates high adaptability. In this paper, we propose an enhanced Nonlinear Model Predictive Control (NMPC) approach for robust and adaptable walking – we term it versatile locomotion, by limiting both the Center of Pressure (CoP) and Divergent Component of Motion (DCM) movements. Due to utilization of the Nonlinear Inverted Pendulum plus Flywheel model, the robot is endowed with the capabilities of CoP manipulation (if equipped with finitesized feet), step location adjustment, upper body rotation, and vertical height variation. Considering the feasibility constraints, especially the usage of relaxed CoP constraints, the NMPC scheme is established as a Quadratically Constrained Quadratic Programming problem, which is solved efficiently by Sequential Quadratic Programming with enhanced solvability. Simulation experiments demonstrate the effectiveness of our method to recruit optimal hybrid strategies in order to realize versatile locomotion, for the robot with finite-sized or point feet.

The learning-from-observation (LfO) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfO system encodes a human demonstration into a series of execution units for a robot, which are referred to as task models. Although previous research has proposed successful task-model encoders, there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises, such as cluttered objects or unrelated human body movements. Inspired by the function of verbal instructions guiding an observer’s visual attention, we propose a verbal focus-of-attention (FoA) system (i.e., spatiotemporal filters) to guide a task-model encoder. For object manipulation, the system first recognizes the name of a target object and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas in which the target object existed in the demonstration. The system then detects the timings of grasp and release that occurred in the filtered areas. The timings serve as a when-to-look FoA filter to confine the period of object manipulation. Finally, a task-model encoder recognizes the task models by employing the FoA filters. We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network. The contributions of this study are as follows: (1) to propose a verbal FoA for LfO, (2) to design an algorithm to calculate FoA filters from verbal input, and (3) to demonstrate the effectiveness of a verbal FoA in localizing an action by comparing it with a state-of-the-art vision system.

The most successful methods for LiDAR-based 3D object detection use sequences of point clouds in order to exploit the increased data density through temporal aggregation. However, common aggregation methods are rarely able to capture fast-moving objects appropriately. These objects are displaced by large distances between frames and naive approaches are not able to successfully leverage the full amount of information spread across time. Yet, especially in autonomous driving scenarios, fast-moving objects are most crucial to detect as they actively take part in highly dynamic traffic situations. This work presents a novel network architecture called VelocityNet which is explicitly designed to temporally align features according to object motion. Our approach extends traditional 3D convolutions by a motion-driven deformation of the convolution kernels across the temporal dimension. The required motion information can be obtained from various sources, ranging from external computation or complementary sensors to an integrated network branch which is trained jointly with the object detection task. The explicit feature alignment allows the training process to focus on the object detection problem and results in a significant increase in detection performance compared to the popular PointPillars baseline, not only for dynamic but also for static objects. We evaluate our approach on the nuScenes dataset and analyze the main reasons for the observed performance gains.

In this paper, we propose a vanishing point aided LiDAR-Visual-Inertial estimator to achieve real-time, low-drift and robust pose estimation. The proposed method is mainly composed of 3 sequential modules, namely IMU-aided vanishing point (VP) detection module, voxel-map based feature depth association module, and visual inertial fixed-lag smoother module. The IMU-aided VP detection module will detect feature points, line segments and vanishing points to establish robust correspondences in successive frames. In particular, we propose to use 1-line RANSAC method to provide stable VP hypotheses and polar grid to accelerate vanishing point hypothesis validation. After that, we propose a novel voxel-map based feature depth association method, to retrieve depth and assign depth to visual feature efficiently. Finally, the visual inertial fixed-lag smoother module is proposed to jointly minimize error terms. Experiments show that our method outperforms the state-of-the-art visual-inertial odometry and LiDAR-visual estimator in both indoor and outdoor environments.

This study validates the effectiveness of a recently developed parallel-actuated shoulder exoskeleton robot for the purpose of characterizing the neuromuscular properties of the human shoulder joint. In particular, shoulder mechanical impedance was quantified, which can be represented by a 2nd order system consisting of spring, damper and inertia. The shoulder exoskeleton robot, which utilizes a new type of 4-bar spherical parallel manipulator (4B-SPM), has inherently low inertia and as a result can provide fast perturbations that are often essential for characterizing neuromuscular properties. The robot was first evaluated by using a physical shoulder mockup with adjustable and known spring and mass properties. The results of the mockup test confirmed the reliability of the robot for the characterization of the mockup properties. Stiffness of the tested springs was accurately quantified with an error of less than 1.6 Nm/rad in any of the tested conditions. A pilot study with 5 human subjects further confirmed that the robot could be successfully used to quantify multi-dimensional human shoulder impedance in both pitch and yaw directions with high reliability (R2 > 0.97). The average human shoulder stiffness and damping at around the neutral arm posture under low muscle activation (< 5% maximum voluntary contraction) were 30.9 Nm/rad and 3.0 Nms/rad, respectively.

We present a dense-indirect SLAM system using external dense optical flows as input. We extend the recent probabilistic visual odometry model VOLDOR [1], by incorporating the use of geometric priors to 1) robustly bootstrap estimation from monocular capture, while 2) seamlessly supporting stereo and/or RGB-D input imagery. Our customized back-end tightly couples our intermediate geometric estimates with an adaptive priority scheme managing the connectivity of an incremental pose graph. We leverage recent advances in dense optical flow methods to achieve accurate and robust camera pose estimates, while constructing fine-grain globally-consistent dense environmental maps. Our open source implementation [https://github.com/htkseason/VOLDOR] operates online at around 15 FPS on a single GTX1080Ti GPU.

The research community presented significant advances in many different Visual-Inertial Navigation System (VINS) algorithms to localize mobile robots or hand-held devices in a 3D environment. While authors of the algorithms of-ten do compare to, at that time, existing competing approaches, their comparison methods, rigor, depth, and repeatability at later points in time have a large spread. Further, with existing simulators and photo-realistic frameworks, the user is not able to easily test the sensitivity of the algorithm under examination with respect to specific environmental conditions and sensor specifications. Rather, tests often include unwillingly many polluting effects falsifying the analysis and interpretations. In addition, edge cases and corresponding failure modes often remain undiscovered due to the limited breadth of the test sequences. Our unified evaluation framework allows, in a fully automated fashion, a reproducible analysis of different VINS methods with respect to specific environmental and sensor parameters. The analyses per parameter are done over a multitude of test sets to obtain both statistically valid results and an average over other, potentially polluting effects with respect to the one parameter under test to mitigate biased interpretations. The automated performance results per method over all tested parameters are then summarized in unified radar charts for a fair comparison across authors and institutions.

In this paper, we develop a novel visual-inertial navigation system with motion constraint (VINS-Motion), which extends the visual-inertial navigation system (VINS) to incorporate vehicle motion constraints for improving the autonomous vehicles localization accuracy. Besides the prior information, IMU measurement residual, and visual measurement residual utilized in VINS, vehicle orientation/velocity constraint is first exploited to constitute motion residual. We minimize the sum of priors and Mahalanobis norms of three kinds of residuals to obtain a maximum posteriori estimation, thus increasing system consistency and accuracy. Stop detection is also added to help eliminate the abnormal jitter of the estimated poses during stopping, thus ensuring reasonability of the trajectory. The pro-posed approach is validated on public datasets and compared against state-of-the-art algorithms, which demonstrates that VINS-Motion achieves significantly higher positioning accuracy.

Recently, quadrotors are gaining significant attention in aerial transportation and delivery. In these scenarios, an accurate estimation of the external force is as essential as the six degree-of-freedom (DoF) pose since it is of vital importance for planning and control of the vehicle. To this end, we propose a tightly-coupled Visual-Inertial-Dynamics (VID) system that simultaneously estimates the external force applied to the quadrotor along with the six DoF pose. Our method builds on the state-of-the-art optimization-based Visual-Inertial system [1], with a novel deduction of the dynamics and external force factor extended from VIMO [2]. Utilizing the proposed dynamics and external force factor, our estimator robustly and accurately estimates the external force even when it varies widely. Moreover, since we explicitly consider the influence of the external force, when compared with VIMO [2] and VINS-Mono [1], our method shows comparable and superior pose accuracy, even when the external force ranges from neglectable to significant. The robustness and effectiveness of the proposed method are validated by extensive real-world experiments and application scenario simulation. We will release an open-source package of this method along with datasets with ground truth force measurements for the reference of the community.

Voxel-based methods have been widely used in point cloud 3D object detection. These methods usually transform points into voxels while suffering from information loss during point cloud voxelization. To address this problem, we propose a novel one-stage Voxelization Information Compensation Network (VIC-Net), which has the ability of loss-free feature extraction. The whole framework consists of a point branch for geometry detail extraction and a voxel branch for efficient proposals generation. Firstly, PointNet++ is adopted to efficiently encode geometry structure features from the raw point clouds. Then based on the encoded point features, two Point2Voxel (P2V) feature fusion modules are proposed to fuse point features with a voxel backbone, including Local P2V and Multi-Scale P2V. The P2V modules respectively integrate local detail features and multi-scale semantic contexts into a sparse voxel backbone. Thirdly, an auxiliary reconstruction loss is employed on the point branch to explicitly guide the point backbone to be aware of real geometry structures. In addition, we extend VIC-Net to a two-stage approach, namely VIC-RCNN, which further utilizes the fine geometry features to refine object locations. Experiments on the KITTI dataset demonstrate that our proposed VIC-Net outperforms other onestage methods and our two-stage method VIC-RCNN achieves new state-of-the-art performance.

Image-based yield detection in agriculture could raise harvest efficiency and cultivation performance of farms. Following this goal, this research focuses on improving instance segmentation of field crops under varying environmental conditions. Five data sets of cabbage plants were recorded under varying lighting outdoor conditions. The images were acquired using a commercial mono camera. Additionally, depth information was generated out of the image stream with Structure-from-Motion (SfM). A Mask R-CNN was used to detect and segment the cabbage heads. The influence of depth information and different colour space representations were analysed. The results showed that depth combined with colour information leads to a segmentation accuracy increase of 7.1%. By describing colour information by colour spaces using light and saturation information combined with depth information, additional segmentation improvements of 16.5% could be reached. The CIELAB colour space combined with a depth information layer showed the best results achieving a mean average precision of 75.

Air Combat is a high-risk activity carried out by trained professionals operating sophisticated equipment. During this activity, a number of trade-offs have to be made, such as the balance between risk and efficiency. A policy that minimizes risk could have very low efficiency, and one that maximizes efficiency may involve very high risk.In this study, we use Reinforcement Learning (RL) to create Control Barrier Functions (CBF) that captures the current risk, in terms of worst-case future separation between the aircraft and an enemy missile. CBFs are usually designed manually as closed-form expressions, but for a complex system such as a guided missile, this is not possible. Instead, we solve an RL problem using high fidelity simulation models to find value functions with CBF properties, that can then be used to guarantee safety in real air combat situations. We also provide a theoretical analysis of what family of RL problems result in value functions that can be used as CBFs in this way.The proposed approach allows the pilot in an air combat scenario to set the exposure level deemed acceptable and continuously monitor the risk related to his/her own safety. Given input regarding acceptable risk, the system limits the choices of the pilot to those that guarantee future satisfaction of the provided bound.

Due to the continuous and flexible nature of continuum robot backbones and the infinite number of parameters required to represent them in configuration space, modeling them accurately and in real-time is challenging. While the constant curvature assumption provides a simple alternative, it is limited in its capabilities as it cannot account for external tip forces. In cases where the backbone deviates from the constant curvature backbone, Euler curves are an interesting alternative for modeling continuum robots. In this paper, we show that a linear approximation of the backbone curvature is sufficiently accurate for estimating the shape of a robot subject to external tip forces. Next, we propose a numerical static model for tendon-driven continuum robots experiencing in-plane external tip forces. In this model, we use Euler arc splines to circumvent the limitations of standard numerical integration schemes required to calculate these curves. The system reduces to solving two nonlinear equations, allowing fast approximation of the backbone shape. The proposed model is validated experimentally on a robot prototype. Average tip error of 3.07% of the robot length is obtained for an average computation time of 0.51 ms.

Anticipating the motion of dynamic objects is critical for making intelligent decisions navigating through an environment while avoiding collisions. In this work, we propose a CNN model that estimates 3D motion of objects using sequences of monocular images. We show that we can train this model without using any manual annotations by using Iterative Closest Points (ICP) to align pointclouds of an object at different points in time. We compare our unsupervised approach to a model that was trained using ground truth supervision, on the KITTI tracking dataset. We further improve our model by training our model on a larger dataset, which would otherwise not be possible due to the lack of ground truth data. We also compare our approach with a 3D object detector that estimates motion using a simple tracking scheme.

Scene flow represents the motion of points in the 3D space, which is the counterpart of the optical flow that represents the motion of pixels in the 2D image. However, it is difficult to obtain the ground truth of scene flow in the real scenes, and recent studies are based on synthetic data for training. Therefore, how to train a scene flow network with unsupervised methods based on real-world data shows crucial significance. A novel unsupervised learning method for scene flow is proposed in this paper, which utilizes the images of two consecutive frames taken by monocular camera without the ground truth of scene flow for training. Our method realizes the goal that training scene flow network with real-world data, which bridges the gap between training data and test data and broadens the scope of available data for training. Unsupervised learning of scene flow in this paper mainly consists of two parts: (i) depth estimation and camera pose estimation, and (ii) scene flow estimation based on four different loss functions. Depth estimation and camera pose estimation obtain the depth maps and camera pose between two consecutive frames, which provide further information for the next scene flow estimation. After that, we used depth consistency loss, dynamic-static consistency loss, Chamfer loss, and Laplacian regularization loss to carry out unsupervised training of the scene flow network. To our knowledge, this is the first paper that realizes the unsupervised learning of 3D scene flow from monocular camera. The experiment results on KITTI show that our method for unsupervised learning of scene flow meets great performance compared to traditional methods Iterative Closest Point (ICP) and Fast Global Registration (FGR). The source code is available at: https://github.com/IRMVLab/3DUnMonoFlow.

Robotic tasks such as manipulation with visual inputs require image features that capture the physical properties of the scene, e.g., the position and configuration of objects. Recently, it has been suggested to learn such features in an unsupervised manner from simulated, self-supervised, robot interaction; the idea being that high-level physical properties are well captured by modern physical simulators, and their representation from visual inputs may transfer well to the real world. In particular, learning methods based on noise contrastive estimation have shown promising results. To robustify the simulation-to-real transfer, domain randomization (DR) was suggested for learning features that are invariant to irrelevant visual properties such as textures or lighting. In this work, however, we show that a naive application of DR to unsupervised learning based on contrastive estimation does not promote invariance, as the loss function maximizes mutual information between the features and both the relevant and irrelevant visual properties. We propose a simple modification of the contrastive loss to fix this, exploiting the fact that we can control the simulated randomization of visual properties. Our approach learns physical features that are significantly more robust to visual domain variation, as we demonstrate using both rigid and non-rigid objects.

Object rearrangement is a widely-applicable and challenging task for robots. Geometric constraints must be carefully examined to avoid collisions and combinatorial issues arise as the number of objects increases. This work studies the algorithmic structure of rearranging uniform objects, where robot-object collisions do not occur but object-object collisions have to be avoided. The objective is minimizing the number of object transfers under the assumption that the robot can manipulate one object at a time. An efficiently computable decomposition of the configuration space is used to create a "region graph", which classifies all continuous paths of equivalent collision possibilities. Based on this compact but rich representation, a complete dynamic programming primitive DFSDP performs a recursive depth first search to solve monotone problems quickly, i.e., those instances that do not require objects to be moved first to an intermediate buffer. DFSDP is extended to solve single-buffer, non-monotone instances, given a choice of an object and a buffer. This work utilizes these primitives as local planners in an informed search framework for more general, non-monotone instances. The search utilizes partial solutions from the primitives to identify the most promising choice of objects and buffers. Experiments demonstrate that the proposed solution returns near-optimal paths with higher success rate, even for challenging non-monotone instances, than other leading alternatives.