ICRA 2021 Papers

Multi-object tracking (MOT) enables mobile robots to perform well-informed motion planning and navigation by localizing surrounding objects in 3D space and time. Existing methods rely on depth sensors (e.g., LiDAR) to detect and track targets in 3D space, but only up to a limited sensing range due to the sparsity of the signal. On the other hand, cameras provide a dense and rich visual signal that helps to localize even distant objects, but only in the image domain. In this paper, we propose EagerMOT, a simple tracking formulation that eagerly integrates all available object observations from both sensor modalities to obtain a well-informed interpretation of the scene dynamics. Using images, we can identify distant incoming objects, while depth estimates allow for precise trajectory localization as soon as objects are within the depth-sensing range. With EagerMOT, we achieve state-of-the-art results across several MOT tasks on the KITTI and NuScenes datasets. Our code is available at https://github.com/aleksandrkim61/EagerMOT

The quadrotor is popularly used in challenging environments due to its superior agility and flexibility. In these scenarios, trajectory planning plays a vital role in generating safe motions to avoid obstacles while ensuring flight smoothness. Although many works on quadrotor planning have been proposed, a research gap exists in incorporating self-adaptation into a planning framework to enable a drone to automatically fly slower in denser environments and increase its speed in a safer area. In this paper, we propose an environmental adaptive planner to adjust the flight aggressiveness effectively based on the obstacle distribution and quadrotor state. Firstly, we design an environmental adaptive safety aware method to assign the priority of the surrounding obstacles according to the environmental risk level and instantaneous motion tendency. Then, we apply it into a multi-layered model predictive contouring control (Multi-MPCC) framework to generate adaptive, safe, and dynamical feasible local trajectories. Extensive simulations and real-world experiments verify the efficiency and robustness of our planning framework. Benchmark comparison also shows superior performances of our method with another advanced environmental adaptive planning algorithm. Moreover, we release our planning framework as open-source ros-packages1 .

Ego-motion estimation is a key requirement for the simultaneous localization and mapping (SLAM) problem. The traditional pipeline goes through feature extraction, feature matching and pose estimation, whose performance depends on the manually designed features. In this paper, we are motivated by the strong performance of deep learning methods in other computer vision and robotics tasks. We replace hand-crafted features with a neural network and directly estimate the relative pose between two adjacent scans from a LiDAR sensor using ENCODE: a dEep poiNt Cloud ODometry nEtwork. Firstly, a spherical projection of the input point cloud is performed to acquire a multi-channel vertex map. Then a multi-layer network backbone is applied to learn the abstracted features and a fully connected layer is adopted to estimate the 6-DoF ego-motion. Additionally, a map-to-map optimization module is applied to update the local poses and output a smooth map. Experiments on multiple datasets demonstrate that the proposed method achieves the best performance in comparison to state-of-the-art methods and is capable of providing accurate poses with low drift in various kinds of scenarios.

This paper proposes an EMG-dependant neural network-based model of human forearm during interaction with a haptic training simulator of sinus endoscopy. We used a conventional lumped mass-spring-damper model as a base model, beside which we took effects of muscle activation level, using surface electromyography (EMG) signals, into consideration. Unknown parameters of a five-parameter mass-spring-damper model are optimised using experimental force and position data with a Levenberg–Marquardt (LM) algorithm. In the training phase, parallel to this lumped model, a neural network (NN) structure is trained to learn the nonlinear mapping between the EMG signals (a way of measuring the muscles activation level that can be interpreted as muscle stiffness) and the parameters of the lumped model. In prediction (operational) phase, the trained neural network makes an estimate of the lumped parameters, using EMG and position data. Therefore, as apposed to conventional constant-parameter (CP) models, the parameters of the lumped model are not fixed in this method and are dependent to the muscle stiffness. Eight trials were performed while the operator was asked to to hold one’s arm in a vertical plane such that their elbow had a right angle keep exerting a quasi-static and also reciprocating force in one direction–a linear motion coaxial to their forearm. Haptic interface was programmed in a way to mimic the impedance model of sinus tissue, a nonlinear viscoelastic Kelvin-Voigt model previously developed by the authors. The estimated forces and the experimental forces are compared for two scenarios: once for the proposed EMG-dependant NN-based model and once again for the constant-parameter lumped model. Results demonstrate the precision improvement on the estimation of the exerted force from human hand to the haptic interface in the proposed model.

This paper presents a decentralized and asynchronous systematic solution for multi-robot autonomous navigation in unknown obstacle-rich scenes using merely onboard resources. The planning system is formulated under gradient-based local planning framework, where collision avoidance is achieved by formulating the collision risk as a penalty of a nonlinear optimization problem. In order to improve robustness and escape local minima, we incorporate a lightweight topological trajectory generation method. Then agents generate safe, smooth, and dynamically feasible trajectories in only several milliseconds using an unreliable trajectory sharing network. Relative localization drift among agents is corrected by using agent detection in depth images. Our method is demonstrated in both simulation and real-world experiments. The source code is released for the reference of the community.

We present an ensemble learning methodology that combines multiple existing robotic grasp synthesis algorithms and obtain a success rate that is significantly better than the individual algorithms. The methodology treats the grasping algorithms as "experts" providing grasp "opinions". An Ensemble Convolutional Neural Network (ECNN) is trained using a Mixture of Experts (MOE) model that integrates these opinions and determines the final grasping decision. The ECNN introduces minimal computational cost overhead, and the network can virtually run as fast as the slowest expert. We test this architecture using open-source algorithms in the literature by adopting GQCNN 4.0, GGCNN and a custom variation of GGCNN as experts and obtained a 6% increase in the grasp success on the Cornell Dataset compared to the best-performing individual algorithm. The performance of the method is also demonstrated using a Franka Emika Panda arm.

Understanding the gap between simulation and reality is critical for reinforcement learning with legged robots, which are largely trained in simulation. However, recent work has resulted in sometimes conflicting conclusions with regard to which factors are important for success, including the role of dynamics randomization. In this paper, we aim to provide clarity and understanding on the role of dynamics randomization in learning robust locomotion policies for the Laikago quadruped robot. Surprisingly, in contrast to prior work with the same robot model, we find that direct sim-to-real transfer is possible without dynamics randomization or on-robot adaptation schemes. We conduct extensive ablation studies in a sim-to-sim setting to understand the key issues underlying successful policy transfer, including other design decisions that can impact policy robustness. We further ground our conclusions via sim-to-real experiments with various gaits, speeds, and stepping frequencies. Additional Details: pair.toronto.edu/understanding-dr/

As environments involving both robots and humans become increasingly common, so does the need to account for people during planning. To plan effectively, robots must be able to respond to and sometimes influence what humans do. This requires a human model which predicts future human actions. A simple model may assume the human will continue what they did previously; a more complex one might predict that the human will act optimally, disregarding the robot; whereas an even more complex one might capture the robot’s ability to influence the human. These models make different trade-offs between computational time and performance of the resulting robot plan. Using only one model of the human either wastes computational resources or is unable to handle critical situations. In this work, we give the robot access to a suite of human models and enable it to assess the performance-computation trade-off online. By estimating how an alternate model could improve human prediction and how that may translate to performance gain, the robot can dynamically switch human models whenever the additional computation is justified. Our experiments in a driving simulator showcase how the robot can achieve performance comparable to always using the best human model, but with greatly reduced computation.

Autonomous exploration allows mobile robots to navigate in initially unknown territories in order to build complete representations of the environments. In many real-life applications, environments often contain dynamic obstacles which can compromise the exploration process by temporarily blocking passages, narrow paths, exits or entrances to other areas yet to be explored. In this work, we formulate a novel exploration strategy capable of explicitly handling dynamic obstacles, thus leading to complete and reliable exploration outcomes in populated environments. We introduce the concept of dynamic frontiers to represent unknown regions at the boundaries with dynamic obstacles together with a cost function which allows the robot to make informed decisions about when to revisit such frontiers. We evaluate the proposed strategy in challenging simulated environments and show that it outperforms a state-of-the-art baseline in these populated scenarios.

Accurate position feedback in a wide range is critical for medical microrobotics and robot-assisted examinations, such as colonoscopy, bronchoscopy and capsule endoscopy examination. Among the many modalities of positioning feedback, magnetic tracking is a preferable method due to the unique advantages of free line of sight, free energy storage and untethered connection. However, the field strength of the magnetic source decreases with the third power of the distance, limiting the effectiveness of position feedback at long distances. In order to maintain a consistently high tracking accuracy in a broad area, this paper presents a new dynamic tracking solution by applying a movable sensor array. In this new solution, the tracking accuracy of the magnet is first determined and optimized within a short range. When the target microrobot carrying the magnet exceeds this optimized range, the sensor array is relocated by an external robotic arm to keep the target in the effective tracking range. Moreover, we also propose a multi-point locating algorithm to minimize the varying background noise. Experimental results show that the proposed method increases the range of magnetic tracking and achieves a satisfactory level of tracking accuracy, which demonstrates significant potentials to improve the position feedback of microrobots in medical applications.

The autonomous navigation in the crowded environment is a challenging task due to the sensor occlusion and the complex nature of the abstract social interactions. And yet, humans are capable of navigating in such complex environment. In this paper, we propose an effective navigation method that combines the learning-based and model-based methods in a way that a cost function that includes human imitation factor learned via deep learning is integrated into the dynamic window approach (DWA) [1]. The experiments conducted on simulations show that by training the robot to imitate the human trajectory, our navigation method is safer and more efficient than the state-of-the-art methods. Additionally, we successfully deployed a physical robot in an actual environment, and we validate that our navigation quality shares similar tendency with human in the path length, travel time, and the collision avoidance.

In the modern manufacturing process, novel technologies enable the collaboration between humans and robots, which increases productivity while keeping flexibility. However, these technologies also lead to new challenges, e.g., maximization of Human-Robot Collaboration (HRC) performance while ensuring safety for the human being in fenceless robot applications. In this paper, an approach of the dynamic human motion projection is proposed for typical assembly tasks. The human upper body is simplified as a five-degree-of-freedom (5DOF) rigid-body model. A control-oriented projection model is proposed, and its parameters are estimated from the test data of human capability. Combined with a human-state estimator and a collision estimator, the "worst-case" collision motion is projected in the HRC scenario. The dynamic projection method is feasible online. Finally, the estimated collision time is adopted to increase the robot’s speed limit, which validates the improvement of HRC’s efficiency.

Modern computer algorithms easily beat world champions in chess or Go, but state-of-the-art robots are still outperformed by two-year-old’s in manipulating the pieces, let alone interacting with more complex objects. This work studied human behavior when moving an underactuated object, a cup with a ball rolling inside creating internal dynamics like sloshing coffee in a cup. The objective was to develop a control model that could replicate human behavior. Human movement data were collected for transporting this cup-and-ball system, both with and without external perturbations. The existing models in the human control literature, including maximum smoothness, optimal feedback control with minimum effort, and dynamic primitives with impedance were revisited for this challenging task. As these control models were primarily developed for unconstrained reaching movements, they could replicate human trajectories when transporting a rigid object. However, they fell short when the object introduced complex interaction forces due to its internal dynamics. Therefore, this study extended the framework of dynamic primitives and used an optimal controller to generate a maximally smooth zero-force trajectory for the impedance operator when interacting with perturbations from the object or the environment. Given the challenges that robot control still faces when interacting with complex objects, these findings may inform the development of bio-inspired controllers for robotic manipulation.

Modeling and understanding the environment is an essential task for autonomous driving. In addition to the detection of objects, in complex traffic scenarios the motion of other road participants is of special interest. Therefore, we propose to use a recurrent neural network to predict a dynamic occupancy grid map, which divides the vehicle surrounding in cells, each containing the occupancy probability and a velocity estimate. During training, our network is fed with sequences of measurement grid maps, which encode the lidar measurements of a single time step. Due to the combination of convolutional and recurrent layers, our approach is capable to use spatial and temporal information for the robust detection of static and dynamic environment. In order to apply our approach with measurements from a moving ego-vehicle, we propose a method for ego-motion compensation that is applicable in neural network architectures with recurrent layers working on different resolutions. In our evaluations, we compare our approach with a state-of-the-art particle-based algorithm on a large publicly available dataset to demonstrate the improved accuracy of velocity estimates and the more robust separation of the environment in static and dynamic area. Additionally, we show that our proposed method for ego-motion compensation leads to comparable results in scenarios with stationary and with moving ego-vehicle.

Highly dynamic environments, with moving objects such as cars or humans, can pose a performance challenge for LiDAR SLAM systems that assume largely static scenes. To overcome this challenge and support the deployment of robots in real world scenarios, we propose a complete solution for a dynamic object aware LiDAR SLAM algorithm. This is achieved by leveraging a real-time capable neural network that can detect dynamic objects, thus allowing our system to deal with them explicitly. To efficiently generate the necessary training data which is key to our approach, we present a novel end-to-end occupancy grid based pipeline that can automatically label a wide variety of arbitrary dynamic objects. Our solution can thus generalize to different environments without the need for expensive manual labeling and at the same time avoids assumptions about the presence of a predefined set of known objects in the scene. Using this technique, we automatically label over 12000 LiDAR scans collected in an urban environment with a large amount of pedestrians and use this data to train a neural network, achieving an average segmentation IoU of 0.82. We show that explicitly dealing with dynamic objects can improve the LiDAR SLAM odometry performance by 39.6% while yielding maps which better represent the environments. A supplementary video1 as well as our test data2 are available online.

We aim to develop an efficient programming method for equipping service robots with the skill of performing sign language motions. This paper addresses the problem of transferring complex dual-arm sign language motions characterized by the coordination among arms and hands from human to robot, which is seldom considered in previous studies of motion retargeting techniques. In this paper, we propose a novel motion retargeting method that leverages graph optimization and Dynamic Movement Primitives (DMPs) for this problem. We employ DMPs in a leader-follower manner to parameterize the original trajectories while preserving motion rhythm and relative movements between human body parts, and adopt a three-step optimization procedure to find deformed trajectories for robot motion planning while ensuring feasibility for robot execution. Experimental results of several Chinese Sign Language (CSL) motions have been successfully performed on ABB’s YuMi dual-arm collaborative robot (14-DOF) with two 6-DOF Inspire-Robotics’ multi-fingered hands, a system with 26 DOFs in total.

In recent years, research and development have been carried out on manipulators equipped with multi-fingered robot hands as end-effectors to perform delicate and dexterous tasks. In high-speed movement of such multi-fingered hand-arms, the weight of the multi-fingered hands slows down the response of the arms. To solve this problem, we propose a control method in which a high-response hand is used to compensate for the delay in a low-response arm in the task of throwing a ball. In particular, the accuracy of the pitching motion is improved by predicting tracking errors based on the arm dynamics and using nonlinear model predictive control to compensate for arm tracking errors by using the hand motion. Experimental results of ball throwing are shown.

This paper proposes a path correction method for surgical robotic systems performing needle handoff manipulations as part of autonomous execution of surgical suturing. During handoff motions, the position and orientation of the needle is subject to perturbations from the idealized planned pose due to uncertainties in camera-robot calibration and needle localization. If, after a perturbation, the system needs to perform subsequent needle regrasp(s), but the robot still follows the originally planned trajectory out of the path planner [1], it has a lower chance of gripping the needle properly. In order to accommodate these unpredictable needle pose perturbations, the proposed path correction method works locally to direct the needle from the wrong pose to the original pose by partial replanning of the robot motion. The reliability of the proposed method is evaluated with three sets of experiments in a simulation environment.

This paper describes a quasi-passive transradial prosthesis designed to restore drumstick grip control for amputee drummers. A compact motor with low rotor resistance driven by Electromyography (EMG) is implemented in the prosthesis to support real-time drum performances. A variety of grip profiles are achieved by adjusting the stiffness and damping of an impedance controller based on machine learning predictions from EMG data. In addition to the prosthetic design, the paper presents a dynamic model that simulates the drumstick trajectories and estimates the control gains for various natural bouncing patterns. We evaluate the effectiveness of the design through human experiments under real-time performing scenarios for two common drumming grip techniques. The results demonstrate that the prosthetic model can support similar bouncing patterns to the drummer’s healthy hand for single, double, and triple-stroke rolling techniques. The results also show that the system supports usable real-time periodic grip change from single to double-stroke, successfully simulating the common paradiddle drumming technique.