ICRA 2021 Papers

The design of aerial-aquatic multirotors can benefit from thruster rotation so that the thrusters can act directly in the lateral directions of surge and sway when submerged. This allows much more effective locomotion underwater as opposed to the aerial configuration where rotational acceleration is used to direct small components of thrust in the lateral directions. However, the introduction of lateral thruster components by rotating the thrusters about their respective arm axes creates additional coupled moment terms. Here, the dynamics of this design is analysed to show that critical angles of thruster rotation can achieve stable surge and sway movements while also decoupling lateral from rotational movements.

This work investigates uncertainty-aware deep learning (DL) in tactile robotics based on a general framework introduced recently for robot vision. For a test scenario, we consider optical tactile sensing in combination with DL to estimate the edge pose as a feedback signal to servo around various 2D test objects. We demonstrate that uncertainty-aware DL can improve the pose estimation over deterministic DL methods. The system estimates the uncertainty associated with each prediction, which is used along with temporal coherency to improve the predictions via a Kalman filter, and hence improve the tactile servo control. The robot is able to robustly follow all of the presented contour shapes to reduce not only the error by a factor of two but also smooth the trajectory from the undesired noisy behaviour caused by previous deterministic networks. In our view, as the field of tactile robotics matures in its use of DL, the estimation of uncertainty will become a key component in the control of physically interactive tasks in complex environments.

Food packing industry workers typically pick a target amount of food by hand from a food tray and place them in containers. Since menus are diverse and change frequently, robots must adapt and learn to handle new foods in a short time-span. Learning to grasp a specific amount of granular food requires a large training dataset, which is challenging to collect reasonably quickly. In this study, we propose ways to reduce the necessary amount of training data by augmenting a deep neural network with models that estimate its uncertainty through self-supervised learning. To further reduce human effort, we devise a data collection system that automatically generates labels. We build on the idea that we can grasp sufficiently well if there is at least one low-uncertainty (high-confidence) grasp point among the various grasp point candidates. We evaluate the methods we propose in this work on a variety of granular foods-coffee beans, rice, oatmeal and peanuts, each of which has a different size, shape and material properties such as volumetric mass density or friction. For these foods, we show significantly improved grasp accuracy of user-specified target masses using smaller datasets by incorporating uncertainty.

For a companion robot that follows a person as an assistant, predicting human walking is important to produce a proactive movement that is helpful to maintain an appropriate area decided by the human personal space. However, fully trusting the prediction may result in obstructing human walking because it is not always accurate. Hence, we consider the estimation of uncertainty (i.e., entropy) of the prediction to enable the robot to move without causing overconfident motion and without being late for the person it follows. To consider this uncertainty of the prediction to the controller, we introduce a reliability value that changes based on the entropy of the prediction. This value expresses the extent the controller should trust the prediction result, and it affects the cost function of our controller. We propose an uncertainty-aware robot controller based on nonlinear model predictive control to realize natural human-followings. We found that our uncertainty-aware control system can produce an appropriate robot movement, such as not obstructing the human walking and avoiding delay, in both simulations using actual human walking data and real-robot experiments.

Curb detection is an essential function of autonomous vehicles in urban areas. However, curbs are difficult to detect in complex urban environments in which many dynamic objects exist. Additionally, curbs appear in a variety of shapes and sizes. Previous studies have been based on the traditional pipeline, which consists of the extraction and aggregation of hand-crafted features that are then fed to classifiers. However, this sequential process is inefficient and designing the hand-crafted features is a complex process. Recently, this kind of process has been replaced by Deep Neural Networks (DNN), in which classifiers and features are learned from large-scale data. Very few works have exploited DNN for the curb detection problem. Most works use multi-modal sensor-based methods that combine images and accumulated 3D point clouds from LIDAR. However, these approaches require synchronization and calibration between sensors. In addition, they do not quantify the uncertainty of their predictions for autonomous system safety. In this paper, we present a two-stage DNN-based curb detection method that includes uncertainty quantification. An autoencoder-based network predicts the curbs, and then conditional neural processes rectify the predictions with uncertainty estimations. The experimental results show that our approach achieves high accuracy and recall in complex areas. We also constructed a large-scale dataset to create benchmarks consisting of approximately 5,224 scans with bird’s-eye view labels collected from urban areas. To the best of our knowledge, there are no public datasets for DNN-based curb detectors. The benchmarks and datasets are publicly available at https://github.com/YounghwaJung/curb_detection_DNN.

We propose ultrasound Doppler imaging and magnetic navigation of collective cell microrobots in whole blood. Cell microrobots are cultured using stem cells and iron microparticles, they have spheroidal structures and can be actuated under external magnetic fields. A collective of cell microrobots can be reversibly gathered and spread due to the tunable magnetic interaction, and are able to exhibit collective motion in whole blood under rotating magnetic fields. Simulation results indicate that the induced blood flow around the collective pattern affects the motion of red blood cells (RBCs), and experimental results show that Doppler signals are observed when emitting ultrasound waves to the microrobots. The induced Doppler signals are affected by the input field frequency and the ultrasound parameters (pulse repetition frequency). Due to the induced three-dimensional blood flow, Doppler signals can be observed when the imaging plane is above the collective microrobots, which enables indirect localization when performing navigation on an uneven surface. Our study investigates a strategy for pattern formation and navigation of collective microrobots under ultrasound Doppler imaging, demonstrating that the integration of collective control approach and medical imaging holds great potential for real-time active delivery tasks.

The problem addressed in this paper is the localisation of a mobile robot using a combination of on-board sensors and Ultra-Wideband (UWB) beacons. Specifically, we consider a scenario in which a mobile robot travels across an area infrastructured with a small number of UWB anchors. The presence of obstacles in the environment introduces an offset in the measurements of the distance between the robot and the UWB anchors causing a degradation in the localisation performance. By using a discrete–time formulation of the system dynamics, we show that, under mild condition, the trajectories can be observed and the offset can be estimated in a finite number of steps. Besides being interesting in its on right, the global observability results offer a clear pathway towards the definition of a new generation of estimation algorithms.

Indoor positioning without GPS is a challenge task, especially, in complex scenes or when sensors fail. In this paper, we develop an ultra-wideband aided visual-inertial positioning system (UVIP) which aims to achieve accurate and robust positioning results in complex indoor environments. To this end, a point-line-based stereo visual-inertial odometry (PL-sVIO) is firstly designed to improve the positioning accuracy in structured or low-textured scenarios by making use of line features. Secondly, a loop closure method is proposed to suppress the drift of PL-sVIO based on image patch features described by a CNN for handing the situation of a large environment and viewpoint variation. Thirdly, an accurate relocalization approach is presented for the case when the visual sensor fails. In this scheme, a top-to-down matching strategy from image to point and line features is presented to improve relocalization performance. Finally, the UWB sensor is combined with the visual-inertial system to further improve the accuracy and robustness of the positioning system and provide the results in a fixed reference frame. Thus, desirable real-time positioning results are derived for complex indoor scenes. Evaluations on challenging public datasets and real-world experiments are conducted to demonstrate that the proposed UVIP can provide more accurate and robust positioning results in complex indoor environments, even in the case when the visual sensor fails or in the absence of UWB anchors.

We present an empirical investigation of a new mapping system based on a graph of panoramic depth images. Panoramic images efficiently capture range measurements taken by a spinning lidar sensor, recording fine detail on the order of a few centimeters within maps of expansive scope on the order of tens of millions of cubic meters. The flexibility of the system is demonstrated by running the same mapping software against data collected by hand-carrying a paired lidar and IMU around a laboratory space at walking pace, moving them outdoors through a campus environment at running pace, driving the sensors on a small wheeled vehicle on- and off-road, flying the sensors through a forest, carrying them on the back of a legged robot navigating an underground coal mine, and mounting them on the roof of a car driven on public roads. The full 3D maps are built online with a median update time of less than ten milliseconds on an embedded NVIDIA Jetson AGX Xavier system.

Low-light image enhancement is a complex and vital task including, recovering color and texture details from low-light images. For automated driving, low-light scenarios will have severe implications for vision-based applications. To address this problem, we propose a real-time unsupervised generative adversarial network (GAN) with multiple discriminators. It includes a multi-scale discriminator, a texture discriminator, and a color discriminator to evaluate images from different perspectives. Furthermore, considering the uneven illumination distribution of images and the different information contained in the channels, we adopte a feature fusion attention module to combine channel attention with pixel attention to extract image features. Experiments show that our method outperforms state-of-the-art methods in qualitative and quantitative evaluation and provides visible improvements in SLAM localization effects.

While many advancements have been made in the development of template models for describing upright-trunk locomotion, the majority of the effort has been focused on the stance phase. In this paper, we develop a new compact dynamic model as a first step toward a fully unified locomotion template model (ULT-model) of an upright-trunk forward hopping system, which will also require a unified control law in the next step. We demonstrate that all locomotion subfunctions are enabled by adding just a point foot mass and a parallel leg actuator to the well-known trunk SLIP model and that a stable limit cycle can be achieved. This brings us closer toward the ultimate goal of enabling closed-loop dynamics for anchor matching and thus achieving simple, efficient, robust and stable upright-trunk gait control, as observed in biological systems.

This paper presents a 3D pointing interface application to signal a UAV’s target in a large-scale environment. This system enables UAVs equipped with a monocular camera to determine which window of a building is selected by a human user in large-scale indoor or outdoor environments. The 3D pointing interface consists of three parts: YOLO, Open- Pose, and ORB-SLAM. YOLO detects the target objects, e.g., windows, OpenPose extracts the user pose, and ORB-SLAM builds a scale-dependent 3D map, a set of 3D sparse feature points. To obtain the visual scale, it performs a calibration step with the user standing in front of the UAV at a certain distance. We detail how we chose the gesture, localize and detect objects, and transform between coordinate systems. The real- world experiment results showed that the 3D pointing interface obtained a 0.73 F1-score average and a 0.58 F1-Score at the maximum distance of 25 meters between UAV and building.

Learning manipulation skills from observing human demonstration videos is a promising aspect for intelligent robotic systems. Recent advances in video to command provide an end-to-end approach to translate a video into robot plans. However, the general video captioning methods focus more on the understanding of the full frame, while they lack the consideration of the spatio-temporal features in videos. In this paper, we proposed the two-stream 2D/3D residual networks for robots to learn manipulation tasks from human demonstration videos. We integrate spatial features with 2D residual network and temporal features with 3D residual network as inputs for RNN layers. An encoder-decoder architecture is then used to encode the spatio-temporal features and sequentially generate the command words. Experimental results on an extended manipulation dataset show that our approach outperforms the state-of-the-art methods. Real-world experiments results on a Baxter robotic arm indicate that our method could produce more accurate commands from video demonstrations.

Underactuated robots often require involved routines for trajectory planning due to their complex dynamics. Flapping-wing aerial vehicles have unsteady aerodynamics and periodic gaits that complicate the planning procedure. In this paper, we improve upon existing methods for flight planning by introducing a two-stage optimization routine to plan flapping flight trajectories. The first stage solves a trajectory optimization problem with a data-driven fixed-wing approximation model trained with experimental flight data. The solution to this is used as the initial guess for a second stage optimization using a flapping-wing model trained with the same flight data. We demonstrate the effectiveness of this approach with a bat robot in both simulation and experimental flight results. The speed of convergence, the dependency on the initial guess, and the quality of the solution are improved, and the robot is able to track the optimized trajectory of a dive maneuver.

To effectively assist human workers in assembly tasks a robot must proactively offer support by inferring their preferences in sequencing the task actions. Previous work has focused on learning the dominant preferences of human workers for simple tasks largely based on their intended goal. However, people may have preferences at different resolutions: they may share the same high-level preference for the order of the sub-tasks but differ in the sequence of individual actions. We propose a two-stage approach for learning and inferring the preferences of human operators based on the sequence of sub-tasks and actions. We conduct an IKEA assembly study and demonstrate how our approach is able to learn the dominant preferences in a complex task. We show that our approach improves the prediction of human actions through cross-validation. Lastly we show that our two-stage approach improves the efficiency of task execution in an online experiment and demonstrate its applicability in a real-world robot-assisted IKEA assembly.

We propose a tree search-based planning algorithm for a robot manipulator to rearrange objects and grasp a target in a dense space. We consider environments where tasks cannot be completed with prehensile planning only. As assuming that a manipulator is only allowed to grasp from the top, we aim to minimize the number of rearrangement actions and the total execution time, which affects the efficiency of manipulation. The proposed search algorithm determines the optimal sequence of object rearrangement with prehensile and non-prehensile grasping until grasping a target. For non-prehensile grasping, a heuristic function is employed to model frictions and contacts between objects and a table. Experimental results in a realistic simulated environment show that the proposed algorithm can reduce the number of rearranged obstacles up to 27% and the total execution time up to 15% with 14 objects compared to the previous work.

In order to achieve autonomous vertical wall climbing, the transition phase from the ground to the wall requires extra consideration inevitably. This paper focuses on the contact sequence planner to transition between flat terrain and vertical surfaces for multi-limbed climbing robots. To overcome the transition phase, it requires planning both multicontact and contact wrenches simultaneously which makes it difficult. Instead of using a predetermined contact sequence, we consider various motions on different environment setups via modeling contact constraints and limb switchability as complementarity conditions. Two safety factors for toe sliding and motor over-torque are the main tuning parameters for different contact sequences. By solving as a nonlinear program (NLP), we can generate several feasible sequences of foot placements and contact forces to avoid failure cases. We verified feasibility with demonstrations on the hardware SiLVIA, a sixlegged robot capable of vertically climbing between two walls by bracing itself in-between using only friction.

This paper presents a novel trajectory optimization formulation to solve the robotic assembly of the belt drive unit. Robotic manipulations involving contacts and deformable objects are challenging in both dynamic modeling and trajectory planning. For modeling, variations in the belt tension and contact forces between the belt and the pulley could dramatically change the system dynamics. For trajectory planning, it is computationally expensive to plan trajectories for such hybrid dynamical systems as it usually requires planning for discrete modes separately. In this work, we formulate the belt drive unit assembly task as a trajectory optimization problem with complementarity constraints to avoid explicitly imposing contact mode sequences. The problem is solved as a mathematical program with complementarity constraints (MPCC) to obtain feasible and efficient assembly trajectories. We validate the proposed method both in simulations with a physics engine and in real-world experiments with a robotic manipulator.

This paper presents a two-stage method to perform trajectory optimisation in multimodal dynamical systems with unknown nonlinear stochastic transition dynamics. The method finds trajectories that remain in a preferred dynamics mode where possible and in regions of the transition dynamics model that have been observed and can be predicted confidently. The first stage leverages a Mixture of Gaussian Process Experts method to learn a predictive dynamics model from historical data. Importantly, this model learns a gating function that indicates the probability of being in a particular dynamics mode at a given state location. This gating function acts as a coordinate map for a latent Riemannian manifold on which shortest trajectories are solutions to our trajectory optimisation problem. Based on this intuition, the second stage formulates a geometric cost function, which it then implicitly minimises by projecting the trajectory optimisation onto the second-order geodesic ODE; a classic result of Riemannian geometry. A set of collocation constraints are derived that ensure trajectories are solutions to this ODE, implicitly solving the trajectory optimisation problem.