ICRA 2022 Papers

Vision-based 3D human pose estimation and shape reconstruction play important roles in robot-assisted healthcare monitoring and personal assistance. However, 3D data captured from a single viewpoint always encounter occlusions and exhibit substantial heterogeneity across different views, resulting in significant challenges for both tasks. Extensive approaches have been proposed to perform each task separately, but few of them present a unified solution. In this paper, we propose a novel network based on signed distance functions, namely PoseSDF, to simultaneously reconstruct 3D lower limb shape and estimate gait pose by two dedicated branches. To promote multi-task learning, several strategies are developed to ensure that these two branches leverage the same latent shape code while exchanging information between them. More importantly, an auxiliary RotNet is incorporated into the inference phase, overcoming the inherent limitations of implicit neural functions under cross-view scenarios. Experimental results demonstrate that our proposed PoseSDF can achieve both high-quality shape reconstruction and precise pose estimation, generalizing well on the data from novel views, gait patterns, as well as real-world.

Fast and accurate pose estimation is essential for many robotic applications such as SLAM, manipulation, and 3D point registration. Existing solutions to this problem suffer from either high computation overhead due to the nonlinear features or accuracy loss due to linear approximation. In this paper, we propose a dual quaternion feedback particle filter (DQFPF) that can capture the nonlinear factors in the observation model and use the optimal control theory to estimate the pose. To avoid particle degeneracy caused by sequential importance sampling and resampling, we present a feedback particle update formula to speed up the optimization with fewer particles being sampled. Simulation results show that in known corresponding cases our approach can converge to the correct pose more efficiently than the state-of-the-art. A similar conclusion can also be drawn in real applications of unknown corresponding cases, i.e., point cloud stitching and visual odometry estimation.

We present a design, model, and control for a novel jumping-flying robot that is called PogoDrone. The robot is composed of a quadrotor with a passive mechanism for jumping. The robot can continuously jump in place or fly like a normal quadrotor. Jumping in place allows the robot to quickly move and operate very close to the ground. For instance, in agricultural applications, the jumping mechanism allows the robot to take samples of soil. We propose a hybrid controller that switches from attitude to position control to allow the robot to fall horizontally and recover to the original position. We compare the jumping mode with the hovering mode to analyze the energy consumption. In simulations, we evaluate the effect of different factors on energy consumption. In real experiments, we show that our robot can repeatedly impact the ground, jump, and fly in a physical environment.

Planning problems can be solved not only by planners, but also by model checkers. While the former yield a plan that requires replanning as soon as any fault occurs, the latter provide a “universal” plan (a.k.a. strategy, policy, or controller) able to make decisions under all circumstances. One of the prohibitive aspects of the latter approach is stemming from this very advantage: since it is defined for all possible states of the system, it is typically so large that it does not fit into small memories of embedded devices. As another consequence of the size, its execution may be slow. In this paper, we provide a solution to this issue by linking the model checkers with decision-tree learners, resulting in decision-tree representations of the synthesized strategies. Not only are they dramatically smaller, but also more explainable and orders-of-magnitude faster to execute than plans with replanning. In addition, we describe a method for model validation and debugging via the model checker and the decision-tree learner in the loop. We illustrate the approach on our case study of a robotic arm for picking items in a real industrial setting.

In this paper, we propose a new framework for solving cable-routing problems with a dual-arm robot, where the objective is to clip a Deformable Linear Object (DLO) into several arbitrarily placed fixtures. The core of the framework is a task-space planner, which builds a roadmap from predefined tasks and employs a replanning strategy based on a genetic algorithm, if problems occur. The manipulation tasks are executed with either individual or coordinated control of the arms. Moreover, hierarchical quadratic programming is used to solve the inverse differential kinematics together with extra feasibility objectives. A vision system first identifies the desired fixture route and structure preserved registration estimates the state of the DLO in real-time. The framework is tested on real-world experiments with a YuMi robot, demonstrating a 90% success rate for 3 fixture problems.

Embedding elastic elements into legged robots through mechanical design enables highly efficient oscillating patterns that resemble natural gaits. However, current trajectory planning techniques miss the opportunity of taking advantage of these natural motions. This work proposes a locomotion planning method that aims to unify traditional trajectory generation with modal oscillations. Our method utilizes task-space linearized modes for generating center of mass trajectories on the sagittal plane. We then use nonlinear optimization to find the gait timings that match these trajectories within the Divergent Component of Motion planning framework. This way, we can robustly translate the modes-aware centroidal motions into joint coordinates. We validate our approach with promising results and insights through experiments on a compliant quadrupedal robot.

Accurate camera pose estimation is a fundamental requirement for numerous applications, such as autonomous driving, mobile robotics, and augmented reality. In this work, we address the problem of estimating the global 6 DoF camera pose from a single RGB image in a given environment. Previous works consider every part of the image valuable for localization. However, many image regions such as the sky, occlusions, and repetitive non-distinguishable patterns cannot be utilized for localization. In addition to adding unnecessary computation efforts, extracting and matching features from such regions pro-duce many wrong matches which in turn degrades the localization accuracy and efficiency. Our work addresses this particular issue and shows by exploiting an interesting concept of sparse 3D models that we can exploit discriminatory environment parts and avoid useless image regions for the sake of a single image localization. Interestingly, through avoiding selecting keypoints from non-reliable image regions such as trees, bushes, cars, pedestrians, and occlusions, our work acts naturally as an outlier filter. This makes our system highly efficient in that minimal set of correspondences is needed and highly accurate as the number of outliers is low. Our work exceeds state-of-the-art methods on outdoor Cambridge Landmarks dataset. With only relying on single image at inference, it outweighs in terms of accuracy methods that exploit pose priors and/or reference 3D models while being much faster. By choosing as little as 100 correspondences, it surpasses similar methods that localize from thousands of correspondences, while being more efficient. In particular, it achieves, compared to these methods, an improvement of localization by 33% on OldHospital scene. Furthermore, It outstands direct pose regressors even those that learn from sequence of images. Our work will be publicly available.

Dexterous robotic cutting needs to demonstrate a skill level with smooth and efficient knife movements. The work performed by the knife mainly generates fracture and overcomes the blade-material friction. This paper presents a recursive least-squares method that repeatedly estimates relevant physical parameters such as Poisson's ratio, fracture toughness, and coefficient of friction, all varying with the knife's movement when cutting a natural food, from force sensor readings. Furthermore, we show that these estimates can be used for generating the knife's trajectory on the fly to either maximize the ease of fracturing or to minimize the rate of work.

Driving automation is gradually replacing human driving maneuvers in different applications such as adaptive cruise control and lane keeping. However, contemporary driving automation applications based on expert systems or prede-fined control strategies are not in line with individual human driver's preference. To overcome this problem, we propose a Personalized Adaptive Cruise Control (P-ACC) system that can learn the driver's car-following preferences from historical data using model-based maximum entropy Inverse Reinforcement Learning (IRL). Once activated in real-time, the P-ACC system first classifies the driver type and the weather type (at that moment). The vehicle is then controlled using the pre-trained IRL model on the cloud of the associated class. The personalized IRL model on the cloud will be updated as more human driving data is collected from various scenarios. Numerical simulation with real-world naturalistic driving data shows that, the accuracy of reproducing the real-world driving profile improves up to 30.1% in terms of speed and 36.5% in terms of distance gap, when P-ACC is compared with the Intelligent Driver Model (IDM). Game engine-based human-in-the-loop simulation demonstrates that, the takeover frequency of the driver during the usage of P-ACC decreases up to 93.4%, compared with that during the usage of IDM-based ACC.

This work explores the use of topological tools for achieving effective non-prehensile manipulation in cluttered, constrained workspaces. In particular, it proposes the use of persistent homology as a guiding principle in identifying the appropriate non-prehensile actions, such as pushing, to clean a cluttered space with a robotic arm so as to allow the retrieval of a target object. Persistent homology enables the automatic identification of connected components of blocking objects in the space without the need for manual input or tuning of parameters. The proposed algorithm uses this information to push groups of cylindrical objects together and aims to minimize the number of pushing actions needed to reach to the target. Simulated experiments in a physics engine using a model of the Baxter robot show that the proposed topology-driven solution is achieving significantly higher success rate in solving such constrained problems relatively to state-of-the-art alternatives from the literature. It manages to keep the number of pushing actions low, is computationally efficient and the resulting decisions and motion appear natural for effectively solving such tasks.

Methods for state estimation that rely on visual information are challenging on legged robots due to rapid changes in the viewing angle of onboard cameras. In this work, we show that by leveraging structure in the way that the robot locomotes, the accuracy of visual-inertial SLAM in these challenging scenarios can be increased. We present a method that takes advantage of the underlying periodic predictability often present in the motion of legged robots to improve the performance of the feature tracking module within a visual-inertial SLAM system. Our method performs multi-session SLAM on a single robot, where each session is responsible for mapping during a distinct portion of the robot's gait cycle. Our method produces lower absolute trajectory error than several state-of-the-art methods for visual-inertial SLAM in both a simulated environment and on data collected on a quadrupedal robot executing dynamic gaits. On real-world bounding gaits, our median trajectory error was less than 35% of the error of the next best estimate provided by state-of-the-art methods.

In this work we present the first initialization methods equipped with explicit performance guarantees that are adapted to the pose-graph simultaneous localization and mapping (SLAM) and rotation averaging (RA) problems. SLAM and rotation averaging are typically formalized as large-scale nonconvex point estimation problems, with many bad local minima that can entrap the smooth optimization methods typically applied to solve them; the performance of standard SLAM and RA algorithms thus crucially depends upon the quality of the estimates used to initialize this local search. While many initialization methods for SLAM and RA have appeared in the literature, these are typically obtained as purely heuristic approximations, making it difficult to determine whether (or under what circumstances) these techniques can be reliably deployed. In contrast, in this work we study the problem of initialization through the lens of spectral relaxation. Specifically, we derive a simple spectral relaxation of SLAM and RA, the form of which enables us to exploit classical linear-algebraic techniques (eigenvector perturbation bounds) to control the distance from our spectral estimate to both the (unknown) ground-truth and the global minimizer of the estimation prob lem as a function of measurement noise. Our results reveal the critical role that spectral graph-theoretic properties of the measurement network play in controlling estimation accuracy; moreover, as a by-product of our analysis we obtain new bounds on the estimation error for the maximum likelihood estimators in SLAM and RA, which are likely to be of independent interest. Finally, we show experimentally that our spectral estimator is very effective in practice, producing initializations of comparable or superior quality at lower computational cost compared to existing state-of-the-art techniques.

Visual perception of an autonomous vehicle is a crucial component of autonomous driving technologies. While visual perception research has achieved promising performance in recent years, modern methods are mostly trained, applied, and tested on single clean images. Recently, deep learning-based perception methods have addressed multiple degrading effects to reflect real-world bad weather cases, but have achieved only limited success, mainly due to 1) less or no temporal information across adjacent frames and 2) poor correlation between image enhancement and visual perception performance. To solve these issues, in this paper we propose a simple and effective video enhancement network incorporating the perception module, one of the high-level vision tasks, which takes video degraded by adverse weather conditions as an input, and produces an enhanced image and a recognition result as output. Besides, this allows us to leverage temporal information across consecutive frames without flow estimation. We also introduce a new training strategy to robustly guide the high-level task model suitable for both high-quality restoration of images and highly accurate perception. Further, considering sequence-level discrimination, we propose a Sequential Contrastive Loss, called SCL, to maximize apparent discrimination over sequential input. By doing this, our algorithm achieves full interaction showing mutual influence between the enhancement and perception tasks. Further, we introduce a novel low memory network dropping out most of the layer connections of dense blocks to reduce memory usage and computational cost while maintaining high performance. Experiment results demonstrate that the proposed method significantly improves the performance on object detection, distance estimation, and memory usage under adverse weather.

For achieving significant levels of autonomy, legged robot behaviors require perceptual awareness of both the terrain for traversal, as well as structures and objects in their surroundings for planning, obstacle avoidance, and high-level decision making. In this work, we present a perception engine for legged robots that extracts the necessary information for developing semantic, contextual, and metric awareness of their surroundings. Our custom sensor configuration consists of (1) an active depth sensor, (2) two monocular cameras looking sideways, (3) a passive stereo sensor observing the terrain, (4) a forward facing active depth camera, and (5) a rotating 3D LIDAR with a large vertical field-of-view (FOV). The mutual overlap in the sensors' FOVs allows us to redundantly detect and track objects of both dynamic and static types. We fuse class masks generated by a semantic segmentation model with LIDAR and depth data to accurately identify and track individual instances of dynamically moving objects. In parallel, active depth and passive stereo streams of the terrain are also fused to map the terrain using the on-board GPU. We evaluate the engine using two different humanoid behaviors, (1) look-and-step and (2) track-and-follow, on the Boston Dynamics Atlas.

Forecasting pedestrians' future motions is essential for autonomous driving systems to safely navigate in urban areas. However, existing prediction algorithms often overly rely on past observed trajectories and tend to fail around abrupt dynamic changes, such as when pedestrians suddenly start or stop walking. We suggest that predicting these highly non-linear transitions should form a core component to improve the robustness of motion prediction algorithms. In this paper, we introduce the new task of pedestrian stop and go forecasting. Considering the lack of suitable existing datasets for it, we release TRANS, a benchmark for explicitly studying the stop and go behaviors of pedestrians in urban traffic. We build it from several existing datasets annotated with pedestrians' walking motions, in order to have various scenarios and behaviors. We also propose a novel hybrid model that leverages pedestrian-specific and scene features from several modalities, both video sequences and high-level attributes, and gradually fuses them to integrate multiple levels of context. We evaluate our model and several baselines on TRANS, and set a new benchmark for the community to work on pedestrian stop and go forecasting.

Robot-aided rehabilitation is pushing forward novel robotic architectures to provide physical therapy. This paper presents a patient-tailored control architecture for upper-limb robot-aided orthopaedic rehabilitation capable of i) adapting the robot workspace on the basis of patient Range of Motion (RoM); ii) generating a tunnel, around the desired path to be followed by the patient, which guarantees spatial autonomy; iii) introducing a back-wall inside the tunnel sliding with variable speed on the basis of patient performance to ensure temporal autonomy; iv) rehabilitating to working gestures, thanks to a DMP-based trajectory planner, with the aim of favoring an effective translation of the patient's motor recovery results to the occupational sphere; v) ensuring a patient-tailored assistance also thanks to the evaluation of performance indicators. The designed system was validated demonstrating the adaptability of the system to orthopaedic patient motor imnrovements.

The success of motion prediction for autonomous driving relies on integration of information from the HD maps. As maps are naturally graph-structured, investigation on graph neural networks (GNNs) for encoding HD maps is burgeoning in recent years. However, unlike many other applications where GNNs have been straightforwardly deployed, HD maps are heterogeneous graphs where vertices (lanes) are connected by edges (lane-lane interaction relationships) of various nature, and most graph-based models are not designed to understand the variety of edge types which provide crucial cues for predicting how the agents would travel the lanes. To overcome this challenge, we propose Path-Aware Graph Attention, a novel attention architecture that infers the attention between two vertices by parsing the sequence of edges forming the paths that connect them. Our analysis illustrates how the proposed attention mechanism can facilitate learning in a didactic problem where existing graph networks like GCN struggle. By improving map encoding, the proposed model surpasses previous state of the art on the Argoverse Motion Forecasting dataset, and won the first place in the 2021 Argoverse Motion Forecasting Competition.

We address the problem of tracking 3D object poses from touch during in-hand manipulations. Specifically, we look at tracking small objects using vision-based tactile sensors that provide high-dimensional tactile image measurements at the point of contact. While prior work has relied on a-priori information about the object being localized, we remove this requirement. Our key insight is that an object is composed of several local surface patches, each informative enough to achieve reliable object tracking. Moreover, we can recover the geometry of this local patch online by extracting local surface normal information embedded in each tactile image. We propose a novel two-stage approach. First, we learn a mapping from tactile images to surface normals using an image translation network. Second, we use these surface normals within a factor graph to both reconstruct a local patch map and use it to infer 3D object poses. We demonstrate reliable object tracking for over 100 contact sequences across unique shapes with four objects in simulation and two objects in the real-world.

Collision risk and smoothness are the most important factors in global path planning. Currently, planning methods that reduce global path collision risk and improve its smoothness through numerical optimization have achieved good results. However, these methods cannot always optimize the path. The reason is all points on the path are considered as decision variables, which leads to the high dimensionality of the defined optimization problem. Therefore, we propose a novel global path optimization method. The method characterizes the path as a parametric curve and then optimizes the curve's parameters with a defined objective function, which successfully reduces the dimension of optimization problem. The proposed method is compared with baseline and state-of-the-art methods. Experimental results show the path optimized by our method is not only optimal in collision risk, but also in efficiency and smoothness. Furthermore, the proposed method is also implemented and tested in both simulation and real robots.