IROS 2022 Papers

2D SLAM is useful for mobile robots that are constrained to a 2D plane, for example in a warehouse, simplifying calculations in respect to the 3D case. The use of an IMU in such a context can enrich the estimation and make it more robust. In this paper we reformulate the IMU preintegration widely used in 3D problems for the 2D case, making use of Lie Theory. The Lie theory based formalization, first derived for a perfectly horizontal plane, allows us to easily extend it to problems where the plane is not orthogonal to the gravity vector. We implement the theory in a factor graph based estimation library, and carry out experiments to validate it on a mobile platform. Two experiments are carried out; on a horizontal and a sloped environment, and the sensor data is processed using our two 2D methods and a state-of-the-art 3D method.

In complex urban environments, the Inertial Navigation System (INS) is important for navigating unmanned ground vehicles (UAVs) for its environment-independency and reliability of real-time localization. It is usually employed as the baseline in the case of other sensors failures, such as the GPS, Lidar, or Cameras. However, one problem for the INS is that its estimation error of localization accumulates over time, and thus the estimated trajectories of the UAVs continue to drift away from their ground truths. To solve this problem, this paper proposes an improved algorithm based on the Invariant Extended Kalman Filter (IEKF) for dead-reckoning of autonomous vehicles, which dynamically adjusts the process noise and the observation noise covariance matrixes through Attention mechanism and Recurrent Neural Network (RNN). The algorithm achieves more robust and accurate dead-reckoning localization in the experiments conducted on the KITTI dataset, reducing the translational error by about 45%compared to the baseline.

Robots that are supposed to interact with or manipulate objects in the world must be able to track the poses of objects in their sensor data. Thus, Detecting and tracking the 6-DoF poses of targeted objects is important for aerial manipulation and is still in the early stage due to the high dynamics and limited onboard capacity of such systems. In this paper, we propose ICK-Track, a novel method for onboard category-level object 6-DoF pose tracking that can be applied to aerial manipulation without using any pre-defined object CAD models. It first utilizes a semi-supervised video segmentation to detect objects in the eye-in-hand RGB-D camera stream to segment the 3D points of objects. Then, canonical keypoints are extracted using iterative farthest point sampling. We propose a novel inter-frame consistent keypoints generation network to generate the corresponding keypoint pairs, which are used together with ICP to estimate the pose changes of objects for tracking. Experimental results show that our method is more robust to viewpoint changes and runs faster than the state-of-the-art methods on category-level pose tracking. We further test our proposed method on a real aerial manipulator. A demo video showing the use of our method on a real aerial manipulator and the implementation of our method are available at: https://github.com/S-JingTao/ICK-Track.

Scanning real-life scenes with modern registration devices typically give incomplete point cloud representations, mostly due to the limitations of the scanning process and 3D occlusions. Therefore, completing such partial representations remains a fundamental challenge of many computer vision applications. Most of the existing approaches aim to solve this problem by learning to reconstruct individual 3D objects in a synthetic setup of an uncluttered environment, which is far from a real-life scenario. In this work, we reformulate the problem of point cloud completion into an objects hallucination task. Thus, we introduce a novel autoencoder-based architecture called HyperPocket that disentangles latent representations and, as a result, enables the generation of multiple variants of the completed 3D point clouds. Furthermore, we split point cloud processing into two disjoint data streams and leverage a hypernetwork paradigm to fill the spaces, dubbed pockets, that are left by the missing object parts. As a result, the generated point clouds are smooth, plausible, and geometrically consistent with the scene. Moreover, our method offers competitive performances to the other state-of-the-art models, enabling a plethora of novel applications.

Control of wheeled humanoid locomotion is a challenging problem due to the nonlinear dynamics and under-actuated characteristics of these robots. Traditionally, feedback controllers have been utilized for stabilization and locomotion. However, these methods are often limited by the fidelity of the underlying model used, choice of controller, and environmental variables considered (surface type, ground inclination, etc). Recent advances in reinforcement learning (RL) offer promising methods to tackle some of these conventional feedback controller issues, but require large amounts of interaction data to learn. Here, we propose a hybrid learning and model-based controller Hybrid LMC that combines the strengths of a classical linear quadratic regulator (LQR) and ensemble deep reinforcement learning. Ensemble deep reinforcement learning is composed of multiple Soft Actor-Critic (SAC) and is utilized in reducing the variance of RL networks. By using a feedback controller in tandem the network exhibits stable performance in the early stages of training. As a preliminary step, we explore the viability of Hybrid LMC in controlling wheeled locomotion of a humanoid robot over a set of different physical parameters in MuJoCo simulator. Our results show that Hybrid LMC achieves better performance compared to other existing techniques and has increased sample efficiency.

Lattice structures allow robotic systems to operate in complex and hazardous environments, e.g. construction, mining and nuclear plants, reliably and effectively. However, current navigation systems for these structures are neither realistic, as they assume simplistic motion primitives and obstacle-free workspaces, nor efficient as they rely solely on global discrete search in an attempt to leverage the modularity of lattices. This paper tackles this gap and studies how robots can navigate lattice structures efficiently. We present a realistic application environment where robots have to avoid obstacles and the structure itself to reach target locations. Our solution couples discrete optimal search, using a domain-dependent heuristic, and sampling-based motion planning to find feasible trajectories in the discrete search space and in the continuous joint space at the same time. We provide two search graph formulations and a path planning approach. Simulation experiments, based on structures and robots created for the Innovate UK Connect-R project, examine scalability to large grid spaces while maintaining performances close to optimal.

Semantic simultaneous localization and mapping is a subject of increasing interest in robotics and AI that directly influences the autonomous vehicles industry, the army industries, and more. One of the challenges in this field is to obtain object classification jointly with robot trajectory estimation. Considering view-dependent semantic measurements, there is a coupling between different classes, resulting in a combinatorial number of hypotheses. A common solution is to prune hypotheses that have a sufficiently low probability and to retain only a limited number of hypotheses. However, after pruning and renormalization, the updated probability is overconfident with respect to the original probability. This is especially problematic for systems that require high accuracy. If the prior probability of the classes is independent, the original normalization factor can be computed efficiently without pruning hypotheses. To the best of our knowledge, this is the first work to present these results. If the prior probability of the classes is dependent, we propose a lower bound on the normalization factor that ensures cautious results. The bound is calculated incrementally and with similar efficiency as in the independent case. After pruning and updating based on the bound, this belief is shown empirically to be close to the original belief.

Cooperative adaptive cruise control (CACC) is a smart transportation solution that can mitigate traffic jams and improve road safety. CACC performance is heavily impacted by communication time delay; moreover, control theory solutions generally compromise control performance by tuning control gains in order to maintain plant stability. We propose a control-machine learning hybrid approach called deep time delay filter (DTDF). DTDF predicts the present (un-delayed) car states given time delayed versions. We successfully train a neural network for the DTDF method and use a physical testbed to show that DTDF can mitigate the effects of constant time delays as large as 5s while maintaining superior control performance compared to that of a baseline control algorithm.

Maintaining balance to external pushes is one of the most important features for a humanoid to walk in a real environment. In particular, methods for counteracting to pushes using the centroidal angular momentum (CAM) control have been actively developed. In this paper, a CAM control scheme based on hierarchical quadratic programming (HQP) is proposed. The scheme of the CAM control consists of CAM tracking control and initial pose return control, which is hierarchically operated based on HQP to ensure the priority of CAM tracking performance. The proposed method is implemented in a capture point (CP) feedback control framework. Through simulations and experiments, the proposed method demonstrated more stable balance control performance than the previous method when the humanoid is walking in the presence of external perturbation.

Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics.

Lower Limb Exoskeletons (LLE) have received considerable interest in strength augmentation, rehabilitation, and walking assistance scenarios. For strength augmentation, LLE is expected to have the capability of reducing metabolic energy. However, the energy for adjusting Center of Gravity (CoG) is a main part of the total energy consumed during walking. This paper proposes a novel Human-exoskeleton Cooperative Balance (HCB) strategy which gives assistive torques balance ability and combine with the direction selected by the pilot to achieve balance walking of human-exoskeleton systems. In which, a Dynamic Torque Primitive Model (DTPM) is designed to plan a bionic assistive torque, and the balance parameters obtained by an Inverted Pendulum Model (IPM) is superimposed on it. Finally, the performance improved by the HCB strategy can break the limitation of traditional strategies and substantially increase the efficiency of assistance. We demonstrated the effectiveness of the proposed HCB strategy on the HUman-powered Augmentation Lower EXoskeleton (HUALEX) system. Experimental results indicate that the proposed HCB is more efficient than traditional strategies.

In this work, we introduce an adaptive control framework for human-robot collaborative transportation of objects with unknown deformation behaviour. The proposed framework takes as input the haptic information transmitted through the object, and the kinematic information of the human body obtained from a motion capture system to create reactive whole-body motions on a mobile collaborative robot. In order to validate our framework experimentally, we compared its performance with an admittance controller during a co-transportation task of a partially deformable object. We additionally demonstrate the potential of the framework while co-transporting rigid (aluminum rod) and highly deformable (rope) objects. A mobile manipulator which consists of an Omni-directional mobile base, a collaborative robotic arm, and a robotic hand is used as the robotic partner in the experiments. Quantitative and qualitative results of a 12-subjects experiment show that the proposed framework can effectively deal with objects of unknown deformability and provides intuitive assistance to human partners.

In order to properly integrate humanoid robots in real-life situations, they must be able to collaborate with humans in completing tasks. One of these tasks is the cooperative transportation of a heavy object, which has been widely studied in the humanoids literature. However, the proposed methods rely heavily on six-axis force/torque (F/T) sensors at the wrists, which medium-sized or even some full-sized humanoid robots do not have. This paper proposes an observer to overcome the lack of F/T sensors. The observer is then coupled with a simplified dynamic model of the transportation task allowing the humanoid robot to carry out the task in a stable way. The method is tested in simulation using a humanoid robot that does not have F/T sensors, a NAO robot, to demonstrate its performance. These tests pointed out that the proposed method successfully estimated the interaction forces while generating stable walking patterns.

Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the action distributions of teacher policies, but also data collected by such policies on the task at hand. We conduct a thorough experimental study of how to use suboptimal teachers on a challenging robotic manipulation benchmark on vision-based stacking with diverse objects. We compare our methods to offline, online, offline-to-online, and kickstarting RL algorithms. By doing so, we find that training on data from both the teacher and student, enables the best performance for limited data budgets. We examine how to best allocate a limited data budget - on the target task - between the teacher and the student policy, and report experiments using varying budgets, two teachers with different degrees of suboptimality, and five stacking tasks that require a diverse set of behaviors. Our analysis, both in simulation and in the real world, shows that our approach is the best across data budgets, while standard offline RL from teacher rollouts is surprisingly effective when enough data is given.

Autonomous vehicles (AVs) rely on environment perception and behavior prediction to reason about agents in their surroundings. These perception systems must be robust to adverse weather such as rain, fog, and snow. However, validation of these systems is challenging due to their complexity and dependence on observation histories. This paper presents a method for characterizing failures of LiDAR-based perception systems for AVs in adverse weather conditions. We develop a methodology based in reinforcement learning to find likely failures in object tracking and trajectory prediction due to sequences of disturbances. We apply disturbances using a physics-based data augmentation technique for simulating LiDAR point clouds in adverse weather conditions. Experiments performed across a wide range of driving scenarios from a real-world driving dataset show that our proposed approach finds high likelihood failures with smaller input disturbances compared to baselines while remaining computationally tractable. Identified failures can inform future development of robust perception systems for AVs.

This paper presents novel quadrotor structures composed of only clockwise rotors. A multirotor unmanned aerial vehicle (UAV) generally has both clockwise and counterclockwise rotors to counteract the torques from the rotors. While the proposed structures have only clockwise rotors, those rotors are tilted to cancel the torques around the yaw angle of the body. This paper investigates the conditions for the proposed structures to achieve static hovering. More specifically, we provide a guideline to design the rotor tilt angles of the proposed structure. Then, this paper presents the design example of the rotor tilt angles and develops a prototype of the proposed quadrotor. The cascaded controller is also developed for the proposed structure. Finally, experimental validation is conducted with a developed prototype and controller.

In this work, an efficient homology guided belief space planning method for obstacle-cluttered environments is presented. The proposed planner follows a two-step approach. First, a h-signature guided rapidly-exploring random tree (HRRT) algorithm is proposed to provide nominal trajecto-ries in different homology classes by constructing homology aware sub-trees in a parallel manner. The HRRT planner is extended to a h-signature guided RRT* algorithm, where an inter-homology-class rewire procedure is proposed, increasing the probability of discovering homology classes in narrow space/passages. The iLQG-based belief space planning algorithm is then employed to find locally optimal trajectories minimizing uncertainties in each homology class.

Sonar sensors play an integral part in underwater robotic perception by providing imagery at long distances where standard optical cameras cannot. They have proven to be an important part in various robotic algorithms including localization, mapping, and structure from motion. Unfortunately, generating realistic sonar imagery for algorithm development is difficult due to the high cost of field trials and lack of simulation methods. To remove these obstacles, we present various upgrades to the sonar simulation method in HoloOcean, our open-source marine robotics simulator. In particular, we improve the noise modeling using a novel cluster-based multipath ray-tracing algorithm, various probabilistic noise models, and material dependence. We also develop and integrate simulated models for side-scan, single-beam, and multibeam profiling sonars.

Although there is extensive research regarding legged manipulators, comparatively little focuses on their User Interfaces (UIs). Towards extending the state-of-art in this domain, in this work, we integrate a Boston Dynamics (BD) Spot® with a light-weight 7 DoF Kinova® robot arm and a Robotiq® 2F-85 gripper into a legged manipulator. Furthermore, we jointly control the robotic platform using an affordance-aware Augmented Reality (AR) Head-Mounted Display (HMD) UI developed for the Microsoft HoloLens 2. We named the combined platform Holo-SpoK. Moreover, we explain how this manipulator colocalises with the HoloLens 2 for its control through AR. In addition, we present the details of our algorithms for autonomously detecting grasp-ability affordances and for the refinement of the positions obtained via vision-based colocalisation. We validate the suitability of our proposed methods with multiple navigation and manipulation experiments. To the best of our knowledge, this is the first demonstration of an AR HMD UI for controlling legged manipulators.