IROS 2022 Papers

Robot manipulation in cluttered environments of-ten requires complex and sequential rearrangement of multiple objects in order to achieve the desired reconfiguration of the target objects. Due to the sophisticated physical interactions involved in such scenarios, rearrangement-based manipulation is still limited to a small range of tasks and is especially vulnerable to physical uncertainties and perception noise. This paper presents a planning framework that leverages the efficiency of sampling-based planning approaches, and closes the manipulation loop by dynamically controlling the planning horizon. Our approach interleaves planning and execution to progressively approach the manipulation goal while correcting any errors or path deviations along the process. Meanwhile, our framework allows the definition of manipulation goals without requiring explicit goal configurations, enabling the robot to flexibly interact with all objects to facilitate the manipulation of the target ones. With extensive experiments both in simulation and on a real robot, we evaluate our framework on three manipulation tasks in cluttered environments: grasping, relocating, and sorting. In comparison with two baseline approaches, we show that our framework can significantly improve planning efficiency, robustness against physical uncertainties, and task success rate under limited time budgets.

In this study, seated walk, a movement of walking while sitting on a chair with casters, is realized on a musculoskeletal humanoid from human teaching. The body is balanced by using buttock-contact sensors implemented on the planar interskeletal structure of the human mimetic musculoskeletal robot. Also, we develop a constrained teaching method in which one-dimensional control command, its transition, and a transition condition are described for each state in advance, and a threshold value for each transition condition such as joint angles and foot contact sensor values is determined based on human teaching. Complex behaviors can be easily generated from simple inputs. In the musculoskeletal humanoid MusashiOLegs, forward, backward, and rotational movements of seated walk are realized.

Simulation remains one of the key methods for testing and validation of robotic perception systems and it also becomes increasingly important for training visuomotor policies for autonomous driving or manipulation. Further, as perception pipelines tend to leverage increasing amounts of modalities, it appears vital to simulate additional cues such as depth maps aside from RGB images. To align simulation with real-world observations, it is key to achieve realistic renderings of these maps, which includes the capability of rendering other dynamic objects in the scene. In this work, we propose an approach to real-time simulation of photo-realistic RGB images and sensor-realistic depth maps, that can contain dynamic objects at user-defined locations. Our method employs a selection of static samples of a pre-recorded database and multimodal cues from CAD models that are fused and warped to synthesize new imagery for a target camera pose. We show the efficacy of our method on newly proposed datasets recorded in a variety of different setups.

Synthetic image generation plays a crucial role in the development of robot vision algorithms, circumventing manual data collection. However, the realism of synthetic images could affect the performance of the algorithms when applied in real-world settings. In this study, we propose a framework to quantitatively assess the realism of synthetic images using a set of realism metrics as a means of performance characterization. We use a commercial rendering engine as a test-bed for generating synthetic images and ascertain that a set of rendering parameters affect specific image metrics through statistical hypothesis testing. We demonstrate that this framework can be used to optimize rendering parameter values and generate synthetic datasets with improved performance on downstream robot vision tasks such as instance segmentation.

Disinfection robots have applications in promoting public health and reducing hospital acquired infections and have drawn considerable interest due to the COVID-19 pan-demic. To disinfect a room quickly, motion planning can be used to plan robot disinfection trajectories on a reconstructed 3D map of the room's surfaces. However, existing approaches discard semantic information of the room and, thus, take a long time to perform thorough disinfection. Human cleaners, on the other hand, disinfect rooms more efficiently by prioritizing the cleaning of high-touch surfaces. To address this gap, we present a novel GPU-based volumetric semantic TSDF (Truncated Signed Distance Function) integration system for semantic 3D reconstruction. Our system produces 3D reconstructions that distinguish high-touch surfaces from non-high-touch surfaces at approximately 50 frames per second on a consumer-grade GPU, which is approximately 5 times faster than existing CPU-based TSDF semantic reconstruction methods. In addition, we extend a UV disinfection motion planning algorithm to incorporate semantic awareness for optimizing coverage of disinfection tra-jectories. Experiments show that our semantic-aware planning outperforms geometry-only planning by disinfecting up to 20% more high-touch surfaces under the same time budget. Further, the real-time nature of our semantic reconstruction pipeline enables future work on simultaneous disinfection and mapping. Code is available at: https://github.com/uiuc-iml/RA-SLAM

In modern highly dynamic robot manipulation, collisions between a robot and objects may be intentionally executed to improve performance. To distinguish between these deliberate contacts and accidental collisions beyond the limit of state-of-the-art human-robot interactions, new sensing approaches are required. This work seeks an easy-to-implement and real-time capable solution to detect the identity of the impacted material. We developed an inertial measurement unit (IMU) based setup that records vibration signals occurring after collisions. Furthermore, a data-set was generated in an unsupervised learning manner using the measurements of collision experiments with several materials commonly used in realistic applications. The data-set was used to train an artificial neural network to classify the type of material involved. Our results show that the neural net detects collisions and a detailed distinction between materials is achieved, even with estimating different human body parts. The unsupervised data-set generation allows for a simple integration of new classes, which provides broader applicability of our approach. As the calculations are running faster than the control cycle of the robot, the output of our classifier can be used in real-time to decide about the robots reaction behavior.

Quadruped robots have proved their robustness to cross complex terrain despite little environment knowledge. Yet advanced locomotion controllers are expected to take advantage of exteroceptive information. This paper presents a complete method to plan and control the locomotion of quadruped robots when 3D information about the surrounding obstacles is available, based on several stages of decision. We first propose a contact planner formulated as a mixed-integer program, optimized on-line at each new robot step. It selects a surface from a set of convex surfaces describing the environment for the next footsteps while ensuring kinematic constraints. We then propose to optimize the exact contact location and the feet trajectories at control frequency to avoid obstacles, thanks to an efficient formulation of quadratic programs optimizing Bezier curves. By relying on the locomotion controller of our quadruped robot Solo, we finally implement the complete method, provided as an open-source package. Its efficiency is asserted by statistical evaluation of the importance of each component in simulation. We have a 100% success rate for our framework, and we show that the deactivation of the contact planning, footstep adaptation and collision avoidance, respectively induced a drop to 70%, 62% and 83% success rate in the worst case, justifying the complete architecture.

Inspired by the digital twinning systems, a novel real-time digital double framework is developed to enhance robot perception of the terrain conditions. Based on the very same physical model and motion control, this work exploits the use of such simulated digital double synchronized with a real robot to capture and extract discrepancy information between the two systems, which provides high dimensional cues in multiple physical quantities to represent differences between the modelled and the real world. Soft, non-rigid terrains cause common failures in legged locomotion, whereby visual perception solely is insufficient in estimating such physical properties of terrains. We used digital double to develop the estimation of the collapsibility, which addressed this issue through physical interactions during dynamic walking. The discrepancy in sensory measurements between the real robot and its digital double are used as input of a learning-based algorithm for terrain collapsibility analysis. Although trained only in simulation, the learned model can perform collapsibility estimation successfully in both simulation and real world. Our evaluation of results showed the generalization to different scenarios and the advantages of the digital double to reliably detect nuances in ground conditions.

Acoustic holography is a newly emerging and promising technique to dynamically generate arbitrary desired holographic acoustic field in 3D space for contactless robotic manipulation. The latest technology supporting complex dynamic holographic acoustic field reconstruction is through phased transducer array (PTA), where the phase profile of emitted acoustic wave from discrete transducers is controlled independently by sophisticated circuits to modulate the acoustic interference field. While the forward kinematics of a phased array based robotic manipulation system is simple and straightforward, the inverse kinematics of the required holographic acoustic field is mathematically non-linear and unsolvable, which substantially limits the application of dynamic holographic acoustic field for robot manipulation. In this work, we propose a physics-based deep learning framework for this phase retrieval inverse kinematics problem so that the target complex hologram could be reconstructed precisely with average MAE of 0.022 and in real time with prediction time of 47 milliseconds on GPU. The accuracy and real time of the proposed method for dynamic holographic acoustic field reconstruction from PTA are demonstrated experimentally.

Visual Inertial Odometry (VIO) is a widely studied localization technique in robotics. State-of-the-art VIO algorithms are composed of two parts: a frontend which performs visual perception and inertial measurement pre-processing, and a backend which fuses vision and inertial measurements to estimate the robot's pose. Both image processing in the frontend and sensor fusion in the backend are computationally expensive, making it very challenging to run the VIO algorithm, especially the optimization-based VIO algorithm in real time on embedded platforms with limited power budget. In this paper, a real-time optimization-based monocular VIO algorithm is proposed based on algorithm-and-hardware co-design and successfully implemented on an embedded platform with only 2.6W processor power consumption. In particular, the time-consuming Harris corner detection (HCD) is accelerated on Field Programmable Gate Array (FPGA), achieving an average 16 × processing time reduction compared with the ARM implementation. Compared with the state-of-the-art HCD accelerator provided by Xilinx, the hardware resource required of our accelerator is largely reduced without any compromise in speed, thanks to the proposed dedicated pruning and paral-lelization techniques. Finally, experiment on the public dataset demonstrates that the proposed real-time VIO algorithm on the FPGA-based platform has comparable accuracy with respect to the existing state-of-the-art VIO algorithm on the desktop, and 3 × faster frontend processing speed over the ARM-based implementation.

This paper presents a novel trajectory planning method for aerial perching. Compared with the existing work, the terminal states and the trajectory durations can be adjusted adaptively, instead of being determined in advance. Further-more, our planner is able to minimize the tangential relative speed on the premise of safety and dynamic feasibility. This feature is especially notable on micro aerial robots with low maneuverability or scenarios where the space is not enough. Moreover, we design a flexible transformation strategy to eliminate terminal constraints along with reducing optimization variables. Besides, we take precise SE(3) motion planning into account to ensure that the drone would not touch the landing platform until the last moment. The proposed method is validated onboard by a palm-sized micro aerial robot with quite limited thrust and moment (thrust-to-weight ratio 1.7) perching on a mobile inclined surface. Sufficient experimental results show that our planner generates an optimal trajectory within 20ms, and replans with warm start in 2ms.

Modern collaborative manipulators operate in unknown environments and share the work space with human coworkers. To ensure flexibility, their kinematic design is redundant which increases the solution space of the inverse kinematics (IK). We propose a real-time capable Predictive Kinematics Controller (PKC) that tracks task space trajectories as a first priority and computes optimal joint trajectories w.r.t. secondary objectives based on model predictive control (MPC). Therefor, the PKC solves a MPC problem in the nullspace of the task space trajectory. We benchmark a direct shooting, a direct collocation and an indirect gradient method in simulation and we identify the direct shooting method as the most efficient. We demonstrate the superior performance of the PKC compared to state-of-the-art local redundancy resolution approaches. In experiments, we show the real-time capability of our implementation.

Unmanned aerial vehicles (UAVs) have been used for many applications in recent years, from urban search and rescue, to agricultural surveying, to autonomous underground mine exploration. However, deploying UAVs in tight, indoor spaces, especially close to humans, remains a challenge. One solution, when limited payload is required, is to use micro-UAVs, which pose less risk to humans and typically cost less to replace after a crash. However, micro-UAVs can only carry a limited sensor suite, e.g. a monocular camera instead of a stereo pair or LiDAR, complicating tasks like dense mapping and markerless multi-person 3D human pose estimation, which are needed to operate in tight environments around people. Monocular approaches to such tasks exist, and dense monocular mapping approaches have been successfully deployed for UAV applications. However, despite many recent works on both marker-based and markerless multi-UAV single-person motion capture, markerless single-camera multi-person 3D human pose estimation remains a much earlier-stage technology, and we are not aware of existing attempts to deploy it in an aerial context. In this paper, we present what is thus, to our knowledge, the first system to perform simultaneous mapping and multi-person 3D human pose estimation from a monocular camera mounted on a single UAV. In particular, we show how to loosely couple state-of-the-art monocular depth estimation and monocular 3D human pose estimation approaches to reconstruct a hybrid map of a populated indoor scene in real time. We validate our component-level design choices via extensive experiments on the large-scale ScanNet and GTA-IM datasets. To evaluate our system-level performance, we also construct a new Oxford Hybrid Mapping dataset of populated indoor scenes.

Actively searching for targets using a multi-agent system in an unknown environment poses a two-pronged prob-lem, where on the one hand we need agents to cover as much of the environment as possible and on the other have a higher density of agents where there are potential targets to maximize detection performance. This paper proposes a fully distributed solution for an ad hoc network of agents to cooperatively search an unknown environment and actively track found targets. The solution combines a distributed pheromone-based coverage control strategy with a distributed target selection mechanism.

State-of-the-art reinforcement learning is now able to learn versatile locomotion, balancing and push-recovery capabilities for bipedal robots in simulation. Yet, the reality gap has mostly been overlooked and the simulated results hardly transfer to real hardware. Either it is unsuccessful in practice because the physics is over-simplified and hardware limitations are ignored, or regularity is not guaranteed, and unexpected hazardous motions can occur. This paper presents a reinforcement learning framework capable of learning ro-bust standing push recovery for bipedal robots that smoothly transfer to reality, providing only instantaneous proprioceptive observations. By combining original termination conditions and policy smoothness conditioning, we achieve stable learning, sim-to-real transfer and safety using a policy without memory nor explicit history. Reward engineering is then used to give insights into how to keep balance. We demonstrate its performance in reality on the lower-limb medical exoskeleton Atalante.

In this work we address the challenging problem of manipulating a flexible link, like a rope, with an aerial robot. Inspired by spraying tasks in construction and maintenance scenarios, we consider the case in which an autonomous end-effector (e.g., a spray nozzle moved by a robot or a human operator) is connected to a fixed point by a rope (e.g., a hose). To avoid collisions between the rope and the environment while the end-effector moves, we propose the use of an aerial robot as a flying companion to properly manipulate the rope away from collisions. The aerial robot is attached to the rope between the end-effector and the fixed point. Assuming no direct control of the end-effector (e.g., when operated by a human), we design a reactive and fast motion planner for the aerial robot. Grounding on the theory of Forced Geometric Fabrics, we design a motion planner that generates trajectories to drive the aerial robot to follow the end-effector, while manipulating the rope to avoid collisions in cluttered environments. To include the complex behavior of the flexible link, we propose a rope model that estimates its real-time state under forces and position-based interactions, as well as collisions with obstacle surfaces. Finally, we evaluate the system behavior and the motion planner performance in simulations, as well as in real-world experiments on an original spray painting application.

In this paper, we propose a trajectory planning framework for a robot that exploits a pre-computed database of end-effector trajectories as the guidance of optimization-based inverse kinematics. We constructed a reachable graph of a robot offline, which represents feasible end-effector paths with corresponding configurations. When performing the online trajectory planning, we applied A* search to the reachable graph to find a feasible path between input start and goal globally in the task space. Its cost function has the separated term dependent on the robot, which comes from the manipulability of configurations preserved in the reachable graph, and that is dependent on the environment. Then, we solve optimization-based inverse kinematics to generate an optimal joint trajectory while utilizing the end-effector trajectory and its corresponding configurations as the guidance to avoid local optimum. We evaluated our framework quantitatively by comparing it with existing methods to confirm that it achieved a high success rate and quality of results while suppressing its computational time. We also qualitatively proved its practicality by applying it to the material handling task in the real-world. This result shows that it improved the performance of the optimization-based inverse kinematics avoiding local optimum and applicability to the different environments of the pre-computed motion database.

In this paper, we propose a new self-reconfiguration scheme for modular robots based on a metamodule design that allows to form a 3D porous structure. The porous structure enables a parallel flow of modules inside it without blocking. The metamodule can also be used to fill its internal volume with an additional number of modules allowing the structure to be compressible and expandable. Hence, it is a potential for improving the self-reconfiguration process. We first present the metamodule model and the porous structure built using it. Then, we describe an algorithm to self-reconfigure the structure from an initial shape to a given goal shape. We evaluated the algorithm in simulation on structures composed of up to 2,700 modules. We studied the performance in term of parallelism, showed that the number of communications is proportional to the number of motions and the execution time varies linearly with the diameter of the configuration.