IROS 2022 Papers

This paper presents a novel control approach, based on the interconnection and damping-assignment passivity-based control (IDA-PBC), to achieve stable and periodic walking for underactuated planar biped robots with one degree of underactuation. The system's physical structure is preserved by assigning a target port-Hamiltonian dynamics to the closed-loop system, which also ensures passivity. The control design ensures that the tracking error to the desired periodic gait converges exponentially to zero, and the convergence rate can be adjusted via gain tuning. Besides, through the hybrid zero dynamics, the stability of the full-order system can be retrieved from the stability of the orbit created in a lower-dimensional manifold. The proposed approach is the first example of a tracking controller based on the IDA-PBC applied to underactuated biped robots. Numerical simulations on a five-link planar biped robot with unactuated ankles validate the approach and show the performance of the closed-loop system.

Human activities recognition is an important task for an intelligent robot, especially in the field of human-robot collaboration, it requires not only the label of sub-activities but also the temporal structure of the activity. In order to automatically recognize both the label and the temporal structure in sequence of human-object interaction, we propose a novel Pyramid Graph Convolutional Network (PGCN), which employs a pyramidal encoder-decoder architecture consisting of an attention based graph convolution network and a temporal pyramid pooling module for downsampling and upsampling interaction sequence on the temporal axis, respectively. The system represents the 2D or 3D spatial relation of human and objects from the detection results in video data as a graph. To learn the human-object relations, a new attention graph convolutional network is trained to extract condensed information from the graph representation. To segment action into sub-actions, a novel temporal pyramid pooling module is proposed, which upsamples compressed features back to the original time scale and classifies actions per frame. We explore various attention layers, namely spatial attention, temporal attention and channel attention, and combine different upsampling decoders to test the performance on action recognition and segmentation. We evaluate our model on two challenging datasets in the field of human-object interaction recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate that our classifier significantly improves both framewise action recognition and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset are improved by 4.3% and 8.5% respectively.

Humans use audio signals in the form of spoken language or verbal reactions effectively when teaching new skills or tasks to other humans. While demonstrations allow humans to teach robots in a natural way, learning from trajectories alone does not leverage other available modalities including audio from human teachers. To effectively utilize audio cues accompanying human demonstrations, first it is important to understand what kind of information is present and conveyed by such cues. This work characterizes audio from human teachers demonstrating multi-step manipulation tasks to a situated Sawyer robot along three dimensions: (1) duration of speech used, (2) expressiveness in speech or prosody, and (3) semantic content of speech. We analyze these features for four different independent variables and find that teachers convey similar semantic content via spoken words for different conditions of (1) demonstration types, (2) audio usage instructions, (3) subtasks, and (4) errors during demonstrations. However, differentiating properties of speech in terms of duration and expressiveness are present for the four independent variables, highlighting that human audio carries rich information, potentially beneficial for technological advancement of robot learning from demonstration methods.

Active inference is a mathematical framework that originated in computational neuroscience. Recently, it has been demonstrated as a promising approach for constructing goal-driven behavior in robotics. Specifically, the active inference controller (AIC) has been successful on several continuous control and state-estimation tasks. Despite its relative success, some established design choices lead to a number of practical limitations for robot control. These include having a biased estimate of the state, and only an implicit model of control actions. In this paper, we highlight these limitations and propose an extended version of the unbiased active inference controller (u-AIC). The u-AIC maintains all the compelling benefits of the AIC and removes its limitations. Simulation results on a 2-DOF arm and experiments on a real 7-DOF manipulator show the improved performance of the u-AIC with respect to the standard AIC. The code can be found at https://github.com/cpezzato/unbiasedaic.

Small untethered soft robots have potential for diverse applications, particularly in constrained spaces where the use of a tethered device would be infeasible. Examples include biomedical applications such as brachytherapy, fine-needle biospy and micro-needle drug delivery. To advance soft robots towards these applications, there is a need to establish methods for tracking and control using clinically-relevant methods. This study demonstrates motion planning and magnetic control of a soft untethered robot, using ultrasound images as feedback. The closed-loop control of the Millipede soft robot is first validated using a camera-based tracker, where the deviation between the planned path and the trajectory of the robot is 1.71 mm. Afterwards, two methods for ultrasound-based tracking capable of estimating the pose of the robot are proposed, a geometric approach and a convolutional neural network (CNN), and their performance is compared using a video camera as ground truth. Following this, the CNN method replaces the camera tracker to estimate the position and orientation of the robot. The closed-loop system using ultrasound images guides the robot through the workspace while avoiding virtual obstacles, and achieves an average tracking error of 1.59 mm and an angle error of 2.24°.

Considering realistic characteristics of acoustic localization methods is crucial for roboticists when developing guidance and control algorithms for small and agile underwater robots. Current simulators either rely purely on geometric distancing, i.e. do not consider dynamic effects such as robot motion during acoustic signal propagation, or they are too complex for usage by non-communication experts and, thus, vulnerable to misconfiguration. We propose an open ROS-based framework that extends existing robot simulators (e. g. Gazebo) by simulating the effects of realistic acoustic ranging for underwater robot localization. Thus, our simulator enables realistic real-time analysis and evaluation of guidance, navigation, and control algorithms in software in-the-loop systems. For this purpose, we incorporate and encapsulate the non-trivial characteristics of acoustic communication and ranging such as robot motion during signal propagation, packet reception failure, and modem timings. This ensures the applicability of the tool by roboticists who are typically non-experts in acoustic communication and guarantees accurate and realistic simulation results. We demonstrate the functionality and performance of our framework and validate it on real-world experimental data on the example of a two-way ranging method. Our open-source release includes well-defined interfaces and parameters as well as a tutorial. This targets other roboticists who can either use our framework directly or easily adapt it to their individual setup, e. g., by adding further acoustic-ranging protocols.

Simultaneous Localization and Mapping (SLAM) has greatly assisted in exploring perceptually-degraded underground environments, such as human-made tunnels, mine tunnels, and caves. However, the recurring sensor failures and spurious loop closures in these scenes bring significant challenges to applying SLAM. This paper proposes an architecture for underground localization and semantic mapping (ULSM) that promotes the robustness of odometry estimation and map-building. In this architecture, a two-stage robust motion compensation method is proposed to adapt to sensor-failure situations. The proposed salient region loop closure detection contributes to avoiding spurious loop closures. Meanwhile, the 2D pose as the initial value for point cloud registration is estimated without additional input. We also design a multi-robot cooperative mapping scheme based on descriptors of the salient region. Extensive experiments are conducted on datasets collected in the Tunnel Circuit of DARPA Subterranean Challenge.

We propose a novel hybrid system (both hardware and software) of an Unmanned Aerial Vehicle (UAV) carrying a miniature Unmanned Ground Vehicle (miniUGV) to perform a complex search and manipulation task. This system leverages the heterogeneous robots to accomplish a task that cannot be done using a single robot system. It enables the UAV to explore a hidden space with a narrow opening through which the miniUGV can easily enter and escape. The hidden space is assumed to be navigable for the miniUGV. The miniUGV uses Infrared (IR) sensors and a monocular camera to search an object in the hidden space. The proposed system takes advantage of a wider field of view (fov) of camera as well as the stochastic nature of the object detection algorithms to guide the miniUGV in the hidden space to find the object. Upon finding the object the miniUGV grabs it using visual servoing and then returns back to its start point from where the UAV retracts it back and transports the object to a safe place. In case there is no object found in the hidden space, UAV continues the aerial search. The tethered miniUGV gives the UAV an ability to act beyond its reach and perform a search and manipulation task which was not possible before for any of the robots individually. The system has a wide range of applications and we have demonstrated its feasibility through repetitive experiments.

In this paper, we present a relative pose estimation algorithm based on lines knowing the vertical direction associated to each image. We demonstrate that a closed-form solution requiring only eight lines between three views is possible. As a linear solution, it is shown that our approach outperforms the standard trifocal estimation based on 13 triplets of lines and can be efficiently inserted into an hypothesize-and-test framework such as RANSAC. We also study our approach on different singular configurations of lines. The method is evaluated on both synthetic data and real-world sequences from KITTI and the Zürich Urban Micro Aerial Vehicle datasets. Our method is compared to 13 lines algorithm as well to points based methods such as 7-points, 5-points and 3-points.

Rearrangement tasks have been identified as a crucial challenge for intelligent robotic manipulation, but few methods allow for precise construction of unseen structures. We propose a visual foresight model for pick-and-place rearrangement manipulation which is able to learn efficiently. In addition, we develop a multi-modal action proposal module which builds on the Goal-Conditioned Transporter Network, a state-of-the-art imitation learning method. Our image-based task planning method, Transporters with Visual Foresight, is able to learn from only a handful of data and generalize to multiple unseen tasks in a zero-shot manner. TVF is able to improve the performance of a state-of-the-art imitation learning method on unseen tasks in simulation and real robot experiments. In particular, the average success rate on unseen tasks improves from 55.4% to 78.5% in simulation experiments and from 30% to 63.3% in real robot experiments when given only tens of expert demonstrations. Video and code are available on our project website: https://chirikjianlab.github.io/tvf/

Fault mitigation in Connected Autonomous Vehicle (CAV) platoons is faster and more reliable if the fault structure is known. In this paper we propose using transmissibility operators, which are relationships that relate a set of velocities with another in the platoon, to classify the faults. Transmissibility operators were shown to be exceptional in signals estimation; however, its also shown to be noncausal and thus can only be used offline. To this end, we propose using Data Aggregation (DAgger), which is an extension in imitation learning to transfer the classification experience from transmissibility operators to a novice machine learning agent to be used online. A heterogeneous CAV platoon was modeled with three different faults separately. These faults are actuator disturbances, false data injection attacks, and communication time delay. The proposed algorithm is then tested on the platoon model and then applied to an experimental setup that consists of three autonomous robots. The overall classification accuracy achieved was 95.8% for the experiment.

Autonomous Driving requires high levels of coordination and collaboration between agents. Achieving effective coordination in multi-agent systems is a difficult task that remains largely unresolved. Multi-Agent Reinforcement Learning has arisen as a powerful method to accomplish this task because it considers the interaction between agents and also allows for decentralized training—which makes it highly scalable. However, transferring policies from simulation to the real world is a big challenge, even for single-agent applications. Multi-agent systems add additional complexities to the Sim-to-Real gap due to agent collaboration and environment synchronization. In this paper, we propose a method to transfer multi-agent autonomous driving policies to the real world. For this, we create a multi-agent environment that imitates the dynamics of the Duckietown multi-robot testbed, and train multi-agent policies using the MAPPO algorithm with different levels of domain randomization. We then transfer the trained policies to the Duckietown testbed and show that when using our method, domain randomization can reduce the reality gap by 90%. Moreover, we show that different levels of parameter randomization have a substantial impact on the Sim-to-Real gap. Finally, our approach achieves significantly better results than a rule-based benchmark.

In-hand manipulation of objects is an important capability to enable robots to carry-out tasks which demand high levels of dexterity. This work presents a robot systems approach to learning dexterous manipulation tasks involving moving objects to arbitrary 6-DoF poses. We show empirical benefits, both in simulation and sim - to- real transfer, of using keypoint-based representations for object pose in policy observations and reward calculation to train a model-free reinforcement learning agent. By utilizing domain randomization strategies and large-scale training, we achieve a high success rate of 83 % on a real TriFinger system, with a single policy able to perform grasping, ungrasping, and finger gaiting in order to achieve arbitrary poses within the workspace. We demonstrate that our policy can generalise to unseen objects, and success rates can be further improved through finetuning. With the aim of assisting further research in learning in-hand manipulation, we provide a detailed exposition of our system and make the codebase of our system available, along with checkpoints trained on billions of steps of experience, at https://s2r2-ig.github.io

In this paper, we present a Domain Randomization and a Domain Adaptation approach to transfer experience for entanglement detection and separation from simulation into a real-world bin-picking application. We investigate the influence of different randomization options in image processing and use a CycleGAN as a further Domain Adaptation method to synthesize simulation data as realistically as possible. On the basis of this adapted data we re-train our detection and separation methods and validate the usefulness of these Sim-to-Real methods. In numerous real-world experiments we show that we achieve a significant increase of up to 71.74 % in the performance of the overall system by using the Sim-to-Real approaches as opposed to the direct transfer.

In the Robot Operating System (ROS), a major middleware for robots, the Transform Library (TF) is a mandatory package that manages transformation information between coordinate systems by using a single-rooted directed tree and providing methods for registering and computing the information. However, the tree has two fundamental problems. The first is its poor scalability: since it accepts only a single thread at a time due to using a single giant lock for mutual exclusion, the access to the tree is sequential. Second, there is a lack of data freshness: it retrieves non-latest synthetic data when computing coordinate transformations because it prioritizes temporal consistency over data freshness. In this paper, we propose methods to solve these problems. First, we decentralize the giant lock to provide performance scalability and show that this results in a throughput 243 times higher than conventional TF on a read-only workload. Second, we design transactional methods based on serializable protocols that prevent anomalies, thus retrieving the freshest data. These transactional methods show a freshness up to 1276 times higher than the conventional one on a read-write combined workload.

Traditional video-based human activity recognition has experienced remarkable progress linked to the rise of deep learning, but this effect was slower as it comes to the downstream task of driver behavior understanding. Understanding the situation inside the vehicle cabin is essential for Advanced Driving Assistant System (ADAS) as it enables identifying distraction, predicting driver's intent and leads to more convenient human-vehicle interaction. At the same time, driver observation systems face substantial obstacles as they need to capture different granularities of driver states, while the complexity of such secondary activities grows with the rising automation and increased driver freedom. Furthermore, a model is rarely deployed under conditions identical to the ones in the training set, as sensor placements and types vary from vehicle to vehicle, constituting a substantial obstacle for real-life deployment of data-driven models. In this work, we present a novel vision-based framework for recognizing secondary driver behaviours based on visual transformers and an additional augmented feature distribution calibration module. This module operates in the latent feature-space enriching and diversifying the training set at feature-level in order to improve generalization to novel data appearances, (e.g., sensor changes) and general feature quality. Our framework consistently leads to better recognition rates, surpassing previous state-of-the-art results of the public Drive&Act benchmark on all granularity levels. Our code will be made publicly available at https://github.com/KPeng9510/TransDARC.

Motion prediction for traffic participants is essential for a safe and robust automated driving system, especially in cluttered urban environments. However, it is highly challenging due to the complex road topology as well as the uncertain intentions of the other agents. In this paper, we present a graph-based trajectory prediction network named the Dual Scale Predictor (DSP), which encodes both the static and dynamical driving context in a hierarchical manner. Different from methods based on a rasterized map or sparse lane graph, we consider the driving context as a graph with two layers, focusing on both geometrical and topological features. Graph neural networks (GNNs) are applied to extract features with different levels of granularity, and features are subsequently aggregated with attention-based inter-layer networks, realizing better local-global feature fusion. Following the recent goal-driven trajectory prediction pipeline, goal candidates with high likelihood for the target agent are extracted, and predicted trajectories are generated conditioned on these goals. Thanks to the proposed dual-scale context fusion network, our DSP is able to generate accurate and human-like multi-modal trajectories. We evaluate the proposed method on the large-scale Argoverse motion forecasting benchmark, and it achieves promising results, outperforming the recent state-of-the-art methods. We release the code on our project website. 11https://github.com/HKUST-Aerial-Robotics/DSP

Space robotics applications, such as Active Space Debris Removal (ASDR), require representative testing before launch. A commonly used approach to emulate the microgravity environment in space is air-bearing based platforms on flat-floors, such as the European Space Agency's Orbital Robotics and GNC Lab (ORGL). This work proposes a control architecture for a floating platform at the ORGL, equipped with eight solenoid-valve-based thrusters and one reaction wheel. The control architecture consists of two main components: a trajectory planner that finds optimal trajectories connecting two states and a trajectory follower that follows any physically feasible trajectory. The controller is first evaluated within an introduced simulation, achieving a 100% success rate at finding and following trajectories to the origin within a Monte-Carlo test. Individual trajectories are also successfully followed by the physical system. In this work, we showcase the ability of the controller to reject disturbances and follow a straight-line trajectory within tens of centimeters.

Developing a good control strategy for biomimetic robots is challenging. Robust control methods require an accurate model of the robot. Nowadays, model-free methods are being extensively explored for the control and navigation of terrestrial robots. In this paper, we consider a novel deep reinforcement learning-based model-free swimming control for our bio-inspired robotic tadpole. To realize this, we utilize dynamic motion primitives, which can represent a large range of motion behaviors, and combine them with a decoupled reinforcement learning framework. The proposed architecture optimizes the motion primitives first to develop a travelling wave undulation pattern in the tail and then to navigate the robot along different predefined paths. Through this framework, effective swimming gait emerges, and the robot is able to navigate well on the surface of water. This framework combines the optimization potential of deep reinforcement learning with stability and generalization properties of dynamic motion primitives. We train and test our method on a simulated model of the robot to demonstrate the effectiveness of the method and also conduct experimental testing on the real robot to verify the results.