ICRA 2024 Papers

In the natural world, benefited from the advantages of the spine, quadrupeds exhibiting extraordinary flexibility which allowing them to move efficiently on variable terrains. The previous researches have indicated the legged robots which efficiently utilizing their spine can achieve rapid and stable locomotion. However, within the field of legged robot dynamics, the design of the spine and understanding how it positively influences locomotion is unclear, which is significant for quadruped robot to achieve efficient and stable walking. In this study, we proposed a model formed by tensegrity spine and rimless wheel to represent quadrupeds, using passive dynamic walking as a method, which has been well-demonstrated for observing the inherent characteristics, exhibited the locomotion characteristic of the model proposed. By numerical simulation, we observed change trend of locomotion performance with the configurations of spine’s shape, and found direction of spine design that have a positive impact on walking. These findings contribute to the design of spine structures in quadruped robots.

To enhance the precision of coarse long-stroke motion axes, complementary short-stroke fine positioning stages are usually introduced. Being mechanically attached, the motion of the combined positioning stages needs to be controlled and synchronized. Therefore, typically suitable model-based controllers of fine stages are designed according to the sophisticated models and identification techniques used. Due to their appealing features, Piezocermamic-based fine positioning stages were successfully utilized in many applications, which recently sparked their use in high-acceleration motion found in wafer scanners, for example, where high-precision motion is required despite the resulting high inertial forces involved. Unfortunately, hard nonlinear behavior is associated with piezoelectric actuators, which adds to the complexity of modeling, control, and synchronization processes. To overcome such a burden, in this study, the design procedure of a model-free control and synchronization technique of piezocermamic-based fine positioning stages is introduced and verified experimentally using a representative precision motion system comprising a planner stage and a uni-axial fine stage under step-and-scan trajectories commonly used in wafer scanners. Despite its simplicity, the herein proposed design procedure can be seamlessly extended to other robotics and automation applications.

Robot learning methods have recently made great strides, but generalization and robustness challenges still hinder their widespread deployment. Failing to detect and address potential failures renders state-of-the-art learning systems not combat-ready for high-stakes tasks. Recent advances in interactive imitation learning have presented a promising framework for human-robot teaming, enabling the robots to operate safely and continually improve their performances over long-term deployments. Nonetheless, existing methods typically require constant human supervision and preemptive feedback, limiting their practicality in realistic domains. This work aims to endow a robot with the ability to monitor and detect errors during task execution. We introduce a model-based runtime monitoring algorithm that learns from deployment data to detect system anomalies and anticipate failures. Unlike prior work that cannot foresee future failures or requires failure experiences for training, our method learns a latent-space dynamics model and a failure classifier, enabling our method to simulate future action outcomes and detect out-of-distribution and high-risk states preemptively. We train our method within an interactive imitation learning framework, where it continually updates the model from the experiences of the human-robot team collected using trustworthy deployments. Consequently, our method reduces the human workload needed over time while ensuring reliable task execution. Our method outperforms the baselines across system-level and unit-test metrics, with 23% and 40% higher success rates in simulation and on physical hardware, respectively. More information at https://ut-austin-rpl.github.io/sirius-runtime-monitor/

A model-based approach for lateral maneuvering of flapping wing UAVs in closed spaces is presented. Bird-size ornithopters do not have asymmetric actuation in the wing due to mechanical complexity, so they rely upon the tail for lateral maneuvering. The prototype E-Flap can deflect the vertical tail to make maneuvers out of the longitudinal plane. This work defines simplified equations for the steady turning maneuver based on the body roll angle. The relation between the velocity of the prototype and the turning radius is also stated. Then, an approach to the attitude is proposed, defining the relation between the deflection of the vertical tail and the roll angle. We prove that, even though this deflection causes a yaw moment, the coupling between yaw and roll dynamics generates also a roll rate. To validate this simplified model, a simple control is presented for continuous circular trajectory tracking inside an indoor flight zone. The objective is to track circular trajectories of a radius 2 times greater than the wingspan at a constant height. Results show a very good agreement between the theoretical and experimental turning radius. In addition, the direct relation between the vertical tail deflection and the roll rate of the ornithopter is identified. Even though the desired radius is not reached, the FWUAV is capable of maintaining a closed turning maneuver for several laps. Therefore, the insight provided by the model proves to be an appropriate approach for aggressive lateral maneuvers of bird-size ornithopters.

Robots have a great potential to help people with movement limitations in activities of daily living, such as dressing. A common problem in almost all dressing tasks is the insertion of a garment’s opening around a part of the human body. The rich contact environment and the deformations of the garment make the task a challenging problem for robots. In this paper, we propose a bi-manual control method for garment opening insertion during robot-assisted dressing. Specifically, we propose a model predictive controller that uses an Attention-based Relational Graph Convolutional Network (ARGCN) for modeling the dynamics of the opening in the presence of the body. We train the model entirely in simulation and validate our method in four real-world dressing scenarios of a medical training manikin. We show that our method generalizes well in the real-world opening insertion tasks achieving an overall success rate of 97.5%, even though the dynamics and the shapes vastly differ from the simulation setup.

Due to the low motion efficiency and maneuver-ability of underwater robots with six degrees of freedom, it is challenging for them to respond quickly to the attitude requirements during underwater autonomous manipulation. This paper presents a novel autonomous underwater robot with fully vectored propulsion and a model predictive control method to achieve more agile and efficient movements autonomously. In detail, we first design a robot with eight vector-distributed thruster layouts for fully vectored propulsion and construct the software architecture based on the robot operating system (ROS). Then, we establish the hydrodynamic model by adopting the Fossen approach and construct a 13-dimensional system state-space equation, which is discretized using the explicit fourth-order Runge-Kutta method. To achieve autonomous manipulation, model predictive control is employed along with physical constraints of the custom-built robot to enable real-time prediction and optimization of the robot’s states for control purposes. Finally, numerical simulations and experiments of the Point-to-Point Motion are conducted to test the robot’s performance. Experimental results reveal that the average error of each direction is 0.0027 m, 0.0031 m, and 0.0368 m in the x-axis, y-axis, and z-axis, respectively, and 0.8502°, 2.1941°, 0.2408° corresponding to three attitude angles, which verify the performance of employing MPC to control an autonomous underwater robot with fully vectored propulsion.

The construction of large-scale space facilities requires the use of on-orbit construction technology. However, several of its key components, such as standard interface design, compliant control methods, and path planning for multi-branch robots, still need improvement before practical application. This paper presents a comprehensive solution for on-orbit construction tasks, encompassing a novel standard interface, docking control method, and path planning method for space multi-branch robots. Firstly, a novel standard interface is introduced, which features multiple mating modes and a lightweight design. Additionally, a compliant docking method is provided to generate lower contact forces along the Z-direction. Furthermore, for four-armed space robots, a hierarchical planning method is proposed, which innovates in environment map construction and locomotion planning. Specifically, the closed-form Minkowski sum method is employed to solve the robot’s free space, and a concise locomotion method is elucidated based on transition support points. Finally, simulations and experiments are conducted.

Most existing robotic datasets capture static scene data and thus are limited in evaluating robots’ dynamic performance. To address this, we present a mobile robot oriented large-scale indoor dataset, denoted as THUD (Tsinghua University Dynamic) robotic dataset, for training and evaluating their dynamic scene understanding algorithms. Specifically, the THUD dataset construction is first detailed, including organization, acquisition, and annotation methods. It comprises both real-world and synthetic data, collected with a real robot platform and a physical simulation platform, respectively. Our current dataset includes 13 larges-scale dynamic scenarios, 90K image frames, 20M 2D/3D bounding boxes of static and dynamic objects, camera poses, and IMU. The dataset is still continuously expanding. Then, the performance of mainstream indoor scene understanding tasks, e.g. 3D object detection, semantic segmentation, and robot relocalization, is evaluated on our THUD dataset. These experiments reveal serious challenges for some robot scene understanding tasks in dynamic scenes. By sharing this dataset, we aim to foster and iterate new mobile robot algorithms quickly for robot actual working dynamic environment, i.e. complex crowded dynamic scenes.

In the last few years, mobile robots such as floor cleaners, assistive robots, and home telepresence have become an essential part of our day-to-day activities. In human-robot interaction, speech is the preferred way of communication, especially in indoor environments. This paper proposes a speech module to rotate the mobile robot. It has two components, namely, a distant automatic speech recognizer and a sound source localizer. To build distant speech recognizer, far-field speech data is collected at 1, 3, and 5-meters distances. The model performs well even at a 5-meters distance with a Word Error Rate of 40.38% and a Character Error Rate of 28.85%. The direction of arrival of the speech signal is computed from the 4-mic circular array microphone. The speech module is integrated with the Robot Operating System and physically demonstrated on Turtlebot3 Waffle Pi. It is observed that the speech recognizer and sound source localizer work well in the reverberant indoor environment with a small single-board computer.

MoRC is a high-performance modular robot controller based on the Functional Mock-up Interface (FMI) standard. The goal is to control any (industrial) robot with electrical drives using a customizable vendor-agnostic control cabinet and an innovative, self-developed software architecture based on exchangeable multi-rate real-time control components with standardized interfaces. On the hardware side, the use of EtherCAT (Ethernet for Control Automation Technology) allows connecting a freely selectable number of COTS (commercial off-the-shelf) electrical drives and sensors. On the software side, this is matched with exchangeable control software modules based on the FMI standard. Those can be interconnected for forming user-defined multi-rate control structures which can be executed as synchronized real-time threads on a central Linux-based multi-core computing unit. That unlocks additional computational potential for advanced high-frequency control algorithms. Control structures can be switched at runtime to handle highly diverse control tasks. This paper presents the architectural concepts as well as first experiments on an industrial robot testbed.

Multi-modal unsupervised domain adaptation (MM-UDA) for 3D semantic segmentation is a practical solution to embed semantic understanding in autonomous systems without expensive point-wise annotations. While previous MM-UDA methods can achieve overall improvement, they suffer from significant class-imbalanced performance, restricting their adoption in real applications. This imbalanced performance is mainly caused by: 1) self-training with imbalanced data and 2) the lack of pixel-wise 2D supervision signals. In this work, we propose Multi-modal Prior Aided (MoPA) domain adaptation to improve the performance of rare objects. Specifically, we develop Valid Ground-based Insertion (VGI) to rectify the imbalance supervision signals by inserting prior rare objects collected from the wild while avoiding introducing artificial artifacts that lead to trivial solutions. Meanwhile, our SAM consistency loss leverages the 2D prior semantic masks from SAM as pixel-wise supervision signals to encourage consistent predictions for each object in the semantic mask. The knowledge learned from modal-specific prior is then shared across modalities to achieve better rare object segmentation. Extensive experiments show that our method achieves state-of-the-art performance on the challenging MM-UDA benchmark. Code will be available at https://github.com/AronCao49/MoPA.

Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing. Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals significantly exacerbates the challenge of exploration. The applicability of such systems is thus typically restricted to simulated or heavily engineered environments since agent exploration in the real-world without the guidance of explicit state estimation and dense rewards can lead to unsafe behavior and safety faults that are catastrophic. In this study, we isolate the root causes behind these limitations to develop a system, called MoDem-V2, capable of learning contact-rich manipulation directly in the uninstrumented real world. Building on the latest algorithmic advancements in model-based reinforcement learning (MBRL), demo-bootstrapping, and effective exploration, MoDem-V2 can acquire contact-rich dexterous manipulation skills directly in the real world. We identify key ingredients for leveraging demonstrations in model learning while respecting real-world safety considerations – exploration centering, agency handover, and actor-critic ensembles. We empirically demonstrate the contribution of these ingredients in four complex visuo-motor manipulation problems in both simulation and the real world. To the best of our knowledge, our work presents the first successful system for demonstration-augmented visual MBRL trained directly in the real world. Visit sites.google.com/view/modem-v2 for videos and more details.

Traffic congestion is a persistent problem in our society. Previous methods for traffic control have proven futile in alleviating current congestion levels leading researchers to explore ideas with robot vehicles given the increased emergence of vehicles with different levels of autonomy on our roads. This gives rise to mixed traffic control, where robot vehicles regulate human-driven vehicles through reinforcement learning (RL). However, most existing studies use precise observations that require domain expertise and hand engineering for each road network’s observation space. Additionally, precise observations use global information, such as environment outflow, and local information, i.e., vehicle positions and velocities. Obtaining this information requires updating existing road infrastructure with vast sensor environments and communication to potentially unwilling human drivers. We consider image observations, a modality that has not been extensively explored for mixed traffic control via RL, as the alternative: 1) images do not require a complete re-imagination of the observation space from environment to environment; 2) images are ubiquitous through satellite imagery, in-car camera systems, and traffic monitoring systems; and 3) images only require communication to equipment. In this work, we show robot vehicles using image observations can achieve competitive performance to using precise information on environments, including ring, figure eight, intersection, merge, and bottleneck. In certain scenarios, our approach even outperforms using precision observations, e.g., up to 8% increase in average vehicle velocity in the merge environment, despite only using local traffic information as opposed to global traffic information.

The utilization of vector-based deep learning techniques has great prospects in the realm of autonomous driving, particularly in the domains of prediction and planning tasks. However, the application of vector-based backbones for prediction and planning tasks may lead to the occurrence of causal confusion. Previous studies have explored the phenomenon of causal confusion, with a specific emphasis on the context of visual imitation learning. As for the vector-based model, we observe that the states of surrounding vehicles can be a nuisance shortcut. In our work, an off-policy approach is proposed to alleviate the issue by incorporating de-confounding supervision. Additionally, to better capture the environmental cues, such as route and traffic lights, in vectorized representation, a decoder utilizing iterative route fusion is devised. By incorporating auxiliary supervision and employing a dedicated decoder, we demonstrate the effectiveness of our methods in reducing causal confusion and improving performance in planning tasks through reactive and nonreactive closed-loop simulations on the nuPlan dataset.

Persistent operation of Autonomous Underwater Vehicles (AUVs) without manual interruption for recharging saves time and total cost for offshore monitoring and data collection applications. In order to facilitate AUVs for long mission durations without ship support, they can be equipped with docking capabilities to recharge in situ at Wave Energy Converter (WEC) with dock recharging stations. However, the power generated at the recharging stations may be constrained depending on the sea conditions. Therefore, a robust mission planning framework is proposed using a centralized Evolutionary Algorithm (EA) and a decentralized Monte Carlo Tree Search (MCTS) method. Both methods incorporate the charge availability constraint at the recharging station in addition to the maximum charge capacity of each AUV. The planner utilizes a time-varying power profile of irregular waves incident at WECs for dock charging and generates efficient mission plans for AUVs by optimizing their time to visit the dock based on the imposed constraint. The effects of increasing the number of AUVs, increasing the number of points of interest in the mission area, and varying sea state on the mission duration are also analyzed.

Flying microrobots have garnered growing research interest owing to their technological intricacies and suitability for various applications leveraging miniaturized size. Electrohydrodynamic (EHD) thrust offers advantages by generating propulsion without moving parts, but real-world use is limited by insufficient thrust generation, manufacturing challenges, fragility, and cost. This work presents the design and development of an optimized ion-propelled flying microrobot that excels in low weight, high thrust-to-weight ratio, and cost efficiency. Regarding design, multiphysics simulations guided structural optimization to increase thrust while decreasing weight. For materials, metal-coated polyethylene terephthalate (PET) film was selected to leverage the combined merits of metal conductivity and polymer flexibility, light weight, and low cost, enabling further weight reduction, easy assembly, robustness, and cost-effectiveness. Various experiments, including voltage-current measurements, ionic wind speed, thrust quantification, and airflow visualization, directed design refinements and validated performance. Through structural optimization, the maximum wind speed attained 2.25 m/s. Flight demonstrations with payloads evidenced the microrobot can stably fly at an inherent 16 mg weight while carrying an additional 72 mg load, achieving a record 5.5 thrust-to-weight ratio. These results open possibilities to incorporate microelectronics, enabling autonomous flight functionality.

Microexpressions are expressions that people inadvertently express, and therefore often represent a person’s true emotion. However, because it has a low intensity and a short duration, it is hard to be recognized correctly. In this paper, we propose a deep learning magnification method to generate macroexpressions from a single microexpression image. In the first stage, we extract the expression information from a single microexpression image. Then, We combine the idea of cyclegan and optical flow consistency to model the extracted expression features as the optical flow field between the neutral face and microexpressions. To extract a reliable optical flow field from the expression information, we design an optical flow refiner. In the second stage, we adopt an encoder-decoder network and let it learn to magnify the optical flow. Finally, the magnified optical flow guided the microexpression images to generate macroexpression images. We compare our single input based network with current two-frames-input based networks. The results show that our method performs better, even in wild images. We fed our magnified images directly into a simple ResNet18 network for recognition, achieving a competitive score under the MEGC2019 standard, compared with recent complex recognition networks.

Massage therapy is helpful for the rehabilitation of various diseases, such as headaches caused by migraines and stress. Existing robotic systems have focused on massage therapy on the torso and limbs, but performing massage motions through suitable actuation on a person’s head has been a challenge. In this paper, we present MiBOT, a head-worn massage robot that actuates two soft tactors to produce touch motions mimicking human massage. A key design principle behind MiBOT is its silent actuation, which we achieve through pneumatic artificial muscles in conjunction with a controller loop to respond to contact pressure. We evaluated the effectiveness of MiBOT in a controlled study and assessed subjects’ blood pressure and heart rate levels while applying MiBOT. We found that our mechanical system generated positive and conclusive quantitative outcomes that are similar to the human-administered massage, decreasing participants’ mean systolic and diastolic blood pressure by 2.8 mmHg and 1.7 mmHg, respectively, as well as calming their heart rate by 8–10% on average.

In this work, we address the problem of real-time dense depth estimation from monocular images for mobile underwater vehicles. We formulate a deep learning model that fuses sparse depth measurements from triangulated features to improve the depth predictions and solve the problem of scale ambiguity. To allow prior inputs of arbitrary sparsity, we apply a dense parameterization method. Our model extends recent state-of-the-art approaches to monocular image based depth estimation, using an efficient encoder-decoder backbone and modern lightweight transformer optimization stage to encode global context. The network is trained in a supervised fashion on the forward-looking underwater dataset, FLSea. Evaluation results on this dataset demonstrate significant improvement in depth prediction accuracy by the fusion of the sparse feature priors. In addition, without any retraining, our method achieves similar depth prediction accuracy on a downward looking dataset we collected with a diver operated camera rig, conducting a survey of a coral reef. The method achieves real-time performance, running at 24 FPS on a NVIDIA Jetson Xavier NX, 160 FPS on a NVIDIA RTX 2080 GPU and 7 FPS on a single Intel i9-9900K CPU core, making it suitable for direct deployment on embedded GPU systems. The implementation of this work is made publicly available at https://github.com/ebnerluca/uw_depth.