ICRA 2024 Papers

Due to the lack of enough real multi-agent data and time-consuming of labeling, existing multi-agent cooperative perception algorithms usually select the simulated sensor data for training and validating. However, the perception performance is degraded when these simulation-trained models are deployed to the real world, due to the significant domain gap between the simulated and real data. In this paper, we propose the first Simulation-to-Reality transfer learning framework for multi-agent cooperative perception using a novel Vision Transformer, named as S2R-ViT, which considers both the Deployment Gap and Feature Gap between simulated and real data. We investigate the effects of these two types of domain gaps and propose a novel uncertainty-aware vision transformer to effectively relief the Deployment Gap and an agent-based feature adaptation module with inter-agent and ego-agent discriminators to reduce the Feature Gap. Our intensive experiments on the public multi-agent cooperative perception datasets OPV2V and V2V4Real demonstrate that the proposed S2R-ViT can effectively bridge the gap from simulation to reality and outperform other methods significantly for point cloud-based 3D object detection.

Robots routinely encounter obstacles and rough terrain, but terrain curvature is seldom included in models for real-time algorithms. We present a closed-form dynamic model for rolling with two planar smooth curves, and apply it to sagittal-plane locomotion problems. We assumed that the body rolls without slip and maintains a single point of contact. Using an auxiliary coordinate system to define the rolling body and terrain as parametric curves, we derived rolling constraints and dynamic equations of motion for model-based control algorithms – specifically Operational Space Control. The formulation was used to simulate an arbitrarily curved rock rolling on undulating terrain and to generate control signals to stabilize it on parabolic terrain. The stabilization problem was solved as a quadratic program in < 3 ms which shows that our formulation is suitable for real-time control algorithms. We also applied this framework to dynamically balance an underactuated 2 degree-of-freedom leg on parabolic terrain and achieve prescribed locomotion tasks for a wheel-leg vehicle on sinusoidal terrain in simulation. A supplementary video is available at https://youtu.be/EtPQEzkqsK8.

Autonomous assistance of people with motor impairments is one of the most promising applications of autonomous robotic systems. Recent studies have reported encouraging results using deep reinforcement learning (RL) in the healthcare domain. Previous studies showed that assistive tasks can be formulated as multi-agent RL, wherein there are two agents: a caregiver and a care-receiver. However, policies trained in multi-agent RL are often sensitive to the policies of other agents. In such a case, a trained caregiver’s policy may not work for different care-receivers. To alleviate this issue, we propose a framework that learns a robust caregiver’s policy by training it for diverse care-receiver responses. In our framework, diverse care-receiver responses are autonomously learned through trials and errors. In addition, to robustify the care-giver’s policy, we propose a strategy for sampling a care-receiver’s response in an adversarial manner during the training. We evaluated the proposed method using tasks in an Assistive Gym. We demonstrate that policies trained with a popular deep RL method are vulnerable to changes in policies of other agents and that the proposed framework improves the robustness against such changes.

This paper presents a robustified time-optimal motion planning approach for navigating an Autonomous Mobile Robot (AMR) from an initial state to a terminal state without colliding with obstacles, even when subjected to disturbances, which are modeled as random process noise and measurement noise. The approach iteratively solves the robustified problem by incorporating updated state-dependent safety margins for collision avoidance, the evolution of which is derived separately from the robustified problem. Additionally, a strategy for selecting an alternative terminal state to reach is introduced, which comes into play when the desired terminal state becomes infeasible considering the disturbances. Both of these contributions are integrated into a robustified motion planning and control pipeline, the efficacy of which is validated through simulation experiments.

Remote center of motion (RCM) describes a robot with a rod-like end-effector operating through a hole in the interface separating the internal space from the external space. Considering that the control of RCM may be influenced by perturbations (noises) and that the end-effector is frequently replaced to complete different tasks, the structural information related to the robot manipulator and its rod-like end-effector may contain errors. This paper proposes an acceleration-level remote center of cyclic motion (ARC2M) control scheme, which takes into account the cyclic motion index and the physical limitations of robot manipulators to achieve repetitive motion planning and RCM control at the acceleration level. Additionally, a parameter calculation method is proposed to compute unknown parameters of the end-effector under the influence of noise. Kalman filter and a neural dynamics-based method are employed to address noises effects, and related theoretical analyses are given. To validate the proposed ARC2M scheme, simulations and physical experiments are carried out. The source code is available at https://github.com/LongJin-lab/ARCM.

Aerial manipulation has received increasing research interest with wide applications of drones. To perform specific tasks, robotic arms with various mechanical structures will be mounted on the drone. It results in sudden disturbances to the aerial manipulator when switching the robotic arm or interacting with the environment. Hence, it is challenging to design a generic and robust control strategy adapted to various robotic arms when achieving multi-task aerial manipulation. In this paper, we present a learning-based control algorithm that allows online trajectory optimization and tracking to accomplish various aerial interaction tasks without manual adjustment. The proposed energy-saved trajectory planning approach integrates coupled dynamics model with a single rigid body to generate the energy-efficient trajectory for the aerial manipulator. Addressing the challenges of precise control when performing aerial manipulation tasks, this paper presents a controller based on deep neural networks that classifies and learns accurate forces and moments caused by different robotic arms and interactions. Moreover, the forces arising from robotic arm motions are delicately used as part of the drone’s power to save energy. Extensive real-world experiments demonstrate that the proposed method can adapt to various robotic arms and interactions when performing multi-task aerial manipulation.

In recent years, the need for robots to transition from isolated industrial tasks to shared environments, including human-robot collaboration and teleoperation, has become increasingly evident. Building on the foundation of Fractal Impedance Control (FIC) introduced in our previous work, this paper presents a novel extension to dualarm tele-cooperation, leveraging the non-linear stiffness and passivity of FIC to adapt to diverse cooperative scenarios. Unlike traditional impedance controllers, our approach ensures stability without relying on energy tanks, as demonstrated in our prior research. In this paper, we further extend the FIC framework to bimanual operations, allowing for stable and smooth switching between different dynamic tasks without gain tuning. We also introduce a telemanipulation architecture that offers higher transparency and dexterity, addressing the challenges of signal latency and low-bandwidth communication. Through extensive experiments, we validate the robustness of our method and the results confirm the advantages of the FIC approach over traditional impedance controllers, showcasing its potential for applications in planetary exploration and other scenarios requiring dexterous telemanipulation. This paper’s contributions include the seamless integration of FIC into multi-arm systems, the ability to perform robust interactions in highly variable environments, and the provision of a comprehensive comparison with competing approaches, thereby significantly enhancing the robustness and adaptability of robotic systems.

Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. These errors result in poor localization of robotic manipulators and create a significant challenge for applications that rely on precise interactions between manipulators and the environment. In this work, we estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models. We apply our proposed approach in both a structured environment as well as an unstructured environment and measure to demonstrate the efficacy of our methods.

The robustness of legged locomotion is crucial for quadrupedal robots in challenging terrains. Recently, Reinforcement Learning (RL) has shown promising results in legged locomotion and various methods try to integrate privileged distillation, scene modeling, and external sensors to improve the generalization and robustness of locomotion policies. However, these methods are hard to handle uncertain scenarios such as abrupt terrain changes or unexpected external forces. In this paper, we consider a novel risk-sensitive perspective to enhance the robustness of legged locomotion. Specifically, we employ a distributional value function learned by quantile regression to model the aleatoric uncertainty of environments, and perform risk-averse policy learning by optimizing the worst-case scenarios via a risk distortion measure. Extensive experiments in both simulation environments and a real Aliengo robot demonstrate that our method is efficient in handling various external disturbances, and the resulting policy exhibits improved robustness in harsh and uncertain situations in legged locomotion.

Multi-robot global localization (MR-GL) with unknown initial positions in a large scale environment is a challenging task. The key point is the data association between different robots’ viewpoints. It also makes traditional Appearance-based localization methods unusable. Recently, researchers have utilized the object’s semantic invariance to generate a semantic graph to address this issue. However, previous works lack robustness and are sensitive to overlap rate of maps, resulting in unpredictable performance in real-world environments. In this paper, we propose a data association algorithm based on neighbor constraints to improve the robustness of the system. We demonstrate the effectiveness of our method on three different datasets, indicating a significant improvement in robustness compared to previous works.

In autonomous robot navigation, the trajectories from path planners are considered to be safe regions, and deviations could endanger vessels. Model Predictive Control (MPC) stands as a popular choice for trajectory tracking problems as it naturally addresses operational constraints, such as dynamics and control constraints. Nevertheless, achieving robustness in changing environments like oceans and rivers, which are constantly subject to significant external disturbances, remains an ongoing challenge for MPC. It must consistently keep the system within a predefined safe region (such as a reference trajectory) even in the presence of model inaccuracies and perturbations. To address this challenge, we present a robust model predictive control strategy utilizing Control Barrier Functions (CBFs), which increases the disturbance-rejection abilities. We verify our method on an autonomous surface vessel in simulation and natural waters, both with external disturbances. Specifically, compared with the traditional MPC method, our proposed MPC-CBF strategy reduces tracking errors by 17.82% and 40.26% in simulations and field experiments, respectively. Although the control effort slightly increases by 7.78% and 4.20%, respectively, these results clearly demonstrate the enhanced resilience of MPC-CBF to disturbances.

In environments like offices, the duration of a robot’s navigation between two locations may vary over time. For instance, reaching a kitchen may take more time during lunchtime since the corridors are crowded with people heading the same way. In this work, we address the problem of routing in such environments with tasks expressed in Metric Interval Temporal Logic (MITL) – a rich robot task specification language that allows us to capture explicit time requirements. Our objective is to find a strategy that maximizes the temporal robustness of the robot’s MITL task. As the first step towards a solution, we define a Mixed-integer linear programming approach to solving the task planning problem over a Varying Weighted Transition System, where navigation durations are deterministic but vary depending on the time of day. Then, we apply this planner to optimize for MITL temporal robustness in Markov Decision Processes, where the navigation durations between physical locations are uncertain, but the time-dependent distribution over possible delays is known. Finally, we develop a receding horizon planner for Markov Decision Processes that preserves guarantees over MITL temporal robustness. We show the scalability of our planning algorithms in simulations of robotic tasks.

Indoor wireless ranging localization is a promising approach for low-power and high-accuracy localization of wearable devices. A primary challenge in this domain stems from non-line of sight propagation of radio waves. This study tackles a fundamental issue in wireless ranging: the unpredictability of real-time multipath determination, especially in challenging conditions such as when there is no direct line of sight. We achieve this by fusing range measurements with inertial measurements obtained from a low cost Inertial Measurement Unit (IMU). For this purpose, we introduce a novel asymmetric noise model crafted specifically for non-Gaussian multipath disturbances. Additionally, we present a novel Levenberg-Marquardt (LM)-family trust-region adaptation of the iSAM2 fusion algorithm, which is optimized for robust performance for our ranging-IMU fusion problem. We evaluate our solution in a densely occupied real office environment. Our proposed solution can achieve temporally consistent localization with an average absolute accuracy of ∼0.3m in real-world settings. Furthermore, our results indicate that we can achieve comparable accuracy even with infrequent range measurements down to 1Hz.

We present in-hand manipulation tasks where a robot moves an object in grasp, maintains its external contact mode with the environment, and adjusts its in-hand pose simultaneously. The proposed manipulation task leads to complex contact interactions which can be very susceptible to uncertainties in kinematic and physical parameters. Therefore, we propose a robust in-hand manipulation method, which consists of two parts. First, an in-gripper mechanics model that computes a naïve motion cone assuming all parameters are precise. Then, a robust planning method refines the motion cone to maintain desired contact mode regardless of parametric errors. Real-world experiments were conducted to illustrate the accuracy of the mechanics model and the effectiveness of the robust planning framework in the presence of kinematics parameter errors.

Quadrotors may crash and cause severe accidents under instantaneously drastic disturbances. To mitigate the effect of such disturbances, these critical issues should be considered: efficient disturbance observation and compensation, full attitude controllability, and instant output power generation of the quadrotor. In this paper, to keep the quadrotor stable even under suddenly drastic disturbances, a novel control framework is presented to by integrating the advantages of active disturbance rejection control (ADRC) as well as geometric control for a quadrotor with bidirectional thrust capabilities. Moreover, to strengthen the adaptability under significant disturbances, a novel switching strategy is introduced into the control framework by virtue of the quadrotor’s bidirectional thrust capabilities. The ADRC scheme is performed when the disturbances are within a range; alternatively, if the disturbances surpass the preset range and the desired control is beyond the ultimate output of the quadrotor, the quadrotor compliantly responds by executing a 180° flip reverse flight to handle such drastic disturbances. Numerical and real-world experiments demonstrate that the proposed robust control strategy has superior performance adapts to instantaneously drastic disturbances.

A consistent spatial-temporal coordination across multiple agents is fundamental for collaborative perception, which seeks to improve perception abilities through information exchange among agents. To achieve this spatial-temporal alignment, traditional methods depend on external devices to provide localization and clock signals. However, hardware-generated signals could be vulnerable to noise and potentially malicious attack, jeopardizing the precision of spatial-temporal alignment. Rather than relying on external hardwares, this work proposes a novel approach: aligning by recognizing the inherent geometric patterns within the perceptual data of various agents. Following this spirit, we propose a robust collaborative perception system that operates independently of external localization and clock devices. The key module of our system, FreeAlign, constructs a salient object graph for each agent based on its detected boxes and uses a graph neural network to identify common subgraphs between agents, leading to accurate relative pose and time. We validate FreeAlign on both real-world and simulated datasets. The results show that, the FreeAlign empowered robust collaborative perception system perform comparably to systems relying on precise localization and clock devices. {\mathbf{Code}}{\mathbf{Code}} will be released.

Collaborative perception facilitates a more comprehensive representation of the environment by leveraging complementary information shared among various agents and sensors. However, practical applications often encounter information disturbance which includes perception packet loss and time delays, and a comprehensive framework that can simultaneously address such issues is absent. In addition, the feature extraction process prior to fusion is not sufficient, as it lacks exploration of the local semantics and context dependencies of individual features. To enhance both accuracy and robustness, this paper introduces a novel framework named Robust Collaborative Perception against Temporal Information Disturbance, which predicts perception information when disturbance occurs. Specifically, the Historical Frame Prediction (HFP) module is introduced to make compensation for information loss with temporal association excavation of historical features. Based on the predicted features generated by the HFP module, the Pyramid Attention Integration (PAI) module is introduced to augment local semantics and incorporate global long-range dependencies through multi-scale window attention. Compared with existing methods on the publicly available dataset OPV2V, our approach exhibits superior performance and expanded robustness in the 3D object detection task. The code will be publicly available at https://github.com/hexunjie/Ro-temd.

Optimal behaviours of a system to perform a specific task can be achieved by leveraging the coupling between trajectory optimization, stabilization, and design optimization. This approach is particularly advantageous for underactuated systems, which are systems that have fewer actuators than degrees of freedom and thus require for more elaborate control systems. This paper proposes a novel co-design algorithm, namely Robust Trajectory Control with Design optimization (RTC-D). An inner optimization layer (RTC) simultaneously performs direct transcription (DIRTRAN) to find a nominal trajectory while computing optimal hyperparameters for a stabilizing time-varying linear quadratic regulator (TVLQR). RTC-D augments RTC with a design optimization layer, maximizing the system’s robustness through a time-varying Lyapunov-based region of attraction (ROA) analysis. This analysis provides a formal guarantee of stability for a set of off-nominal states. The proposed algorithm has been tested on two different underactuated systems: the torque-limited simple pendulum and the cart-pole. Extensive simulations of off-nominal initial conditions demonstrate improved robustness, while real-system experiments show increased insensitivity to torque disturbances.

This paper develops a controller synthesis method for ensuring an admissible bound of external forces on biped robots in a desired level. We first introduce the authors’ preceding results on the norm-based stability criterion for a biped walking constructed on its linear inverted pendulum model (LIPM). More precisely, an induced norm can be taken to formulate the fact that the balance for a biped robot is achieved if its zero moment point (ZMP) always stays in the supporting region at each step. Based on this norm-based criterion, we aim at making the maximum energy of external forces admissible for balancing the biped robot be a pregiven desired bound γ(> 0). To achieve this objective, a robust controller is designed through the linear matrix inequality (LMI)-based approach. More importantly, a necessary and sufficient condition for the existence of a robust controller leading to the desired bound is characterized by some LMI conditions. The effectiveness of the overall arguments is validated through some comparative simulation results of a biped walking robot with external forces.