IROS 2024 论文列表

Realizing bipedal locomotion on humanoid robots with point feet is especially challenging due to their highly underactuated nature, high degrees of freedom, and hybrid dynamics resulting from impacts. With the goal of addressing this challenging problem, this paper develops a control framework for realizing dynamic locomotion and implements it on a novel point foot humanoid: ADAM. To this end, we close the loop between Hybrid Zero Dynamics (HZD) and Hybrid linear inverted pendulum (HLIP) based step length regulation. To leverage the full-order hybrid dynamics of the robot, walking gaits are first generated offline by utilizing HZD. These trajectories are stabilized online through the use of a HLIP based regulator. Finally, the planned trajectories are mapped into the full-order system using a task space controller incorporating inverse kinematics. The proposed method is verified through numerical simulations and hardware experiments on the humanoid robot ADAM marking the first humanoid point foot walking. Moreover, we experimentally demonstrate the robustness of the realized walking via the ability to track a desired reference speed, robustness to pushes, and locomotion on uneven terrain.

Automation of hydraulic material handling machinery is currently limited to semi-static pick-and-place cycles. Dynamic throwing motions which utilize the passive joints, can greatly improve time efficiency as well as increase the dumping workspace. In this work, we use Reinforcement Learning (RL) to design dynamic controllers for material handlers with under-actuated arms as commonly used in logistics. The controllers are tested both in simulation and in real-world experiments on a 12-ton test platform. The method is able to exploit the passive joints of the gripper to perform dynamic throwing motions. With the proposed controllers, the machine is able to throw individual objects to targets outside the static reachability zone with good accuracy for its practical applications. The work demonstrates the possibility of using RL to perform highly dynamic tasks with heavy machinery, suggesting a potential for improving the efficiency and precision of autonomous material handling tasks.

Underwater images suffer from color distortion, haze, and poor visibility due to light refraction and absorption in water. These challenges significantly impact the utilization of Autonomous Underwater Vehicles (AUVs) or marine robots. Typically, color and brightness distortions manifest at lower frequencies, while edge and texture distortions are prevalent at higher frequencies. Traditional methods struggle to concurrently rectify these mixed distortions as they primarily concentrate on the spatial domain. To address these issues, we introduce the Dynamic SpectraFormer, which enhances under-water images through a frequency domain transformer. The Dynamic SpectraFormer introduces an ultra-high-resolution sparse spectrum attention module, which could capture the long-term dependency without losing the universal approximating power. Additionally, we have developed a dynamic spectrum weight generation layer that serves as an adaptive spectrum band selector, accentuating critical frequency bands and suppressing less relevant ones. Consequently, this method significantly improves underwater image quality by addressing both high-and low-frequency distortions. Our extensive ablation studies and comparative evaluations consolidate the Dynamic SpectraFormer’s efficacy across multiple underwater image enhancement benchmarks. The source code is available at https://github.com/arifence2024/DynamicSpectraFormer.git.

Path planning is crucial in numerous robotic applications. Reconfigurable robots possess the capability to alter their shape, enabling access to confined spaces, a task challenging for fixed-shape robots. However, existing path planners for reconfigurable robots are typically designed with predefined motion patterns for reconfiguration, lacking adaptation to space availability and executing reconfiguration only when the robot is static. This reliance on predefined patterns limits the potential of reconfigurable robots to navigate through confined spaces. This paper proposes a novel path-planning approach based on dynamic reconfiguration to address this limitation. The proposed method employs two nested A* algorithms modified to handle reconfiguration and efficient search, termed Dynamic Reconfiguration integrated Nested A* (DRiNA*). Experimental results demonstrate the proposed method’s ability to find feasible paths for robot navigation using dynamic reconfigurations in confined spaces, surpassing the capabilities of existing path planners. The scalability of the proposed method to reconfigurable robots with varying numbers of blocks is also confirmed. Additionally, DRiNA* significantly reduces energy consumption compared to existing path planners.

This paper presents a framework for dynamic object catching using a quadruped robot’s front legs while it stands on its rear legs. The system integrates computer vision, trajectory prediction, and leg control to enable the quadruped to visually detect, track, and successfully catch a thrown object using an onboard camera. Leveraging a fine-tuned YOLOv8 model for object detection and a regression-based trajectory prediction module, the quadruped adapts its front leg positions iteratively to anticipate and intercept the object. The catching maneuver involves identifying the optimal catching position, controlling the front legs with Cartesian PD control, and closing the legs together at the right moment. We propose and validate three different methods for selecting the optimal catching position: 1) intersecting the predicted trajectory with a vertical plane, 2) selecting the point on the predicted trajectory with the minimal distance to the center of the robot’s legs in their nominal position, and 3) selecting the point on the predicted trajectory with the highest likelihood on a Gaussian Mixture Model (GMM) modelling the robot’s reachable space. Experimental results demonstrate robust catching capabilities across various scenarios, with the GMM method achieving the best performance, leading to an 80% catching success rate.

Dynamic model is essential for robust and reliable robotic fish motion control. Despite considerable efforts in robotic fish dynamic modeling, background flow has not been well considered yet, leading to the deterioration of applying robotic fish to practice. In this paper, we propose a novel dynamic model, termed Flow-Aware Robotic fish Model (FARM), that with well consideration to background flow using Koopman operators without increasing computation complexity. Specifically, we first collect motion data of the robotic fish in different background flow fields, and then obtain a linear approximation (the dynamic model) of nonlinear dynamics through carefully selected lifted functions. The obtained model can predict the next state based on the current state, control input, and average flow velocity of the local flow field. We evaluate the effectiveness of obtained model by comparing the Root Mean Square Error (RMSE) of predicted motion trajectories with real trajectories in various flow field environments. The results indicate that FARM is highly promising for obtaining a reliable dynamic model and can achieve comparable prediction accuracy even in unseen flow field environments with rough flow maps.

The development of new measurement systems for mobile robots has been a growing interest in recent years, particularly tactile systems based on bioinspired sensor antennae, such as whiskers and antennae found in animals and insects. This work focuses on studying a mobile robot equipped with such systems. Specifically, a dynamic model is developed for an omnidirectional robot with a sensing antenna, considering the planar motion of the system and taking into account the gravity effect. The extended Hamilton principle is applied to derive the equations of motion for the mobile platform, while the boundary-value problem is formulated for the antenna. Subsequently, modal analysis is employed to obtain a unique solution for the sensor antenna model, which is then validated using experimental data.

Imitation Learning (IL) is a promising paradigm for learning dynamic manipulation of deformable objects since it does not depend on difficult-to-create accurate simulations of such objects. However, the translation of motions demonstrated by a human to a robot is a challenge for IL, due to differences in the embodiments and the robot’s physical limits. These limits are especially relevant in dynamic manipulation where high velocities and accelerations are typical. To address this problem, we propose a framework that first maps a dynamic demonstration into a motion that respects the robot’s constraints using a constrained Dynamic Movement Primitive. Second, the resulting object state is further optimized by quasi-static refinement motions to optimize task performance metrics. This allows both efficiently altering the object state by dynamic motions and stable small-scale refinements. We evaluate the framework in the challenging task of bag opening, designing the system BILBO: Bimanual dynamic manipulation using Imitation Learning for Bag Opening. Our results show that BILBO can successfully open a wide range of crumpled bags, using a demonstration with a single bag. See supplementary material at https://sites.google.com/view/bilbo-bag.

As a medical microrobot, magnetic-controlled capsule robots (MCRs) are pivotal in internal diagnostics and therapeutic interventions. Achieving high-precision localization of MCRs is essential for the successful execution of medical procedures. This paper introduces a novel Dung Beetle Optimizer (DBO)-based localization method for MCR, demonstrating high localization accuracy and flexibility in static magnetic field environments and under the control of existing magnetic control systems. With the aid of an FPGA-based parallel measurement system, it can effectively eliminate measurement distortion. The average position and orientation errors could achieve 0.53 mm and 0.60° when performing 600 iterations per computation, and further increasing the number of iterations reduces the errors, which is superior to existing methods. Experimental validations underscore the method’s robust performance and compatibility with existing magnetic control systems.

We proposed a novel dual-process optimization approach for parts collection order and route planning in parts warehouses. Conventional multi-agent parts collection typically uses the vehicle routing problem (VRP), which focuses on minimizing the number of agents and costs. However, the model does not fully leverage the vehicle’s potential. Moreover, multi-agent path finding (MAPF) focuses on route planning and avoiding path conflicts, ignoring the order of part collection. The proposed approach integrates algorithms from the traveling salesman problem (TSP) and path planning, and modifies them to suit the dynamic and complex environment of parts warehouses. This integration streamlines the collection process and considerably reduces the operational time. Thus, the study can improve automation and efficiency in parts warehouse management and improve optimization techniques. The proposed method achieved more than tenfold acceleration compared with the ideal centralized optimization, without cost increments. As the number of agents and part collections increases, centralized optimization requires a metaheuristic approach, which results in solution degradation. However, the proposed approach maintains over tenfold acceleration and produces solutions with shorter operational times. Furthermore, we conducted an ablation study comparing six methods, from entirely independent to centralized optimization, demonstrating that the proposed approach effectively balances computational time and solution accuracy.

Human Mesh Reconstruction (HMR) from monocular video plays an important role in human-robot interaction and collaboration. However, existing video-based human mesh reconstruction methods face a trade-off between accurate reconstruction and smooth motion. These methods design networks based on either RNNs or attention mechanisms to extract local temporal correlations or global temporal dependencies, but the lack of complementary long-term information and local details limits their performance. To address this problem, we propose a Dual-branch Graph Transformer network for 3D human mesh Reconstruction from video, named DGTR. DGTR employs a dual-branch network including a Global Motion Attention (GMA) branch and a Local Details Refine (LDR) branch to par-allelly extract long-term dependencies and local crucial information, helping model global human motion and local human details (e.g., local motion, tiny movement). Specifically, GMA utilizes a global transformer to model long-term human motion. LDR combines modulated graph convolutional networks and the transformer framework to aggregate local information in adjacent frames and extract crucial information of human details. Experiments demonstrate that our DGTR outperforms state-of-the-art video-based methods in reconstruction accuracy and maintains competitive motion smoothness. Moreover, DGTR utilizes fewer parameters and FLOPs, which validate the effectiveness and efficiency of the proposed DGTR. Code is publicly available at https://github.com/TangTao-PKU/DGTR.

Recognising and tracking human actions from videos is crucial for human-robot collaborative assembly (HRCA). However, traditional action segmentation methods suffer from limited scene adaptability, partly because they conceptualise actions as unified verb-object entities with complete semantics. To overcome this, we propose a compositional action segmentation method. Following the human-robot shared assembly taxonomy, we deconstruct an assembly action into four elements: action verb, manipulated object, target object and tool. Our approach employs individual segmentation models for each action element, and then integrates general knowledge from large language models and domain-specific knowledge from predefined rules to form semantic-complete actions. Our method’s emphasis on general action elements and a modular design endows it with greater flexibility and adaptability than traditional approaches. Another attribute of our method is its capability to segment actions of each hand concurrently, facilitating more nuanced HRCA. Comparative experiments validate the superiority of our method over traditional action segmentation methods. More details can be found at https://github.com/LISMS-AKL-NZ/DuCAS.

Reliable deployment of Unmanned Aerial Vehicles (UAVs) in cluttered unknown environments requires accurate sensors for Global Navigation Satellite System (GNSS)-denied localization and obstacle avoidance. Such a requirement limits the usage of cheap and micro-scale vehicles with constrained payload capacity if industrial-grade reliability and precision are required. This paper investigates the possibility of offloading the necessity to carry heavy sensors to another member of the UAV team while preserving the desired capability of the smaller robot intended for exploring narrow passages. A novel cooperative guidance framework offloading the sensing requirements from a minimalistic secondary UAV to a superior primary UAV is proposed. The primary UAV constructs a dense occupancy map of the environment and plans collision-free paths for both UAVs to ensure reaching the desired secondary UAV’s goals even in areas not accessible by the bigger robot. The primary UAV guides the secondary UAV to follow the planned path while tracking the UAV using Light Detection and Ranging (LiDAR)-based relative localization. The proposed approach was verified in real-world experiments with a heterogeneous team of a 3D LiDAR-equipped primary UAV and a micro-scale camera-equipped secondary UAV moving autonomously through unknown cluttered GNSS-denied environments with the proposed framework running fully on board the UAVs.

This paper proposes DriViDOC: a framework for Driving from Vision through Differentiable Optimal Control, and its application to learn autonomous driving controllers from human demonstrations. DriViDOC combines the automatic inference of relevant features from camera frames with the properties of nonlinear model predictive control (NMPC), such as constraint satisfaction. Our approach leverages the differentiability of parametric NMPC, allowing for end-to-end learning of the driving model from images to control. The model is trained on an offline dataset comprising various human demonstrations collected on a motion-base driving simulator. During online testing, the model demonstrates successful imitation of different driving styles, and the interpreted NMPC parameters provide insights into the achievement of specific driving behaviors. Our experimental results show that DriViDOC outperforms other methods involving NMPC and neural networks, exhibiting an average improvement of 20% in imitation scores.

Recently, LLM-powered driver agents have demonstrated considerable potential in the field of autonomous driving, showcasing human-like reasoning and decision-making abilities. However, current research on aligning driver agent behaviors with human driving styles remains limited, partly due to the scarcity of high-quality natural language data from human driving behaviors. To address this research gap, we propose a multi-alignment framework designed to align driver agents with human driving styles through demonstrations and feedback. Notably, we construct a natural language dataset of human driver behaviors through naturalistic driving experiments and post-driving interviews, offering high-quality human demonstrations for LLM alignment. The framework’s effectiveness is validated through simulation experiments in the CARLA urban traffic simulator and further corroborated by human evaluations. Our research offers valuable insights into designing driving agents with diverse driving styles. The implementation of the framework 1 and details of the dataset 2 can be found at the link.

Animatronic robots hold the promise of enabling natural human-robot interaction through lifelike facial expressions. However, generating realistic, speech-synchronized robot expressions poses significant challenges due to the complexities of facial biomechanics and the need for responsive motion synthesis. This paper introduces a novel, skinning-centric approach to drive animatronic robot facial expressions from speech input. At its core, the proposed approach employs linear blend skinning (LBS) as a unifying representation, guiding innovations in both embodiment design and motion synthesis. LBS informs the actuation topology, facilitates human expression retargeting, and enables efficient speech-driven facial motion generation. This approach demonstrates the capability to produce highly realistic facial expressions on an animatronic face in real-time at over 4000 fps on a single Nvidia RTX 4090, significantly advancing robots' ability to replicate nuanced human expressions for natural interaction. To foster further research and development in this field, the code has been made publicly available at: https://github.com/library87/OpenRoboExp.

Recent advancements in foundation models (FMs) have unlocked new prospects in autonomous driving, yet the experimental settings of these studies are preliminary, oversimplified, and fail to capture the complexity of real-world driving scenarios in human environments. It remains under-explored whether FM agents can handle long-horizon navigation tasks with free-from dialogue and deal with unexpected situations caused by environmental dynamics or task changes. To explore the capabilities and boundaries of FMs faced with the challenges above, we introduce DriVLMe, a video-language-model-based agent to facilitate natural and effective communication between humans and autonomous vehicles that perceive the environment and navigate. We develop DriVLMe from both embodied experiences in a simulated environment and social experiences from real human dialogue. While DriVLMe demonstrates competitive performance in both open-loop benchmarks and closed-loop human studies, we reveal several limitations and challenges, including unacceptable inference time, imbalanced training data, limited visual understanding, challenges with multi-turn interactions, simplified language generation from robotic experiences, and difficulties in handling on-the-fly unexpected situations like environmental dynamics and task changes. Nevertheless, DriVLMe offers a promising new direction for autonomous driving agents that need to navigate not just complex environments but also complex social interactions.

The sim-to-real gap, a long-standing challenge in the field of robotics, has garnered significant attention. Essentially, it is important to learn robust representation models that can be seamlessly applied in both simulation and real world. Traditional approaches like domain randomization have demonstrated success in zero-short setting, by creating representations that are resilient and adaptable through the augmentation of diversity within simulations. However, they suffer from the need for extensive training across a range of parameter variances, and dependency on heuristic approaches. In this work, we present a novel reinforcement learning architecture named Soft Attention-Augmented Actor-Critic (Soft3AC) for sim-to-real robotic tasks without the need for heuristic domain randomization. Our approach achieves the learning of semantically task-relevant feature representations that exhibit resilience against appearance gaps. This is realized by employing an architectural design that separates current perceptions from historical perceptions in memory, fostering abstract spatial-temporal understanding. Simultaneously, the introduction of an attention mechanism enables a more contextual processing. We validated our method through conducting a valve rotation task with a robotic hand, under both sim-to-sim and sim-to-real conditions. The results indicate that our model adeptly bridges the appearance gap observed in sim-to-sim and sim-to-real transfers. Our method demonstrated its ability to be deployed directly into the real world in a domain randomization free zero-shot manner.

Generalizing visual reinforcement learning is fundamental to robot visual navigation, involving the acquisition of a policy from interactions with source environments to facilitate adaptation to analogous, yet unfamiliar target environments. Recent advancements capitalize on data augmentation techniques, self-supervised learning methods, and the generative adversarial network framework to train policy neural networks with enhanced generalizability. However, current methods, upon extracting domain-general latent features, further utilize these features to train the reinforcement learning policy, resulting in a decline in the performance of the learned policy guiding the agent to accomplish tasks. To tackle these challenges, a framework of self-expert imitation with purifying latent features was devised, empowering the policy to achieve robust and stable zero-shot generalization performance in visually similar domains previously unseen, without diminishing the performance of guiding the agent to accomplish tasks. The extraction method of domain-general latent features is proposed to enhance their quality based on the variational autoencoder. Extensive experiments have shown that our policy, compared with state-of-the-art counterparts, does not diminish the performance of the policy guiding the agent to accomplish tasks after generalization.