IROS 2024 论文列表

Legged robots are emerging, and legged locomotion is in critical need, which requires precise leg-body kinematics to execute control commands or plan motion trajectories. This paper proposes online state estimation to determine legged kinematics of robots with an arbitrary number of legs, which includes the kinematic parameters of the leg-body transformation, time offset and the leg link lengths. In particular, we advocate an in-place dance gait for kinematic determination where the toes remain static on the ground and serve as static landmarks as in SLAM. As a visual-inertial sensor is typically available onboard robot and located at the floating base, we leverage efficient MSCKF-based visual-inertial navigation to estimate legged kinematics. To this end, we analytically derive the legged kinematic measurements and tightly fuse them along with visual-inertial measurements for MSCKF update of both the leg’s kinematics and body’s motion. The proposed method has been extensively validated in both simulations and experiments with different quadrupeds, showing its robustness and accuracy.

While lower-limb exoskeletons have been increasingly used for gait assistance and rehabilitation, most of them continue to function as assistive devices in the exoskeleton-user relationship as a leader and follower. This limits the user’s ability to interactively contribute to gait control. Therefore, this study proposes an interactive user-exoskeleton control strategy to translate the exoskeletons into interactive compliant companion devices with the exoskeleton-user relationship as the collaborator. This strategy is implemented through online adaptive impedance control with gravity compensation (OAIC-GC). It relies solely on internal pose feedback (joint position) rather than external sensors such as electromyography, torque, or force, as utilized in other assist-as-needed (AAN) control methods. The OAIC-GC can automatically capture the mechanical impedance dynamics of the user’s lower limbs during walking and thus facilitate adaptive, versatile, and personalized gait assistance. It is evaluated using a real lower-limb exoskeleton system with six degrees of freedom (DOFs) across different users engaging in various activities. These activities include symmetrical and asymmetrical walking on a split-belt treadmill at different speeds, as well as walking up stairs. The results indicate a significant improvement in the exoskeleton’s performance in terms of adaptability and movement smoothness under all activities when compared to traditional control. The proposed control reduces joint assistance torque across all exoskeleton joints, enhancing user interaction and comfort. This enables users to actively control their gait patterns, enabling the exoskeleton to operate in an interactive assist-as-needed (IAAN) mode.

We represent a vehicle dynamics model for autonomous driving near the limits of handling via a multilayer neural network. Online adaptation is desirable in order to address unseen environments. However, the model needs to adapt to new environments without forgetting previously encountered ones. In this study, we apply Continual-MAML to overcome this difficulty. It enables the model to adapt to the previously encountered environments quickly and efficiently by starting updates from optimized initial parameters. We evaluate the impact of online model adaptation with respect to inference performance and impact on control performance of a model predictive path integral (MPPI) controller using the TRIKart platform. The neural network was pre-trained using driving data collected in our test environment, and experiments for online adaptation were executed on multiple different road conditions not contained in the training data. Empirical results show that the model using Continual-MAML outperforms the fixed model and the model using gradient descent in test set loss and online tracking performance of MPPI.

The task-specific optimization of robotic systems has since the inception of the field been divided into the optimization of the robot and the optimization of the layout of its workstations. In this letter, we argue that these two problems are interdependent and should be treated as such. To this end, we present a unified problem formulation that enables for the simultaneous optimization of both the robot kinematics and the workstation layout. We demonstrate the effectiveness of our approach by jointly optimizing a robotic milling system. To compare our approach to the state of the art, we optimize the robot’s kinematics and layout separately. The results show that our approach outperforms the state of the art and that simultaneous optimization leads to up to eight times better solutions.

In this paper, a kinematically modular approach to robot control is presented. The method involves structures called Elementary Dynamic Actions and a network model combining these elements. With this control framework, a rich repertoire of movements can be generated by combination of basic modules. The problems of solving inverse kinematics, managing kinematic singularity and kinematic redundancy are avoided. The modular approach is robust against contact and physical interaction, which makes it particularly effective for contact-rich manipulation. Each kinematic module can be learned by Imitation Learning, thereby resulting in a modular learning strategy for robot control. The theoretical foundations and their implementation on a real robot are presented. Using a KUKA LBR iiwa robot, three tasks were considered: (1) generating a sequence of discrete movements, (2) generating a combination of discrete and rhythmic movements, and (3) a drawing and erasing task. The results obtained show that this modular approach has the potential to simplify the generation of a diverse range of robot actions.

We study the motion model of a sensor rigidly mounted inside a ball. Due to the rigid placement inside the ball, the geometry of the sensor trajectory resembles a 3D curate trochoid. A new calibration method for spherical systems estimates the extrinsic parameters of the sensor with respect to the balls center of rotation. We deploy the calibration and motion model on our spherical mobile mapping platform to estimate the trajectory of a LiDAR sensor and compare it to trajectories of state-of-the-art LiDAR-Inertial odometry (LIO) methods. The motion model, which is solely based on IMU measurements, produces comparable results to the LIO methods, sometimes even outperforming them in positional accuracy. Although the LIO methods provide better rotational accuracy due to the utilization of LiDAR data, they struggle to reproduce the trochoidal nature of the trajectory and only provide pose estimations at the LiDAR frequency, whereas the motion model produces a more consistent trochoidal trajectory at the much higher IMU frequency. The results demonstrate the difficulty that current LIO methods have on spherical systems and indicate that our motion model is suitable for overcoming these issues.

The paper explores the challenging task of performing a non-prehensile manipulation of several balls synchronously rolling on the curved hands of Butterfly robots. Each Butterfly robot represents a standard benchmark hardware setup, comprising a DC motor rotating a butterfly-shaped frame in a vertical plane, with a ball moving freely upon it, equipped with integrated computer vision, communication, programmable control, and computation interfaces. The combined dynamics of the considered system, consisting of N ≥ 2 such robots, is inherently underactuated, characterized by N active and N passive degrees of freedom, as well as N independent unilateral constraints that model the interactions between the frames and the balls, assuming no slipping. We focus on designing a model-based centralized feedback controller to achieve synchronized rotations of the balls. We assume the accuracy of our mathematical model and the feasibility of implementing a discretized version of the proposed continuous-time controller with a sufficiently small sampling time, that, in particular, is necessary for numerical differentiation. Relying on orbital stability of nominal periodic solution of the closed-loop system, we will experimentally check robustness to various inevitable challenges such as noises, disturbances, uncertainties, and communication delays. Hence, our concentration lies in designing an orbitally stabilizing controller for the underactuated models. The primary contribution is proposing one set of transverse coordinates, enabling transverse-linearization-based controller design, accompanied by pertinent closed-loop system analysis tools, thereby enhancing the efficacy of solving the manipulation task. Analytical and model-based arguments are validated through successful simulations and experiments conducted on two Butterfly robots, thereby emphasizing the validity and practicality of the proposed approach.

The application of microrobots in the biomedical field has attracted great interest, among which drug transportation is one of the application scenarios. Traditional studies used the global magnetic field to control single microrobot, therefore it is impossible to control multiple microrobots. To address this problem, this paper develops a local magnetic field generation system to realize the independent control of multiple microrobots. The proposed multi-microrobot motion system integrates perception, planning, and actuation, enabling autonomous multi-task drug delivery. In our system, we first develop a printed circuit board (PCB) array magnetic driven microrobot system based on a micro coil array, then the Yolov8 framework is employed for the target/environment recognition, accurately identifying microrobots and magnetic fluids, while the Rapidly-exploring Random Trees (RRT) algorithm is used for path planning. We have conducted experiments on obstacle avoidance, droplet transport, and drug fusion. The results clearly demonstrate the significant potential of magnetic fielddriven microcoil array devices in transportation and drug fusion engineering.

Mobile robots operating in convoys have a limited view of the terrain to be traversed if it is occluded by the preceding vehicle. Furthermore, the preceding vehicle might change the terrain geometry and eventually significantly alter its traversability by driving over the terrain. When the following vehicles do not consider such changes, they can use spurious terrain appearance and geometry to decide whether to follow in the tracks of the previous vehicle or to avoid them since the preceding vehicle’s tracks can make the terrain untraversable. We propose to predict the terrain changes induced by the robot traversal on the traversed terrain and thus support the decision-making of the following vehicles. The developed model projects the robot wheel footprint along the planned robot path and combines the projection with the terrain appearance and prior terrain elevation. The coupled model is used in a convolutional neural network that predicts the elevation after traversal. The footprint projection component is designed so that learned networks can be transferred to vehicles with different wheel footprints without relearning the model. The proposed model is verified using a dataset captured using a real, one-ton, six-wheel robot traversing rigid roads and vegetated fields.

This paper presents the OmniRace approach to controlling a racing drone with 6-degree of freedom (DoF) hand pose estimation and gesture recognition. To our knowledge, this is the first technology enabling low-level control of high-speed drones through gestures. OmniRace employs a gesture interface based on computer vision and a deep neural network to estimate 6-DoF hand pose. The advanced machine learning algorithm robustly interprets human gestures, allowing users to control drone motion intuitively. Real-time control tests validate the system’s effectiveness and its potential to revolutionize drone racing and other applications. Experimental results conducted in simulation environment revealed that OmniRace allows the users to complite the UAV race track significantly (by 25.1%) faster and to decrease the length of the test drone path (from 102.9 to 83.7 m). Users preferred the gesture interface for attractiveness (1.57 UEQ score), hedonic quality (1.56 UEQ score), and lower perceived temporal demand (32.0 score in NASA-TLX), while noting the high efficiency (0.75 UEQ score) and low physical demand (19.0 score in NASA-TLX) of the baseline remote controller. The deep neural network attains an average accuracy of 99.75% when applied to both normalized datasets and raw datasets. OmniRace can potentially change the way humans interact with and navigate racing drones in dynamic and complex environments. The source code is available at https://github.com/SerValera/OmniRace.git.

Adopting omnidirectional Field of View (FoV) cameras in aerial robots vastly improves perception ability, significantly advancing aerial robotics’s capabilities in inspection, reconstruction, and rescue tasks. However, such sensors also elevate system complexity, e.g., hardware design, and corresponding algorithm, which limits researchers from utilizing aerial robots with omnidirectional FoV in their research. To bridge this gap, we propose OmniNxt, a fully open-source aerial robotics platform with omnidirectional perception. We design a high-performance flight controller Nxt-FC and a multi-fisheye camera set for OmniNxt. Meanwhile, the compatible software is carefully devised, which empowers OmniNxt to achieve accurate localization and real-time dense mapping with limited computation resource occupancy. We conducted extensive real-world experiments to validate the superior performance of OmniNxt in practical applications. All the hardware and software are open-access at3, and we provide docker images of each crucial module in the proposed system. Project page: https://hkust-aerial-robotics.github.io/OmniNxt.

Meta-Reinforcement Learning (Meta-RL) is a machine learning paradigm aimed at learning reinforcement learning policies that can quickly adapt to unseen tasks with few-shot data. Nevertheless, applying Meta-RL to real-world applications faces challenges due to the cost of data acquisition. To address this problem, offline Meta-RL has emerged as a promising solution, focusing on learning policies from pre-collected data that can effectively and rapidly adapt to unseen tasks. In this paper, we propose a new offline Meta-RL method called Meta-Actor-Critic with Evolving Gradient Agreement (MACEGA). MACEGA utilizes an evolutionary approach to estimate meta-gradients conductive to generalization across unseen tasks. During meta-training, gradient evolution is utilized to meta-update the value network and policies. Moreover, we use gradient agreement as an optimization objective for meta-learning, thereby enhancing the generalization ability of the meta-policy. We experimentally demonstrate the robustness of MACEGA in handling offline data quality. Furthermore, extensive experiments on various benchmarks provide empirical evidence that MACEGA outperforms previous state-of-the-art methods in generalizing to unseen tasks, thus demonstrating its potential for real-world applications.

Offline Reinforcement Learning (RL) offers an attractive alternative to interactive data acquisition by leveraging pre-existing datasets. However, its effectiveness hinges on the quantity and quality of the data samples. This work explores the use of more readily available, albeit off-dynamics datasets, to address the challenge of data scarcity in Offline RL. We propose a novel approach using conditional Diffusion Probabilistic Models (DPMs) to learn the joint distribution of the large-scale off-dynamics dataset and the limited target dataset. To enable the model to capture the underlying dynamics structure, we introduce two contexts for the conditional model: (1) a continuous dynamics score allows for partial overlap between trajectories from both datasets, providing the model with richer information; (2) an inverse-dynamics context guides the model to generate trajectories that adhere to the target environment’s dynamic constraints. Empirical results demonstrate that our method significantly outperforms several strong baselines. Ablation studies further reveal the critical role of each dynamics context. Additionally, our model demonstrates that by modifying the context, we can interpolate between source and target dynamics, making it more robust to subtle shifts in the environment.

In agricultural robotics, effective observation and localization of fruits present challenges due to occlusions caused by other parts of the tree, such as branches and leaves. These occlusions can result in false fruit localization or impede the robot from picking the fruit. The objective of this work is to push away branches that block the fruit’s view to increase their visibility. Our setup consists of an RGB-D camera and a robot arm. First, we detect the occluded fruit in the RGB image and estimate its occluded part via a deep learning generative model in the depth space. The direction to push to clear the occlusions is determined using classic image processing techniques. We then introduce a 3D extension of the 2D Hough transform to detect straight line segments in the point cloud. This extension helps detect tree branches and identify the one mainly responsible for the occlusion. Finally, we clear the occlusion by pushing the branch with the robot arm. Our method uses a combination of deep learning for fruit appearance estimation, classic image processing for push direction determination, and 3D Hough transform for branch detection. We validate our perception methods through real data under different lighting conditions and various types of fruits (i.e. apple, lemon, orange), achieving improved visibility and successful occlusion clearance. We demonstrate the practical application of our approach through a real robot branch pushing demonstration.

Visual SLAM uses visual information, typically point features, to localise a camera and, at the same time, map the environment. In recent years, there has been interest in using scene-understanding capabilities to enhance the mapping process and object-level SLAM systems have appeared in response. However, most of the previous work is limited to prestored object models or pre-trained networks to represent the objects, which limits working scenarios or uses representations with limited scope, such as cubes or quadrics. To address this, we propose to use superquadrics as the object representation and, in this paper, present a proof of principle SLAM system in which object-based mapping is fully integrated with camera tracking via keyframe optimisation. The system was tested on simulated and real datasets, and the results show that the system can achieve lightweight and comparatively good object representation whilst also giving good camera trajectories estimates under certain scenarios.

Classification of different object surface material types can play a significant role in the decision-making algorithms for mobile robots and autonomous vehicles. RGB-based scene-level semantic segmentation has been well-addressed in the literature. However, improving material recognition using the depth modality and its integration with SLAM algorithms for 3D semantic mapping could unlock new potential benefits in the robotics perception pipeline. To this end, we propose a complementarity-aware deep learning approach for RGB-D-based material classification built on top of an object-oriented pipeline. The approach further integrates the ORB-SLAM2 method for 3D scene mapping with multiscale clustering of the detected material semantics in the point cloud map generated by the visual SLAM algorithm. Extensive experimental results with existing public datasets and newly contributed real-world robot datasets demonstrate a significant improvement in material classification and 3D clustering accuracy compared to state-of-the-art approaches for 3D semantic scene mapping.

We consider the task of generating segmentation masks for the target object from an object manipulation instruction, which allows users to give open vocabulary instructions to domestic service robots. Conventional segmentation generation approaches often fail to account for objects outside the camera’s field of view and cases in which the order of vertices differs but still represents the same polygon, which leads to erroneous mask generation. In this study, we propose a novel method that generates segmentation masks from open vocabulary instructions. We implement a novel loss function using optimal transport to prevent significant loss where the order of vertices differs but still represents the same polygon. To evaluate our approach, we constructed a new dataset based on the REVERIE dataset and Matterport3D dataset. The results demonstrated the effectiveness of the proposed method compared with existing mask generation methods. Remarkably, our best model achieved a +16.32% improvement on the dataset compared with a representative polygon-based method.

We have developed a new method to estimate a Next Viewpoint (NV) which is effective for pose estimation of simple-shaped products for product display robots in retail stores. Pose estimation methods using Neural Networks (NN) based on an RGBD camera are highly accurate, but their accuracy significantly decreases when the camera acquires few texture and shape features at a current view point. However, it is difficult for previous mathematical model-based methods to estimate effective NV which is because the simple shaped objects have few shape features. Therefore, we focus on the relationship between the pose estimation and NV estimation. When the pose estimation is more accurate, the NV estimation is more accurate. Therefore, we develop a new pose estimation NN that estimates NV simultaneously. Experimental results showed that our NV estimation realized a pose estimation success rate 77.3%, which was 7.4pt higher than the mathematical model-based NV calculation did. Moreover, we verified that the robot using our method displayed 84.2% of products.

Robots that assist humans in their daily lives should be able to locate specific instances of objects in an environment that match a user’s desired objects. This task is known as instance-specific image goal navigation (InstanceImageNav), which requires a model that can distinguish different instances of an object within the same class. A significant challenge in robotics is that when a robot observes the same object from various 3D viewpoints, its appearance may differ significantly, making it difficult to recognize and locate accurately. In this paper, we introduce a method called SimView, which leverages multi-view images based on a 3D semantic map of an environment and self-supervised learning using SimSiam to train an instance-identification model on-site. The effectiveness of our approach was validated using a photorealistic simulator, Habitat Matterport 3D, created by scanning actual home environments. Our results demonstrate a 1.7-fold improvement in task accuracy compared with contrastive language-image pre-training (CLIP), a pre-trained multimodal contrastive learning method for object searching. This improvement highlights the benefits of our proposed fine-tuning method in enhancing the performance of assistive robots in InstanceImageNav tasks. The project website is https://emergentsystemlabstudent.github.io/MultiViewRetrieve/.