IROS 2024 论文列表

This work proposes an optimal control scheme for a trajectory-tracking Automated Guided Vehicle considering motion and collision constraints in a warehouse environment. We outline how the simpler obstacle avoidance constraints in the Cartesian Coordinate Frame (CCF) can be retained, while projecting the tricycle kinematics to the Frenet Coordinate Frame (FCF) for track progress. The Nonlinear Model Predictive Control (NMPC) scheme is subsequently implemented using acados and its real-time feasibility is demonstrated in simulation and aboard a test vehicle at a warehouse.

Object detection forms a key component in Unmanned Aerial Vehicles (UAVs) for completing high-level tasks that depend on the awareness of objects on the ground from an aerial perspective. In that scenario, adversarial patch attacks on an onboard object detector can severely impair the performance of upstream tasks. This paper proposes a novel model-agnostic defense mechanism against the threat of adversarial patch attacks in the context of UAV-based object detection. We formulate adversarial patch defense as an occlusion removal task. The proposed defense method can neutralize adversarial patches located on objects of interest, without exposure to adversarial patches during training. Our lightweight single-stage defense approach allows us to maintain a model-agnostic nature, that once deployed does not require to be updated in response to changes in the object detection pipeline. The evaluations in digital and physical domains show the feasibility of our method for deployment in UAV object detection pipelines, by significantly decreasing the Attack Success Ratio without incurring significant processing costs. As a result, the proposed defense solution can improve the reliability of object detection for UAVs.

Place recognition is an important task for robots and autonomous cars to localize themselves and close loops in pre-built maps. While single-modal sensor-based methods have shown satisfactory performance, cross-modal place recognition that retrieving images from a point-cloud database remains a challenging problem. Current cross-modal methods transform images into 3D points using depth estimation for modality conversion, which are usually computationally intensive and need expensive labeled data for depth supervision. In this work, we introduce a fast and lightweight framework to encode images and point clouds into place-distinctive descriptors. We propose an effective Field of View (FoV) transformation module to convert point clouds into an analogous modality as images. This module eliminates the necessity for depth estimation and helps subsequent modules achieve real-time performance. We further design a non-negative factorization-based encoder to extract mutually consistent semantic features between point clouds and images. This encoder yields more distinctive global descriptors for retrieval. Experimental results on the KITTI dataset show that our proposed methods achieve state-of-the-art performance while running in real time. Additional evaluation on the HAOMO dataset covering a 17 km trajectory further shows the practical generalization capabilities. We have released the implementation of our methods as open source at: https://github.com/haomo-ai/ModaLink.git.

Recent studies reveal that well-performing reinforcement learning (RL) agents in training often lack resilience against adversarial perturbations during deployment. This highlights the importance of building a robust agent before deploying it in the real world. Most prior works focus on developing robust training-based procedures to tackle this problem, including enhancing the robustness of the deep neural network component itself or adversarially training the agent on strong attacks. In this work, we instead study an input transformation-based defense for RL. Specifically, we propose using a variant of vector quantization (VQ) as a transformation for input observations, which is then used to reduce the space of adversarial attacks during testing, resulting in the transformed observations being less affected by attacks. Our method is computationally efficient and seamlessly integrates with adversarial training, further enhancing the robustness of RL agents against adversarial attacks. Through extensive experiments in multiple environments, we demonstrate that using VQ as the input transformation effectively defends against adversarial attacks on the agent’s observations.

Making fast and accurate consensus decisions through local communication and decentralised control in a swarm of simple robots can be a very challenging endeavour. In swarms of robots with limited capabilities, consensus decisions can be made using simple voting rules. In our study, the robots use rules based on the cross-inhibition model, which describes a voting mechanism observed in the house-hunting honeybee, that has been shown to efficiently allow consensus achievement in distributed robotic systems. The cross-inhibition mechanism has been shown to lead to a highly stable consensus, preventing the correction of possible group decision errors which can happen, for example, due to high noise in robots’ estimations. In this paper, we investigate the impact of miscommunication on the speed-accuracy trade-off in consensus decision-making in the context of a binary discrimination problem—i.e., choosing collectively the best of two options. We evaluate the accuracy of decision-making theoretically, using continuous and finite-size models, and experimentally in a collective perception scenario, using swarms of 100 simulated robots and 50 real Kilobots. Our study suggests that a certain level of miscommunication (or communication noise) among agents can increase the decision’s accuracy and, thus, can serve an important functional role in making collective decisions in robot swarms.

Screening of the lower gastrointestinal (GI) tract is of paramount importance for the early detection of precancerous lesions in the intestine, with an impact on reducing the high death rate of patients affected by cancer worldwide. Colonoscopy, i.e. standard procedure for screening the colon, is effective in reducing the incidence of colorectal cancer worldwide, nonetheless, this procedure remains an invasive method of screening, that typically causes discomfort and requires sedation for the patient. The SoftSCREEN system, a tethered robotic capsule designed for colonoscopy, aims to enable minimally invasive diagnosis of intestinal diseases through its innovative design that incorporates elastic tracks for locomotion and inflatable toroidal chambers for adaptable geometry to match the local lumen of the GI tract. After demonstrating the viability of the proposed design in a large-scale proof of concept in our previous work, the authors present here a miniaturised version of the SoftSCREEN system. We assess its performance in multiple phantom tests and evaluate the effect of pressure regulation on its locomotion. The conducted extensive tests demonstrate the capability of the soft robot to move inside intricate passages, capture internal images, and adjust its geometry to optimise traction. The results underscore the potential of the proposed design, offering promising advancements in the development of a robotic platform for efficient front-wheel locomotion and accurate intestinal screening.

Miniature robots capable of untethered operation hold great promise for performing diagnostic and therapeutic procedures in hard-to-reach regions within the human body. Nonetheless, navigating these complex and diverse physiological environments remains a significant challenge. To effectively navigate the tortuous pathways inside the human body, it is essential to equip miniature robots with flexible body structures that can adapt to complex geometries and develop efficient actuation strategies for deformed robots. In this study, we present a miniature soft robot featuring a zigzag body structure, imparting the robot with remarkable deformation capabilities that enable it to adapt to confined and tortuous spaces. This robot is equipped with arrays of magnetic legs, enabling robust locomotion propelled by traveling metachronal waves. We demonstrate that the robot can crawl on both flat surfaces and slopes. Leveraging its in-plane flexibility and discrete actuation system, this robot can navigate through intricate environments with precise control using magnetic fields. Our work provides valuable insights into the development of crawling robots with enhanced agility and adaptability, creating opportunities for their future use in a wide range of biomedical applications.

We propose MetaEMG, a meta-learning approach for fast adaptation in intent inferral on a robotic hand orthosis for stroke. One key challenge in machine learning for assistive and rehabilitative robotics with disabled-bodied subjects is the difficulty of collecting labeled training data. Muscle tone and spasticity often vary significantly among stroke subjects, and hand function can even change across different use sessions of the device for the same subject. We investigate the use of metalearning to mitigate the burden of data collection needed to adapt high-capacity neural networks to a new session or subject. Our experiments on real clinical data collected from five stroke subjects show that MetaEMG can improve the intent inferral accuracy with a small session- or subject-specific dataset and very few fine-tuning epochs. To the best of our knowledge, we are the first to formulate intent inferral on stroke subjects as a meta-learning problem and demonstrate fast adaptation to a new session or subject for controlling a robotic hand orthosis with EMG signals.

Safe Reinforcement Learning (Safe RL) is one of the prevalently studied subcategories of trial-and-error-based methods with the intention to be deployed on real-world systems. In safe RL, the goal is to maximize reward performance while minimizing constraints, often achieved by setting bounds on constraint functions and utilizing the Lagrangian method. However, deploying Lagrangian-based safe RL in real-world scenarios is challenging due to the necessity of threshold fine-tuning, as imprecise adjustments may lead to suboptimal policy convergence. To mitigate this challenge, we propose a unified Lagrangian-based model-free architecture called Meta Soft Actor-Critic Lagrangian (Meta SAC-Lag). Meta SAC-Lag uses meta-gradient optimization to automatically update the safety-related hyperparameters. The proposed method is designed to address safe exploration and threshold adjustment with minimal hyperparameter tuning requirement. In our pipeline, the inner parameters are updated through the conventional formulation and the hyperparameters are adjusted using the meta-objectives which are defined based on the updated parameters. Our results show that the agent can reliably adjust the safety performance due to the relatively fast convergence rate of the safety threshold. We evaluate the performance of Meta SAC-Lag in five simulated environments against Lagrangian baselines, and the results demonstrate its capability to create synergy between parameters, yielding better or competitive results. Furthermore, we conduct a real-world experiment involving a robotic arm tasked with pouring coffee into a cup without spillage. Meta SAC-Lag is successfully trained to execute the task, while minimizing effort constraints. The success of Meta SAC-Lag in performing the experiment is intended to be a step toward practical deployment of safe RL algorithms to learn the control process of safety-critical real-world systems without explicit engineering.

Scene Rearrangement Planning (SRP) has recently emerged as a crucial interior scene task; however, current approaches still face two primary issues. First, prior works define the action space of SRP using handcrafted coarse-grained actions, which are inflexible for scene arrangement transition and impractical for real-world deployment. Secondly, the scarcity of realistic indoor scene rearrangement data hinders popular data-hungry learning approaches and quantitative evaluation. To tackle these issues, we propose a fine-grained action space definition and curate a large-scale scene rearrangement dataset to facilitate the training of learning approaches and comprehensive benchmarking. Building upon this dataset, we introduce a novel framework, PLATO, designed for efficient agent training and inference. Our approach features an exPert-assisted curriculum Learning (PL) paradigm that possesses a Behavior Cloning (BC) and an offline Reinforcement Learning (RL) curriculum for agent training, along with an advanced tree-search-based planner enhanced by an Adaptive Trade-Off (ATO) strategy to improve expert agent performance further. We demonstrate the superior performance of our method over baseline agents through extensive experiments and provide a detailed analysis to elucidate its rationale. Our project website can be accessed at plato.github.io.

We propose MaskingDepth, a semi-supervised learning framework for monocular depth estimation. MaskingDepth is designed to enforce consistency between the depths obtained from strongly-augmented images and the pseudo-depths derived from weakly-augmented images, which enables mitigating the reliance on large ground-truth depth quantities. In this framework, we leverage uncertainty estimation to only retain high-confident depth predictions from the weakly-augmented branch as pseudo-depths. We also present a novel data augmentation, dubbed K-way disjoint masking, that takes advantage of a naïve token masking strategy as an augmentation, while avoiding its scale ambiguity problem between depths from weakly-and strongly-augmented branches and risk of missing small-scale objects. Experiments on KITTI and NYU-Depth-v2 datasets demonstrate the effectiveness of each component, its robustness to the use of fewer depth-annotated images, and superior performance compared to other state-of-the-art semi-supervised learning methods for monocular depth estimation.

Visual mask learning has received increasing attention in the field of visual object tracking. However, most existing studies merely utilize visual mask learning works as pre-training models without fully exploiting their potential for visual representation. In this paper, we present a novel approach for learning tracking target features, leveraging an encoder-decoder architecture with a masked mutual guidance tracking(MMG). Initially, we perform joint visual feature extraction on both the template and search areas. Subsequently, these features undergo separate self-decoding processes, followed by mutual guidance decoding to reconstruct the original search and template images. This process fosters mutual understanding between the images, facilitating improved learning of object states and shapes across different frames. During the inference phase, we offload the decoder and implement a simple and effective tracker. Experimental results indicate that our proposed method is effective that the mutual guidance strategy can achieve state-of-the-art performance on five tracking datasets.

For biodiversity and forestry applications, end-users desire maps of forests that are fully detailed—from the forest floor to the canopy. Terrestrial laser scanning and aerial laser scanning are accurate and increasingly mature methods for scanning the forest. However, individually they are not able to estimate attributes such as tree height, trunk diameter and canopy density due to the inherent differences in their field-of-view and mapping processes. In this work, we present a pipeline that can automatically generate a single joint terrestrial and aerial forest reconstruction. The novelty of the approach is a marker-free registration pipeline, which estimates a set of relative transformation constraints between the aerial cloud and terrestrial sub-clouds without requiring any co-registration reflective markers to be physically placed in the scene. Our method then uses these constraints in a pose graph formulation, which enables us to finely align the respective clouds while respecting spatial constraints introduced by the terrestrial SLAM scanning process. We demonstrate that our approach can produce a fine-grained and complete reconstruction of large-scale natural environments, enabling multi-platform data capture for forestry applications without requiring external infrastructure.

Embodied Question Answering (EQA) serves as a benchmark task to evaluate the capability of robots to navigate within novel environments and identify objects in response to human queries. However, existing EQA methods often rely on simulated environments and operate with limited vocabularies. This paper presents a map-based modular approach to EQA, enabling real-world robots to explore and map unknown environments. By leveraging foundation models, our method facilitates answering a diverse range of questions using natural language. We conducted extensive experiments in both virtual and real-world settings, demonstrating the robustness of our approach in navigating and comprehending queries within unknown environments.

Manta rays exhibit complex motion behavior through their flexible fins. This study proposes a novel manta ray-inspired soft robotic swimmer with bistable flapping wings for high-speed and multi-modal swimming. The wings are created with prestressed bistable composite and actuated by small McKibben artificial muscles. Pressurizing and depressurizing the McKibben actuators integrated into the flapping wings generates alternating snap-throughs between two stable states, yielding swimming. Experiments are set up and conducted to analyze how the robot responses vary as a function of input pressures and actuation frequencies for both bistable and monostable modes. Experimental results show that the highest swimming velocity is 0.58 body lengths (BL) per second (equivalent to 12.23 cm/s), and the maximum turning angle speed is 22.5° per second with a smaller turning radius by holding the fins in asymmetric positions for bistable modes. Multimodal swimming motions are achieved including forward and backward translating, turning, and flip-turning.

While the integration of Multi-modal Large Language Models (MLLMs) with robotic systems has significantly improved robots’ ability to understand and execute natural language instructions, their performance in manipulation tasks remains limited due to a lack of robotics-specific knowledge. Conventional MLLMs are typically trained on generic image-text pairs, leaving them deficient in understanding affordances and physical concepts crucial for manipulation. To address this gap, we propose ManipVQA, a novel framework that infuses MLLMs with manipulation-centric knowledge through a Visual Question-Answering (VQA) format. This approach encompasses tool detection, affordance recognition, and a broader understanding of physical concepts. We curated a diverse dataset of images depicting interactive objects, to challenge robotic understanding in tool detection, affordance prediction, and physical concept comprehension. To effectively integrate this robotics-specific knowledge with the inherent vision-reasoning capabilities of MLLMs, we leverage a unified VQA format and devise a fine-tuning strategy. This strategy preserves the original vision-reasoning abilities while incorporating the newly acquired robotic insights. Empirical evaluations conducted in robotic simulators and across various vision task benchmarks demonstrate the robust performance of ManipVQA. The code and dataset are publicly available at https://github.com/SiyuanHuang95/ManipVQA.

To substantially enhance robot intelligence, there is a pressing need to develop a large model that enables general-purpose robots to proficiently undertake a broad spectrum of manipulation tasks, akin to the versatile task-planning ability exhibited by LLMs. The vast diversity in objects, robots, and manipulation tasks presents huge challenges. Our work introduces a comprehensive framework to develop a foundation model for general robotic manipulation that formalizes a manipulation task as contact synthesis. Specifically, our model takes as input object and robot manipulator point clouds, object physical attributes, target motions, and manipulation region masks. It outputs contact points on the object and associated contact forces or post-contact motions for robots to achieve the desired manipulation task. We perform extensive experiments both in the simulation and real-world settings, manipulating articulated rigid objects, rigid objects, and deformable objects that vary in dimensionality, ranging from one-dimensional objects like ropes to two-dimensional objects like cloth and extending to three-dimensional objects such as plasticine. Our model achieves average success rates of around 90%. Supplementary materials and videos are available on our project website at https://manifoundationmodel.github.io/.

Modern robotic perception is highly dependent on neural networks. It is well known that neural network-based perception can be unreliable in real-world deployment, especially in difficult imaging conditions. Out-of-distribution detection is commonly proposed as a solution for ensuring reliability in real-world deployment. Previous work has shown that normalizing flow models can be used for out-of-distribution detection to improve reliability of robotic perception tasks. Specifically, camera parameters can be optimized with respect to the likelihood output from a normalizing flow, which allows a perception system to adapt to difficult vision scenarios. With this work we propose to use the absolute gradient values from a normalizing flow, which allows the perception system to optimize local regions rather than the whole image. By setting up a table top picking experiment with exceptionally difficult lighting conditions, we show that our method achieves a 60% higher success rate for an object detection task compared to previous methods.

For humanoid whole body contact motions, it is important to recognize the existence of whole body contacts and the contact forces. The challenges in recognizing the existence of whole body contacts and the contact forces in life-size humanoids are: 1) the measurement part with low mechanical strength must be tolerant of high load and 2) it is difficult to model thick elastic bodies with high impact tolerance and uneven sensor placements when applied to various shapes of the whole body. This paper proposes a method of constructing a load tolerant tactile sensor by separating the loaded part from the measuring part with magnetism and protecting the measuring part inside the frame of the robot. For modeling difficulties, this paper proposes learning the relationship between the change in the detected physical quantity due to deformation of the elastic body and the contact force distribution. This paper shows through experiments that the proposed tactile sensor based on a robot frame is load tolerant enough to support the weight of a life-sized humanoid, and that it can acquire contact force distribution and the robot is able to acclimate to external forces.