IROS 2024 论文列表

This paper presents a time-optimal path parameterization (TOPP) method for cooperative multi-arm robotic systems (MARS) manipulating heavy objects with third-order constraints that include jerk, torque rate and wrench rate limits. The method is based on a problem reformulation as a sequential linear program and provides a unified planning approach that is faster than previous convex optimization techniques. The equivalence to a reachability-based TOPP is shown and simulation results for a cooperative MARS consisting of two 7 degree of freedom (DOF) robots and a tightly grasped object with 6 DOFs are provided.

In this paper, we present a Radar-Inertial Odometry (RIO) method based on the nonlinear optimization of factor graphs in a sliding window fashion. Our method makes use of a light-weight, low-power, inexpensive and commonly available hardware enabling easy deployment on small Unmanned Aerial Vehicles (UAV)s. We keep the state estimation problem bounded by employing partial marginalization of the oldest states, rendering the method real-time capable. We compare the implemented approach to the state-of-the-art multi-state Extended Kalman Filter (EKF)-based method in a one-to-one fashion. That is, we implemented in a single custom C++ RIO framework both estimation back-ends with all other parts shared and thus identical for a fair direct comparison. In the real-world flight experiments, we compare the two methods and show that both perform similarly in terms of accuracy when the linearization point is not far from the true state. Upon wrong initialization, the factor graph approach heavily outperforms the EKF approach. We also acknowledge that the influence of undetected outliers can overwhelm the inherent benefits of the nonlinear optimization approach leading to the insight that the estimator front-end has an important (and often underestimated) role in the overall performance. The open source code and datasets can be found here: https://github.com/aau-cns/aaucns_rio.

This study investigates the effectiveness of a passive tightly coupled ultra-wideband (UWB) based inertial navigation system for indoor positioning of mobile platforms. Unlike conventional methods that rely on time difference of arrival (TDOA) or two-way ranging (TWR) measurements, the proposed approach utilizes local reception timestamps directly. An error state Kalman filter with right quaternion error definition is used in the state estimation process. Evaluation is performed first in a Matlab simulation environment and then, using a dataset acquired by flying a quadcopter while monitored by a motion capture system. Timestamp measurements were acquired using custom firmware flashed onto Decawave DWM 1000-DEV hardware. Our findings demonstrate that the proposed system outperforms traditional TDOA methods, providing accurate measurements even in the presence of communication interruptions, with as few as one anchor.

This work aims to present a three-dimensional vehicle dynamics state estimation under varying signal quality. Few researchers have investigated the impact of three-dimensional road geometries on the state estimation and, thus, neglect road inclination and banking. Especially considering high velocities and accelerations, the literature does not address these effects. Therefore, we compare two- and three-dimensional state estimation schemes to outline the impact of road geometries. We use an Extended Kalman Filter with a point-mass motion model and extend it by an additional formulation of reference angles. Furthermore, virtual velocity measurements significantly improve the estimation of road angles and the vehicle’s side slip angle. We highlight the importance of steady estimations for vehicle motion control algorithms and demonstrate the challenges of degraded signal quality and Global Navigation Satellite System dropouts. The proposed adaptive covariance facilitates a smooth estimation and enables stable controller behavior. The developed state estimation has been deployed on a high-speed autonomous race car at various racetracks. Our findings indicate that our approach outperforms state-of-the-art vehicle dynamics state estimators and an industry-grade Inertial Navigation System. Further studies are needed to investigate the performance under varying track conditions and on other vehicle types.

Though larger vessels may be well-equipped to deal with wavy conditions, smaller vessels are often more susceptible to disturbances. This paper explores the development of a nonlinear model predictive control (NMPC) system for Uncrewed Surface Vessels (USVs) in wavy conditions to minimize average roll. The NMPC is based on a prediction method that uses information about the vessel’s dynamics and an assumed wave model. This method is able to mitigate the roll of an under-actuated USV in a variety of conditions by adjusting the weights of the cost function. The results show a reduction of 39 % of average roll with a tuned controller in conditions with 1.75-metre sinusoidal waves. A general and intuitive tuning strategy is established. This preliminary work is a proof of concept which sets the stage for the leveraging of wave prediction methodologies to perform planning and control in real time for USVs in real-world scenarios and field trials.

In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highly-detailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction. However, the predominant reliance on RGB imaging presupposes ideal lighting conditions—a premise frequently unmet in robotic applications plagued by poor lighting or visual obstructions. This limitation overlooks the capabilities of infrared (IR) cameras, which excel in low-light detection and present a robust alternative under such adverse scenarios. To tackle these issues, we introduce Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging. By leveraging a thermal mapping and structural thermal constraint derived from the thermal characteristics of IR imaging, our method showcases unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short. We conduct extensive experiments to demonstrate that Thermal-NeRF can achieve superior quality compared to existing methods. Furthermore, we contribute a dataset for IR-based NeRF applications, paving the way for future research in IR NeRF reconstruction, see https://github.com/Cerf-Volant425/Thermal-NeRF.

Thermal ablation therapy is a major minimally invasive treatment. One of the challenges is that the targeted region and therapeutic progression are often invisible to clinicians, requiring feedback provided in numerical information or imaging. Several emerging imaging modalities offer visualization of the ablation-induced necrosis formation; however, relying solely on necrosis monitoring can result in tissue overheating and endangering patients. Some of the necrosis monitoring modalities are known for their capabilities in temperature sensing, but the principles on which they are based have several limitations, such as sensitivity to the tissue motion and their environment. In this study, we propose a necrosis progression-based temperature estimation technique as an added safety feature for avoiding overheating. This model-based method does not require additional sensing hardware. It is designed to work as an independent estimator or a complimentary estimation component with other thermometers for improved robustness. For this objective, the Neural State Space model is used to approximate the ablation therapy, whose theoretical models involve nonlinear partial differential equations. Then, the Extended Kalman Filter is designed based on the model. The simulation study shows the estimation module robustly estimates the tissue temperature under several types of noise. The maximum estimation error observed before terminating ablation was around 1 °C, and the desired safety feature was successfully demonstrated. The estimator is expected to be used in a variety of necrosis monitoring modalities to guarantee more precise and safer treatment. More ambitiously, the architecture with the Neural State Space model and Extended Kalman Filter is generalizable to other medical/biological procedures involving nonlinear and patient/environment-specific physics and even to procedures having no reliable theoretical models.

Recent research on mobile robots has focused on increasing their adaptability to unpredictable and unstructured environments using soft materials and structures. However, the determination of key design parameters and control over these compliant robots are predominantly iterated through experiments, lacking a solid theoretical foundation. To improve their efficiency, this paper aims to provide mathematics modeling over two locomotion, crawling and swimming. Specifically, a dynamic model is first devised to reveal the influence of the contact surfaces’ frictional coefficients on displacements in different motion phases. Besides, a swimming kinematics model is provided using coordinate transformation, based on which, we further develop an algorithm that systematically plans human-like swimming gaits, with maximum thrust obtained. The proposed algorithm is highly generalizable and has the potential to be applied in other soft robots with similar multiple joints. Simulation experiments have been conducted to illustrate the effectiveness of the proposed modeling.

Much of HRI research focuses on personalizing robots in order to ease societal acceptance, and favour their uptake. Companion robots are indeed conceived as a potential solution to numerous societal concerns, among which aging population, and individuals’ isolation. In such a perspective, personalization is indeed key, for it ensures individuals feel comfortable using robots in their daily lives and environments. This also depends upon the so called cultural competences the machine possesses. In fact, how humans behave largely depends upon their heritage, and overall understanding of the environment. An identical reaction, posture or expression might, indeed, be perceived very differently according to the culture of the person exposed to it.

Intubation is essential for ventilating critically ill patients and involves precise maneuvering of a laryngoscope to place an endotracheal tube (ETT). However, training for this procedure is fraught with challenges. Traditional methods, relying on manikins or training with a single sensing modality, fail to adequately convey important interaction information. Such challenge is heightened in pediatric intubation due to anatomical differences that demand greater precision. Furthermore, integrating multiple sensing modalities into a laryngoscope without changing its size presents a significant design challenge, critical for maintaining realistic training scenarios. To overcome these obstacles, we developed a sensorized laryngoscope system equipped with a force-torque sensor, a 9-axis inertial measurement unit (IMU), and tactile sensors. This system, validated in a preliminary user study, provides online feedback on angles, forces, and grip strength through an online feedback GUI. Adopting a learning-by-demonstration approach with both experts and novices, the initial validation confirmed the system’s potential, paving the way for expanded trials with more participants.

Humans rely on the synergy of their senses for most essential tasks. For tasks requiring object manipulation, we seamlessly and effectively exploit the complementarity of our senses of vision and touch. This paper draws inspiration from such capabilities and aims to find a systematic approach to fuse visual and tactile information in a reinforcement learning setting. We propose Masked Multimodal Learning (M3L), which jointly learns a policy and visual-tactile representations based on masked autoencoding. The representations jointly learned from vision and touch improve sample efficiency, and unlock generalization capabilities beyond those achievable through each of the senses separately. Remarkably, representations learned in a multimodal setting also benefit vision-only policies at test time. We evaluate M3L on three simulated environments with both visual and tactile observations: robotic insertion, door opening, and dexterous in-hand manipulation, demonstrating the benefits of learning a multimodal policy. Videos of the experiments and the open-source code are available at https://sferrazza.cc/m3l_site.

In the last decade, data-driven approaches have become popular choices for quadrotor control, thanks to their ability to facilitate the adaptation to unknown or uncertain flight conditions. Among the different data-driven paradigms, Deep Reinforcement Learning (DRL) is currently one of the most explored. However, the design of DRL agents for Micro Aerial Vehicles (MAVs) remains an open challenge. While some works have studied the output configuration of these agents (i.e., what kind of control to compute), there is no general consensus on the type of input data these approaches should employ. Multiple works simply provide the DRL agent with full state information, without questioning if this might be redundant and unnecessarily complicate the learning process, or pose superfluous constraints on the availability of such information in real platforms. In this work, we provide an in-depth benchmark analysis of different configurations of the observation space. We optimize multiple DRL agents in simulated environments with different input choices and study their robustness and their sim-to-real transfer capabilities with zero-shot adaptation. We believe that the outcomes and discussions presented in this work supported by extensive experimental results could be an important milestone in guiding future research on the development of DRL agents for aerial robot tasks.

Multi-agent pathfinding plays a crucial role in various robot applications. Recently, deep reinforcement learning methods have been adopted to solve large-scale planning problems in a decentralized manner. Nonetheless, such approaches pose challenges such as non-stationarity and partial observability. In this paper, we address these challenges by integrating a state representation model into a multi-agent proximal policy optimization framework. To do so, we propose to utilize a state representation model which extracts representation features from the global map and leverages this information to enhance the training process. Our approach involves decoupling the feature extractor from the agent training process, enabling a more accurate representation of the global state that remains unbiased by the actions of the agents. Furthermore, our modularized approach offers the flexibility to replace the representation model with another model or modify tasks within the global map, without the retraining of the agents. We demonstrated the effectiveness of our approach by comparing three multi-agent proximal policy optimization frameworks. Our experimental results demonstrate that our approach improves the average episode reward compared to the other approaches.

In this paper, we describe the design of the Barkour benchmark for measuring robot agility in navigating complex environments. Despite the growing interest in developing agile robot locomotion skills, the field lacks systematic benchmarks to measure the performance of robotic control systems and hardware in agility-focused tasks. This motivated us to propose the Barkour benchmark, an obstacle course designed to quantify agility across various robotic platforms. Inspired by dog agility competitions, the course features diverse obstacles and a time-based scoring mechanism, encouraging researchers to develop controllers that enable robots to move quickly, precisely, and with adaptability. This benchmark is challenging as it demands diverse motion skills and the time-based scoring requires control precision at high speed. Along with the design details presented in the paper, we release our simulated environment setups in MuJoCo-XLA and the CAD model of a custom-designed quadruped robot to facilitate future research to reproduce the Barkour setup (available at sites.google.com/view/barkour). We hope these together will accelerate the pace of robot agility research.

In the labor-intensive Roll-On/Roll-Off (RO/RO) terminal environment, research on vehicle transport robots with mobility, stability, and reliability is receiving increasing attention. This paper presents a novel control framework for a Straddle-Type Dual-Body vehicle transfer robot. Initially, fine segmentation and processing of point clouds from different areas of the robot are performed, switching perception strategies for different areas based on event triggers. For target pose estimation, a traversal-based point cloud matrix fitting algorithm is designed. Additionally, for loading and unloading operations, a docking controller based on real-time target detection is developed to ensure minimal lateral and angular errors during target docking. Finally, the proposed control framework is validated through operations of the vehicle transfer robot in outdoor RO/RO terminal yards. Experimental results indicate that the average docking error remains within 3cm, with a 6.5% reduction in docking time under the same conditions. The docking precision and stability performance of the vehicle transfer robot surpass traditional methods, demonstrating satisfactory performance.

LiDAR data of urban scenarios poses unique challenges, such as heterogeneous characteristics and inherent class imbalance. Therefore, large-scale datasets are necessary to apply deep learning methods. Instance augmentation has emerged as an efficient method to increase dataset diversity. However, current methods require the time-consuming curation of 3D models or costly manual data annotation. To overcome these limitations, we propose Text3DAug, a novel approach leveraging generative models for instance augmentation. Text3DAug does not depend on labeled data and is the first of its kind to generate instances and annotations from text. This allows for a fully automated pipeline, eliminating the need for manual effort in practical applications. Additionally, Text3DAug is sensor agnostic and can be applied regardless of the LiDAR sensor used. Comprehensive experimental analysis on LiDAR segmentation, detection and novel class discovery demonstrates that Text3DAug is effective in supplementing existing methods or as a standalone method, performing on par or better than established methods, however while overcoming their specific drawbacks. The code is publicly available.1

In spatial navigation, the shift from manual cartography to digital map representations has revolutionized how we interact with and comprehend outdoor and indoor environments. While digital mapping has substantially advanced outdoor navigation with robust techniques like satellite imagery and sophisticated data labeling, the full potential of indoor digital mapping remains untapped. Accurate indoor mapping promises to enhance the operational efficiency of mobile robots, improving their ability to interact with human environments, and bolstering emergency response capabilities. However, its realization is impeded by the complexity of current methods and the need for heavy manual labor, expert knowledge, and specialized equipment. To address these challenges, we introduce Text2Map – a novel methodology that harnesses natural language navigational instructions, the power of off-the-shelf Large Language Models (LLMs), and Few-shot Learning, to create graph-based digital maps of indoor spaces. This approach simplifies the mapping process for widespread use, leveraging crowd-sourceable ubiquitous navigation instructions as a data source without requiring specialized map data formats or hardware. Our paper presents the Text2Map system architecture, details the creation of the first dedicated dataset, and evaluates the system’s efficacy, highlighting the substantial potential and scalability of our approach. Text2Map achieves a Graph-Edit-Distance (GED) ranging from 0.5X to 2X the total number of regions in a building and an Edge Similarity score between 0.87 and 0.9. These results highlight the precision, robustness, and effectiveness of our methodology. Our work paves the way for a more accessible and streamlined approach to indoor digital mapping, setting the stage for broader adoption in human and mobile robot navigation applications.

Generating varied scenarios through simulation is crucial for training and evaluating safety-critical systems, such as autonomous vehicles. Yet, the task of modeling the trajectories of other vehicles to simulate diverse and meaningful close interactions remains prohibitively costly. Adopting language descriptions to generate driving behaviors emerges as a promising strategy, offering a scalable and intuitive method for human operators to simulate a wide range of driving interactions. However, the scarcity of large-scale annotated language-trajectory data makes this approach challenging. To address this gap, we propose Text-to-Drive (T2D) to synthesize diverse driving behaviors via Large Language Models (LLMs). We introduce a knowledge-driven approach that operates in two stages. In the first stage, we employ the embedded knowledge of LLMs to generate diverse language descriptions of driving behaviors for a scene. Then, we leverage LLM’s reasoning capabilities to synthesize these behaviors in simulation. At its core, T2D employs an LLM to construct a state chart that maps low-level states to high-level abstractions. This strategy aids in downstream tasks such as summarizing low-level observations, assessing policy alignment with behavior description, and shaping the auxiliary reward, all without needing human supervision. With our knowledge-driven approach, we demonstrate that T2D generates more diverse trajectories compared to other baselines and offers a natural language interface that allows for interactive incorporation of human preference. Please check our website for more examples: here

Deep learning plays a critical role in vision-based satellite pose estimation. However, the scarcity of real data from the space environment means that deep models need to be trained using synthetic data, which raises the Sim2Real domain gap problem. A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. Event sensors have been shown to provide some robustness against lighting variations in vision-based pose estimation. However, challenging lighting conditions due to strong directional light can still cause undesirable effects in the output of commercial off-the-shelf event sensors, such as noisy/spurious events and inhomogeneous event densities on the object. Such effects are non-trivial to simulate in software, thus leading to Sim2Real gap in the event domain. To close the Sim2Real gap in event-based satellite pose estimation, the paper proposes a test-time self-supervision scheme with a certifier module. Self-supervision is enabled by an optimisation routine that aligns a dense point cloud of the predicted satellite pose with the event data to attempt to rectify the inaccurately estimated pose. The certifier attempts to verify the corrected pose, and only certified test-time inputs are backpropagated via implicit differentiation to refine the predicted landmarks, thus improving the pose estimates and closing the Sim2Real gap. Results show that the our method outperforms established test-time adaptation schemes.