IROS 2024 论文列表

Despite extensive research on sliding mode control (SMC) design for quadrotors, the existing approaches have certain limitations. Euler angle-based SMC formulations suffer from poor performance in high-pitch or -roll maneuvers. Quaternion-based SMC approaches have unwinding issues and complex architecture. Coordinate-free methods are slow and only almost globally stable. This paper presents a new six degrees of freedom SMC flight controller to address the above limitations. We use a cascaded architecture with a position controller in the outer loop and a quaternion-based attitude controller in the inner loop. The position controller generates the desired trajectory for the attitude controller using a coordinate-free approach. The quaternion-based attitude controller uses the natural characteristics of the quaternion hypersphere, featuring a simple structure while providing global stability and avoiding unwinding issues. We compare our controller with three other common control methods conducting challenging maneuvers like flip-over and high-speed trajectory tracking in the presence of model uncertainties and disturbances. Our controller consistently outperforms the benchmark approaches with less control effort and actuator saturation, offering highly effective and efficient flight control.

Traversing 3-D complex environments has always been a significant challenge for legged locomotion. Existing methods typically rely on external sensors such as vision and lidar to preemptively react to obstacles by acquiring environmental information. However, in scenarios like nighttime or dense forests, external sensors often fail to function properly, necessitating robots to rely on proprioceptive sensors to perceive diverse obstacles in the environment and respond promptly. This task is undeniably challenging. Our research finds that methods based on collision detection can enhance a robot’s perception of environmental obstacles. In this work, we propose an end-to-end learning-based quadruped robot motion controller that relies solely on proprioceptive sensing. This controller can accurately detect, localize, and agilely respond to collisions in unknown and complex 3D environments, thereby improving the robot’s traversability in complex environments. We demonstrate in both simulation and real-world experiments that our method enables quadruped robots to successfully traverse challenging obstacles in various complex environments. The videos and appendix can be found at Quad-Traverse-Go2.github.io

3D Multi-Object Tracking (MOT) has achieved tremendous achievement thanks to the rapid development of 3D object detection and 2D MOT. Recent advanced works generally employ a series of object attributes, e.g., position, size, velocity, and appearance, to provide the clues for the association in 3D MOT. However, these cues may not be reliable due to some visual noise, such as occlusion and blur, leading to tracking performance bottlenecks. To reveal the dilemma, we conduct extensive empirical analysis to expose the key bottleneck of each clue and how they correlate with each other. The analysis results motivate us to efficiently absorb the merits among all cues and adaptively produce an optimal tracking manner. Specifically, we present Location and Velocity Quality Learning, which efficiently guides the network to estimate the quality of predicted object attributes. Based on these quality estimations, we propose a quality-aware object association (QOA) strategy to leverage the quality score as an important reference factor for achieving robust association. Despite its simplicity, extensive experiments indicate that the proposed strategy significantly boosts tracking performance by 2.2% AMOTA and our method outperforms all existing state-of-the-art works on nuScenes by a large margin. Moreover, QTrack achieves 51.1%, 54.8% and 56.6% AMOTA tracking performance on the nuScenes test sets with BEVDepth, VideoBEV, and StreamPETR models respectively, which significantly reduces the performance gap between the pure camera and LiDAR-based trackers.

Underwater object detection has attracted increasing interest for its wide application in various underwater tasks. However, due to underwater image quality degradation and the lack of large-scale underwater object datasets, many underwater detectors suffer from low detection performance. To address the issues, we not only propose a novel underwater transformer detector with multi-scale feature enhancement and query optimization, named QO-Net, but also construct a new underwater object detection dataset, called UODD. Specifically, a Conv-Trans Layer is developed as the unit of QO-Net, which effectively learns multi-scale image feature representation through CNN and simultaneously captures the dependencies among different positions in the sequence data through Transformer, enabling QO-Net to process underwater image sequence information over longer distances. An effective combination can enhance the representation of multi-scale features. Then, QO-Net develops a positional query enhancement strategy to optimize the spatial prior of positional queries, thereby speeding up the convergence of the network training. In addition, UODD also contains more than 20,000 underwater images for training and validation, with a variety of rich underwater categories. Extensive experiments on UODD, Brackish, and TrashCan datasets demonstrate that QO-Net presents favorable detection performance against state-of-the-art methods in terms of robustness and accuracy.

Accurately predicting deformable linear object (DLO) dynamics is challenging, especially when the task requires a model that is both human-interpretable and computationally efficient. In this work, we draw inspiration from the pseudo-rigid body method (PRB) and model a DLO as a serial chain of rigid bodies whose internal state is unrolled through time by a dynamics network. This dynamics network is trained jointly with a physics-informed encoder that maps observed motion variables to the DLO's hidden state. To encourage the state to acquire a physically meaningful representation, we leverage the forward kinematics of the PRB model as a decoder. We demonstrate in robot experiments that the proposed DLO dynamics model provides physically interpretable predictions from partial observations while being on par with black-box models regarding prediction accuracy. The project code is available at: tinyurl.com/prb-networks

This paper propose a novel transfer learning approach, Pseudo-Domain Adversarial Network (PDAN), to tackle the issue of electrode displacement in Electrical Impedance Tomography (EIT). Electrode displacement, caused by human movement or improper operation, significantly affects the accuracy of EIT by introducing data errors. Existing solutions either modify the electrode assembly at a high cost or employ recognition algorithms that require retraining from scratch. To overcome these limitations, our work leverages the power of transfer learning to enhance model performance in the target domain by utilizing knowledge from a related task in the source domain. PDAN extends the capabilities of deep adversarial learning by incorporating noisy images to simulate post-electrode rotation scenarios, aiding in the reduction of negative impacts caused by minor electrode displacements. Our method demonstrates superior performance in classifying leg posture data, achieving around 90% accuracy, and proving robust against sensor electrode offset. Experimental results across various datasets validate the effectiveness of PDAN, indicating its potential in addressing complex real-world situations with improved generalization capabilities.

We propose a novel framework for few-shot learning by leveraging large-scale vision-language models such as CLIP [1]. Motivated by unimodal prototypical networks for few-shot learning, we introduce Proto-CLIP which utilizes image prototypes and text prototypes for few-shot learning. Specifically, Proto-CLIP adapts the image and text encoder embeddings from CLIP in a joint fashion using few-shot examples. The embeddings from the two encoders are used to compute the respective prototypes of image classes for classification. During adaptation, we propose aligning the image and text prototypes of the corresponding classes. Such alignment is beneficial for few-shot classification due to the reinforced contributions from both types of prototypes. Proto-CLIP has both training-free and fine-tuned variants. We demonstrate the effectiveness of our method by conducting experiments on benchmark datasets for few-shot learning, as well as in the real world for robot perception1.

Recent works in field robotics highlighted the importance of resiliency against different types of terrains. Boreal forests, in particular, are home to many mobility-impeding terrains that should be considered for off-road autonomous navigation. Also, being one of the largest land biomes on Earth, boreal forests are an area where autonomous vehicles are expected to become increasingly common. In this paper, we address the issue of classifying boreal terrains by introducing BorealTC, a publicly available dataset for proprioceptive-based terrain classification (TC). Recorded with a Husky A200, our dataset contains 116 min of Inertial Measurement Unit (IMU), motor current, and wheel odometry data, focusing on typical boreal forest terrains, notably snow, ice, and silty loam. Combining our dataset with another dataset from the literature, we evaluate both a Convolutional Neural Network (CNN) and the novel state space model (SSM)-based Mamba architecture on a TC task. We show that while CNN outperforms Mamba on each separate dataset, Mamba achieves greater accuracy when trained on a combination of both. In addition, we demonstrate that Mamba’s learning capacity is greater than a CNN for increasing amounts of data. We show that the combination of two TC datasets yields a latent space that can be interpreted with the properties of the terrains. We also discuss the implications of merging datasets on classification. Our source code and dataset are publicly available online: https://github.com/norlab-ulaval/BorealTC.

In the field of robotics, researches have sought to control robots capable of dealing with a variety of environments and tasks generically, through the use of foundation models. Among these, the systems for robot behavior planning utilizing video have also been proposed. The system enables the generation of robot behaviors that are not dependent on specific environments or tasks. This is achieved by generating videos based on text input, which utilizes the vast knowledge inherent in the foundation models. Also, by using a visual interface such as video, it is possible to confirm the behavioral indicators on which the robot is operating. Although, there are few examples of research on robot behavior planning utilizing video. Previous studies have emphasized the verification of behavior generation utilizing video, with simplified object manipulation for testing on simulations. This is not enough to demonstrate the usefulness of robot behavior planning utilizing video in real-world environments. In addition, the systems from previous studies are not open, and such systems have not been sufficiently discussed. This paper attempts to construct robot behavior planning utilizing video as an open system, and to verify the validity of the behavior planning using actual machines. In this paper, we first focus on using Robotis's TURTLEBOT3 Waffle Pi and Mobile Manipulator(referred to as "Waffle") to construct robot behavior planning system utilizing video. Second, we create planning videos targeting the pick-and-place motion using the proposed system, and control the arm part of Waffle in the actual machine verification. Finally, by comparing the target coordinates from the planning video with the coordinates observed from the actual machine, we can confirm whether it is possible to control Waffle as planned. Errors are calculated from the coordinate comparison, and the control is performed again. Based on the results, we verify whether the proposed system is useful for contro... Show More

Nighttime UAV tracking under low-illuminated scenarios has achieved great progress by domain adaptation (DA). However, previous DA training-based works are deficient in narrowing the discrepancy of temporal contexts for UAV trackers. To address the issue, this work proposes a prompt-driven temporal domain adaptation training framework to fully utilize temporal contexts for challenging nighttime UAV tracking, i.e., TDA. Specifically, the proposed framework aligns the distribution of temporal contexts from daytime and nighttime domains by training the temporal feature generator against the discriminator. The temporal-consistent discriminator progressively extracts shared domain-specific features to generate coherent domain discrimination results in the time series. Additionally, to obtain high-quality training samples, a prompt-driven object miner is employed to precisely locate objects in unannotated nighttime videos. Moreover, a new benchmark for long-term nighttime UAV tracking is constructed. Exhaustive evaluations on both public and self-constructed nighttime benchmarks demonstrate the remarkable performance of the tracker trained in TDA framework, i.e., TDA-Track. Real-world tests at nighttime also show its practicality. The code and demo videos are available at https://github.com/vision4robotics/TDA-Track.

Visual object tracking has significantly promoted autonomous applications for unmanned aerial vehicles (UAVs). However, learning robust object representations for UAV tracking is especially challenging in complex dynamic environments, when confronted with aspect ratio change and occlusion. These challenges severely alter the original information of the object. To handle the above issues, this work proposes a novel progressive representation learning framework for UAV tracking, i.e., PRL-Track. Specifically, PRL-Track is divided into coarse representation learning and fine representation learning. For coarse representation learning, two innovative regulators, which rely on appearance and semantic information, are designed to mitigate appearance interference and capture semantic information. Furthermore, for fine representation learning, a new hierarchical modeling generator is developed to intertwine coarse object representations. Exhaustive experiments demonstrate that the proposed PRL-Track delivers exceptional performance on three authoritative UAV tracking benchmarks. Real-world tests indicate that the proposed PRL-Track realizes superior tracking performance with 42.6 frames per second on the typical UAV platform equipped with an edge smart camera. The code, model, and demo videos are available at https://github.com/vision4robotics/PRL-Track.

Expressing images with Multi-Resolution (MR) features has been widely adopted in many computer vision tasks. In this paper, we introduce the MR concept into Bird's-Eye-View (BEV) semantic segmentation for autonomous driving. This introduction enhances our model's ability to capture both global and local characteristics of driving scenes through our proposed residual learning. Specifically, given a set of MR BEV query maps, the lowest resolution query map is initially updated using a View Transformation (VT) encoder. This updated query map is then upscaled and merged with a higher resolution query map to undergo further updates in a subsequent VT encoder. This process is repeated until the resolution of the updated query map reaches the target. Finally, the lowest resolution map is added to the target resolution to generate the final query map. During training, we enforce both the lowest and final query maps to align with the ground-truth BEV semantic map to help our model effectively capture the global and local characteristics. We also propose a visual feature interaction network that promotes interactions between features across images and across feature levels, thus highly contributing to the performance improvement. We evaluate our model on a large-scale real-world dataset. The experimental results show that our model outperforms the SOTA models in terms of IoU metric. Codes are available at https://github.com/d1024choi/ProgressiveQueryReneNet

Soft robots exhibit complex behaviors despite simple control, due to their inherently compliant hardware which passively deforms upon contact ± a concept commonly referred to as morphological computation. To fully determine the behavior of soft robots, not only their control software but also their passive behavior needs to be programmed. We show that deliberate programming of passive deformation in soft fingertips can significantly influence the grasping and manipulation performance of various robotic grippers. For this, the fingertips display strategically modulated compliance levels across their palmar surface, realized through adjustments to the local thickness of a lattice structure within their soft material, resulting in desired passive deformation. The grippers are operated by human participants, solving diverse tasks involving a variety of objects. We analyze 2025 human trials and show that the distinct passive behaviors programmed into the fingertips significantly affect grasping and manipulation performance. Furthermore, we discovered that specific compliance profiles consistently demonstrate superior performance, indicating that not merely inherent softness by itself, but a purposeful combination of varying compliance levels plays a pivotal role in successful soft interaction.

Developing a robotic application requires thorough testing of the complete system to ensure its reliability. However, depending on the target environment, real-life testing can be difficult to carry out, which favors simulations. Also, some techniques like those based on machine learning, may require large varieties of sensor data, which can be gathered in simulation with ease, whereas doing the same in real environments can pose a great challenge.This work presents a flexible approach to the procedural generation of tunnel networks suitable for underground robotics simulations. The method starts with a graph representation of an underground environment, and applies a custom meshing strategy to generate tunnels that follow the graph structure. This mesh can then be imported into the desired simulation software. The ease of use of this method allows for the testing of robotic applications in an arbitrary number of different environments in completely automated workflows.

Modern robots need to adapt to diverse human partners with whom they collaborate. To this end, learning a representation of human capabilities enables the robot to personalize their behaviour to their collaborators across multiple tasks. We propose CApability Modeling from Observations (CAMO), a model-based estimation algorithm, in which human capabilities that parameterize a given model are inferred from observations of the human behaviour on known collaborative tasks. We apply the method to joint limit learning in order to predict future trajectories of a 7-DOF manipulator arm. Furthermore, we show that CAMO can be used as a sub-task assignment routine in a simulated human–robot collaboration scenario, allowing the robot to adapt its task allocation to perform tasks that the person is not able to do.

We present a homotopic approach to solving challenging, optimization-based motion planning problems. The approach uses Homotopy Optimization, which, unlike standard continuation methods for solving homotopy problems, solves a sequence of constrained optimization problems rather than a sequence of nonlinear systems of equations. The insight behind our proposed algorithm is formulating the discovery of this sequence of optimization problems as a search problem in a multidimensional homotopy parameter space. Our proposed algorithm, the Probabilistic Homotopy Optimization algorithm, switches between solve and sample phases, using solutions to easy problems as initial guesses to more challenging problems. We analyze how our algorithm performs in the presence of common challenges to homotopy methods, such as bifurcation, folding, and disconnectedness of the homotopy solution manifold. Finally, we demonstrate its utility via a case study on two dynamic motion planning problems. the cart-pole and the MIT Humanoid.

Learning from demonstration (LfD) has emerged as a promising approach enabling robots to acquire complex tasks directly from human demonstrations. However, tasks involving surface interactions on freeform 3D surfaces present unique challenges in modeling and execution, especially when geometric variations exist between demonstrations and robot execution. This paper proposes a novel framework called probabilistic surface interaction primitives (ProSIP), which systematically incorporates the surface path and the local surface features into the learning procedure. An instrumented tool allows seamless recording and execution of human demonstrations. By design, ProSIPs are independent of time, invariant to rigid-body displacements, and apply to any robotic platform with a Cartesian controller. The framework is employed for an edge-cleaning task of bathroom sinks. The generalization capability to various object geometries and significantly distorted objects is demonstrated. Simulations and an experimental setup with a 9-degrees-of-freedom robotic platform confirm the performance.

Performing an inspection task while maintaining the privacy of the inspected site is a challenging balancing act. In this work, we are motivated by the future of nuclear arms control verification, which requires both a high level of privacy and guaranteed correctness. For scenarios with limitations on sensors and stored information due to the potentially secret nature of observable features, we propose a robotic verification procedure that provides map-free exploration to perform a source verification task without requiring, nor revealing, any task-irrelevant, site-specific information. We provide theoretical guarantees on the privacy and correctness of our approach, validated by extensive simulated and hardware experiments.

Autonomous vehicles require online learning capabilities to enable long-term, unattended operation. However, long-term online learning is accompanied by the problem of forgetting previously learned knowledge. This paper introduces an online learning framework that includes a catastrophic forgetting prevention mechanism, named Long-Short-Term Online Learning (LSTOL). The framework consists of a set of shortterm learners and a long-term controller, where the former is based on the concept of ensemble learning and aims to achieve rapid learning iterations, while the latter contains a simple yet efficient probabilistic decision-making mechanism combined with four control primitives to achieve effective knowledge maintenance. A novel feature of the proposed LSTOL is that it avoids forgetting while learning autonomously. In addition, LSTOL makes no assumptions about the model type of short-term learners and the continuity of the data. The effectiveness of the proposed framework is demonstrated through experiments across well-known datasets in autonomous driving, including KITTI and Waymo. The source code for the method implementation is publicly available at https://github.com/epan-utbm/lstol.