ICRA 2024 Papers

The Michigan Robotics MBot is a low-cost mobile robot platform that has been used to train over 1,400 students in autonomous navigation since 2014 at the University of Michigan and our collaborating colleges. The MBot platform was designed to meet the needs of teaching robotics at scale to match the growth of robotics as a field and an academic discipline. Transformative advancements in robot navigation over the past decades have led to a significant demand for skilled roboticists across industry and academia. This demand has sparked a need for robotics courses in higher education, spanning all levels of undergraduate and graduate experiences. Incorporating real robot platforms into such courses and curricula is effective for conveying the unique challenges of programming embodied agents in real-world environments and sparking student interest. However, teaching with real robots remains challenging due to the cost of hardware and the development effort involved in adapting existing hardware for a new course. In this paper, we describe the design and evolution of the MBot platform, and the underlying principals of scalability and flexibility which are keys to its success.

HD map construction is a fundamental and challenging task in autonomous driving to understand the surrounding environment. Recently, Camera-LiDAR BEV feature fusion methods have attracted increasing attention in HD map construction task, which can significantly boost the benchmark. However, existing fusion methods ignore modal interaction and utilize very simple fusion strategy, which suffers from the problems of misalignment and information loss. To tackle this, we propose a novel Multi-modal BEV feature fusion method named MBFusion. Specifically, to solve the semantic misalignment problem between Camera and LiDAR features, we design Cross-modal Interaction Transform (CIT) module to make these two feature spaces interact knowledge with each other to enhance the feature representation by the cross-attention mechanism. Then, we propose a Dual Dynamic Fusion (DDF) module to automatically select valuable information from different modalities for better feature fusion. Moreover, MBFusion is simple, and can be plug-and-played into existing pipelines. We evaluate MBFusion on three architectures, including HDMapNet, VectorMapNet, and MapTR, to show its versatility and effectiveness. Compared with the state-of-the-art methods, MBFusion achieves 3.6% and 4.1% absolute improvements on mAP on the nuScenes and the Argoverse2 datasets, respectively, demonstrating the superiority of our method.

The sim-to-real gap poses a significant challenge in RL-based multi-agent exploration due to scene quantization and action discretization. Existing platforms suffer from the inefficiency in sampling and the lack of diversity in Multi-Agent Reinforcement Learning (MARL) algorithms across different scenarios, restraining their widespread applications. To fill these gaps, we propose MAexp, a generic platform for multi-agent exploration that integrates a broad range of state-of-the-art MARL algorithms and representative scenarios. Moreover, we employ point clouds to represent our exploration scenarios, leading to high-fidelity environment mapping and a sampling speed approximately 40 times faster than existing platforms. Furthermore, equipped with an attention-based Multi-Agent Target Generator and a Single-Agent Motion Planner, MAexp can work with arbitrary numbers of agents and accommodate various types of robots. Extensive experiments are conducted to establish the first benchmark featuring several high-performance MARL algorithms across typical scenarios for robots with continuous actions, which highlights the distinct strengths of each algorithm in different scenarios.

We present a novel optimization-based Visual-Inertial SLAM system designed for multiple partially over-lapped camera systems, named MAVIS. Our framework fully exploits the benefits of wide field-of-view from multi-camera systems, and the metric scale measurements provided by an inertial measurement unit (IMU). We introduce an improved IMU pre-integration formulation based on the exponential function of an automorphism of SE2(3), which can effectively enhance tracking performance under fast rotational motion and extended integration time. Furthermore, we extend conventional front-end tracking and back-end optimization module designed for monocular or stereo setup towards multi-camera systems, and introduce implementation details that contribute to the performance of our system in challenging scenarios. The practical validity of our approach is supported by our experiments on public datasets. Our MAVIS won the first place in all the vision-IMU tracks (single and multi-session SLAM) on Hilti SLAM Challenge 2023 with 1.7 times the score compared to the second place 1.

Data-driven methods have great advantages in modeling complicated human behavioral dynamics and dealing with many human-robot interaction applications. However, collecting massive and annotated real-world human datasets has been a laborious task, especially for highly interactive scenarios. On the other hand, algorithmic data generation methods are usually limited by their model capacities, making them unable to offer realistic and diverse data needed by various application users. In this work, we study trajectory-level data generation for multi-human or human-robot interaction scenarios and propose a learning-based automatic trajectory generation model, which we call Multi-Agent TRajectory generation with dIverse conteXts (MATRIX). MATRIX is capable of generating interactive human behaviors in realistic diverse contexts. We achieve this goal by modeling the explicit and interpretable objectives so that MATRIX can generate human motions based on diverse destinations and heterogeneous behaviors. We carried out extensive comparison and ablation studies to illustrate the effectiveness of our approach across various metrics. We also presented experiments that demonstrate the capability of MATRIX to serve as data augmentation for imitation-based motion planning.

Depth perception is crucial for a wide range of robotic applications. Multi-frame self-supervised depth estimation methods have gained research interest due to their ability to leverage large-scale, unlabeled real-world data. However, the self-supervised methods often rely on the assumption of a static scene and their performance tends to degrade in dynamic environments. To address this issue, we present Motion-Aware Loss, which leverages the temporal relation among consecutive input frames and a novel distillation scheme between the teacher and student networks in the multi-frame self-supervised depth estimation methods. Specifically, we associate the spatial locations of moving objects with the temporal order of input frames to eliminate errors induced by object motion. Meanwhile, we enhance the original distillation scheme in multi-frame methods to better exploit the knowledge from a teacher network. MAL is a novel, plug-and-play module designed for seamless integration into multi-frame self-supervised monocular depth estimation methods. Adding MAL into previous state-of-the-art methods leads to a reduction in depth estimation errors by up to 4.2% and 10.8% on KITTI and CityScapes benchmarks, respectively.

With the increase of robots navigating through crowded environments in our daily lives, the demand for designing a socially-aware navigation method considering humanrobot interaction has risen. When developing and assessing socially-aware navigation methods, pedestrian motion modeling plays a significant role. However, existing pedestrian models often struggle in complex environments and do not have the capacity to generate diverse pedestrian styles.In this paper, we propose multi-agent reinforcement learning with local coordination for individual diversity (MAC-ID), which can synthesize diverse pedestrian motions via local coordination factor (LCF). Our experiments have demonstrated that the manipulation of the LCF induces interpretable changes in pedestrian behaviors, along with a superior performance compared to existing pedestrian motion models. For evaluating socially-aware navigation methods using MAC-ID, we present a novel benchmark called BSON. It offers realistic and diverse social environments with pedestrians modeled via MAC-ID. We have trained and compared various navigation methods in BSON using a newly proposed metric called socially-aware navigation score (SNS). Through BSON, users can evaluate their socially-aware navigation methods and compare them to baselines.

This paper presents a lumped-parameter dynamic model of a pressure driven eversion robot carrying a catheter through its hollow core. A simulation framework based on the model is developed in MATLAB and is used for understanding the underlying physics, for identifying the regions of operation, and for demonstrating that, for a range of input commands, the catheter can be used as an actuation mechanism for propelling eversion; an approach especially useful for miniaturised systems. Simulations are experimentally validated on the MAMMOBOT system, which is a miniature steerable soft growing robot for early breast cancer detection. It was demonstrated that for most regions of operation experimental results compare well with simulation exhibiting an error less than 4%. Only one region of operation demonstrated larger deviations due possibly to unmodeled dynamics, which will be investigated in future work.

This work focuses on understanding and identifying the drag forces applied to a rotary-wing Micro Aerial Vehicle (MAV). We propose a lumped drag model that concisely describes the aerodynamical forces the MAV is subject to, with a minimal set of parameters. We only rely on commonly available sensor information onboard a MAV, such as accelerometer data, pose estimate, and throttle commands, which makes our method generally applicable. The identification uses an offline gradient-based method on flight data collected over specially designed trajectories. The identified model allows us to predict the aerodynamical forces experienced by the aircraft due to its own motion in real-time and, therefore, will be useful to distinguish them from external perturbations, such as wind or physical contact with the environment. The results show that we are able to identify the drag coefficients of a rotary-wing MAV through onboard flight data and observe the close correlation between the motion of the MAV, the measured external forces, and the predicted drag forces.

Robotic gas distribution mapping improves the understanding of a hazardous gas dispersion while putting the human operator out of danger. Generating an accurate gas distribution map quickly is of utmost importance in situations such as gas leaks and industrial incidents, so that the efficient use of resources in response to incidents can be facilitated. In this paper, to incorporate the operational requirement on map granularity, we propose a low-to-high resolution path planner that first guides a single robots to quickly and sparsely sample the region of interest to generate a low resolution gas distribution map, followed by high resolution sampling informed by the low resolution map as a prior. The low resolution prior acts as a coverage survey allowing the algorithm to perform a relatively exploitative search of high concentration regions, resulting in overall shorter mission times. The proposed framework is designed to iteratively identify the next best T locations to sample, which prioritises the potentially high reward locations, while ensuring that the robot can travel to and sample the chosen locations within a user specified map update cycle. We present a simulation study to demonstrate the alternating exploration-exploitation like behaviour along with bench-marking its performance in contrast to the traditional sampling path planners and various reward functions.

Anticipating driver intention is an important task when vehicles of mixed and varying levels of human/machine autonomy share roadways. Driver intention can be leveraged to improve road safety, such as warning surrounding vehicles in the event the driver is attempting a dangerous maneuver. In this work, we propose a novel method of utilizing both in-cabin and external camera data to improve state-of-the-art performance in predicting future driver actions. Compared to existing methods, our approach explicitly extracts object and road-level features from external camera data, which we demonstrate are important features for predicting driver intention. Using our handcrafted features as inputs for both a transformer and a long-short-term-memory-based architecture, we empirically show that jointly utilizing in-cabin and external features improves performance compared to using in-cabin features alone. Furthermore, our models predict driver maneuvers more accurately and sooner than existing approaches, with an accuracy of 87.5% and an average prediction time of 4.35 seconds before the maneuver takes place. We release our model configurations and training scripts on https://github.com/ykung83/Driver-Intent-Prediction.

Localizing ground penetrating radar (LGPR) has been proven to be a promising technology for robot localization in various dynamic environments. However, the extreme scarcity of underground features introduces false candidate matches and brings unique challenges to this task. In this paper, we propose a sequence-based framework for LGPR to address the aforementioned issues. Specifically, we first introduce a trainable strategy to extract robust underground features in multi-weather conditions. By further using sequential information, our LGPR system can observe richer underground scene contexts, and the associated multi-frame scans could also improve the performance of underground place recognition. We demonstrate the superiority of our proposed method by comparing it against several recent state-of-the-art baseline methods applied to GPR image tasks. Experimental results on large public and self-collected datasets show that our proposed framework significantly improves the performance of various baselines in different scenarios.

Centralized control of a multi-agent system improves upon distributed control especially when multiple agents share a common task e.g., sorting different materials in a recycling facility. Traditionally, each agent in a sorting facility is tuned individually which leads to suboptimal performance if one agent is less efficient than the others. Centralized control overcomes this bottleneck by leveraging global system state information, but it can be computationally expensive. In this work, we propose a novel framework called Longitudinal Control Volumes (LCV) to model the flow of material in a recycling facility. We then employ a Kalman Filter that incorporates local measurements of materials into a global estimation of the material flow in the system. We utilize a model predictive control algorithm that optimizes the rate of material flow using the global state estimate in real-time. We show that our proposed framework outperforms distributed control methods by 40-100% in simulation and physical experiments.

Existing 3D understanding datasets typically provide annotations for a limited number of object classes, with sufficient examples per class. However, real-world object classes are not equally represented in practical settings, leading to poor performance on rarely-occurring categories if the class imbalance is neglected. In this work, we address the challenge of 3D semantic segmentation with a long-tail distribution of classes. Common methods to reduce class imbalance during training include data re-sampling, loss re-weighting, and transfer learning. In contrast, our work proposes to effectively utilize network classifier weights in 3D models to balance the training on long-tail class distributions. While previous work in the 2D domain has studied imposing constraints on the classifier weights to regularize the training, it is sensitive to hyper-parameter choices and has not been yet explored for the 3D domain. To address these challenges, our work proposes adaptive regularization for frequent classes and sampling-based regularization for rare classes that alleviate the need to manually select thresholds and can dynamically focus training on the hard classes. Our experiments on the large-scale Scan-Net200 benchmark show that our method achieves improved performance, surpassing methods that rely on re-sampling, re-weighting, and pre-training.

We aim to address key challenges in long-horizon embodied exploration and navigation by proposing a long-horizon object transport task called Long-HOT and a novel modular framework for temporally extended navigation. Agents in Long-HOT need to efficiently find and pick up target objects that are scattered in the environment, carry them to a goal location with load constraints, and optionally have access to a container. We propose a modular topological graph-based transport policy (HTP) that explores efficiently with the help of weighted frontiers. Our hierarchical approach uses a combination of motion planning algorithms to reach point goals within explored locations and object navigation policies for moving towards semantic targets at unknown locations. Experiments on both our proposed Habitat transport task and on MultiOn benchmarks show that our method outperforms baselines and prior works. Further, we analyze the agent’s behavior for the usage of the container and demonstrate meaningful generalization to harder transport scenes with training only on simpler versions of the task.

The timber-harvesting industry is lagging its peer industries, such as mining and agriculture, with respect to deployment of robotic, AI and autonomous technologies. In this paper, we tackle automation of a critical task that arises in transporting logs from the forest to the sawmill: the log loading operation. This work is motivated by the acute shortages of human operators and the need to improve the efficiencies of timber-harvesting processes. To this end, we demonstrate the full autonomy pipeline for the log loading operation with a fixed-base manipulator (a.k.a., the crane), starting with perception of logs around the machine, then grasp planning for where to grasp logs, through motion planning and control of the log loading maneuver. Our main contribution is in the full integration of the necessary elements to achieve a completely autonomous loading cycle, where the crane picks up and loads all logs within its reach on a trailer. Notable features of our implementation are a generalizable perception stack, a grasp planner to pick up multiple logs at a time and an extensive experimental campaign conducted outdoors, on a commercial log loader retrofitted for autonomy. Our results demonstrate an overall 87% success rate of the log loading operation, with primary failure cases due to log segmentation errors and deficiencies in the final height adjustment algorithm for grasping logs. We also present detailed timing results of the main parts of the autonomy pipeline, which support the feasibility of deployment in operational environment.

The recent advancements in transformer-based visual trackers have led to significant progress, attributed to their strong modeling capabilities. However, as performance improves, running latency correspondingly increases, presenting a challenge for real-time robotics applications, especially on edge devices with computational constraints. In response to this, we introduce LiteTrack, an efficient transformer-based tracking model optimized for high-speed operations across various devices. It achieves a more favorable trade-off between accuracy and efficiency than the other lightweight trackers. The main innovations of LiteTrack encompass: 1) asynchronous feature extraction and interaction between the template and search region for better feature fushion and cutting redundant computation, and 2) pruning encoder layers from a heavy tracker to refine the balnace between performance and speed. As an example, our fastest variant, LiteTrack-B4, achieves 65.2% AO on the GOT-10k benchmark, surpassing all preceding efficient trackers, while running over 100 fps with ONNX on the Jetson Orin NX edge device. Moreover, our LiteTrack-B9 reaches competitive 72.2% AO on GOT-10k and 82.4% AUC on TrackingNet, and operates at 171 fps on an NVIDIA 2080Ti GPU. The code and demo materials will be available at https://github.com/TsingWei/LiteTrack.

Not relying on ground-truth data for training, self-supervised semantic visual odometry (SVO) has recently gained considerable attention. Within self-supervised SVO, feature representation inconsistency between semantic/depth and pose tasks presents a significant challenge, as it may disrupt cross-task feature representations and lead to notable performance degradation. Regrettably, existing self-supervised SVO lacks an effective solution to address this obstacle, for either overlooking this issue or exploiting a too heavy architecture. In response to this challenge, we propose a groundbreaking solution within the Single-Stream architecture, known as Lite-SVO, which is a lightweight yet efficient multi-feature sharing architecture. Lite-SVO is designed to bolster self-supervised SVO, facilitating its adoption on edge devices without compromising accuracy and performance. The crucial innovation lies in the multi-feature sharing architecture, which fuses the semantic and depth maps as pose features, thus significantly reducing the model complexity and boosting the speed in edge devices. Built upon the novel feature sharing framework, Lite-SVO further optimizes the feature sharing representation to improve the performance. Specifically, a cross-feature sharing module alleviates the impact of object boundary in depth estimation, while a multi-feature sharing module focuses on extracting and fusing spatial features to enhance pose estimation. Experimental results demonstrate that our method is at least 84.46% faster than the state-of-the-art Single-Stream approaches, and excitingly, our method’s pose accuracy is about 79.83% higher than theirs.

This paper proposes a novel flexible vibrational actuator with a structural anisotropy and its control method to diversify the vibrational behavior. First, the analytical model of the proposed actuator, which comprises a rectangular cross-sectional flexible beam and a rotational-type motor, is introduced. Regarding the structural anisotropy, the rotational axis of the motor is nonparallel to both principal axes of bending stiffness of the beam. Then, the vibrational phenomenon of the actuator is theoretically revealed. It is shown that using the synthetic wave input constituting two sine waves based on the resonance frequencies for the principal axes of the beam, the vibrational orbit of the tip of the beam can be controlled in the same manner as the Lissajous curve. Finally, the proposed method is experimentally validated. The Lissajous curve-based vibrational orbit control is performed using a prototype actuator. Furthermore, an application to underactuated-type locomotor is demonstrated.