ICRA 2024 Papers

In shared autonomy, human-robot handover for object delivery is crucial. Accurate robot predictions of human hand motion and intentions enhance collaboration efficiency. However, low prediction accuracy increases mental and physical demands on the user. In this work, we propose a system for predicting hand motion and intended target during human-robot handover using Inverse Reinforcement Learning (IRL). A set of feature functions were designed to explicitly capture users’ preferences during the task. The proposed approach was experimentally validated through user studies. Results indicate that the proposed method outperformed other state-of-the-art methods (PI-IRL, BP-HMT, RNNIK-MKF and CMk=5) with users feeling comfortable reaching upto 60% of the total distance to the target for handover with 90% target prediction accuracy. The target prediction accuracy reaches 99.9% when less than 20% of the task remains.

Continuous exploration without interruption is important in scenarios such as search and rescue and precision agriculture, where consistent presence is needed to detect events over large areas. Ergodic search already derives continuous trajectories in these scenarios so that a robot spends more time in areas with high information density. However, existing literature on ergodic search does not consider the robot's energy constraints, limiting how long a robot can explore. In fact, if the robots are battery-powered, it is physically not possible to continuously explore on a single battery charge. Our paper tackles this challenge, integrating ergodic search methods with energy-aware coverage. We trade off battery usage and coverage quality, maintaining uninterrupted exploration by at least one agent. Our approach derives an abstract battery model for future state-of-charge estimation and extends canonical ergodic search to ergodic search under battery constraints. Empirical data from simulations and real-world experiments demonstrate the effectiveness of our energy-aware ergodic search, which ensures continuous exploration and guarantees spatial coverage.

The ability to fast climb and descend is crucial for Unmanned Aerial Vehicle (UAV) applications in the mountains. The slower descent speed will affect the UAV’s working efficiency in reaching the rescue area. However, during the fast descent of the rotorcraft, a chaotic flow field rampages as the rotorcraft falls into its wake flow. This is known as the vortex ring. Therefore, the safe descent velocity of consumer UAVs is usually limited to approximately 3m/s. This limitation reduces the potential of UAVs to execute tasks in mountainous and plateau regions. To broaden the task capability constrained by the maximum descending speed, it is necessary to jointly analyze the flow field and the energy consumption during descending. Existing research mainly focused on how to avoid entering the vortex ring instead of offering sufficient power to fly with it. In this paper, in order to achieve an efficient rotorcraft for rescuing in mountainous and plateaus, we break through the maximum-descending-speed of a coaxial rotors UAV. Hence, a power consumption managing pipeline is proposed to extend the power tolerance of the UAV. Specifically, a theoretic model for the coaxial rotors is proposed to analyze the induced velocity and energy consumption during vertical descending. Then, the theoretic model is verified to be consistent with the Computational Fluid Dynamics (CFD) and wind tunnel experiment results. Finally, we optimized the tolerance of the power and dynamic system according to the theoretic model. With this pipeline, our real-time flight achieved 8m/s controlled vertical-descent-speed (CVDS), which is a leading result in both quadrotors and coaxial UAVs.

In the fields of robotic perception and computer vision, achieving accurate semantic segmentation of low-light or nighttime scenes is challenging. This is primarily due to the limited visibility of objects and the reduced texture and color contrasts among them. To address the issue of limited visibility, we propose a hierarchical gated convolution unit, which simultaneously expands the receptive field and restores edge texture. To address the issue of reduced texture among objects, we propose a dual closed-loop bipartite matching algorithm to establish a total loss function consisting of the unsupervised illumination enhancement loss and supervised intersection-over-union loss, thus enabling the joint minimization of both losses via the Hungarian algorithm. We thus achieve end-to-end training for a semantic segmentation network especially suitable for handling low-light scenes. Experimental results demonstrate that the proposed network surpasses existing methods on the Cityscapes dataset and notably outperforms state-of-the-art methods on both Dark Zurich and Nighttime Driving datasets.

Aggressive time-optimal control of quadcopters poses a significant challenge in the field of robotics. The state-of-the-art approach leverages reinforcement learning (RL) to train optimal neural policies. However, a critical hurdle is the sim-to-real gap, often addressed by employing a robust inner loop controller —an abstraction that, in theory, constrains the optimality of the trained controller, necessitating margins to counter potential disturbances. In contrast, our novel approach introduces high-speed quadcopter control using end-to-end RL (E2E) that gives direct motor commands. To bridge the reality gap, we incorporate a learned residual model and an adaptive method that can compensate for modeling errors in thrust and moments. We compare our E2E approach against a state-of-the-art network that commands thrust and body rates to an INDI inner loop controller, both in simulated and real-world flight. E2E showcases a significant 1.39-second advantage in simulation and a 0.17-second edge in real-world testing, highlighting end-to-end reinforcement learning’s potential. The performance drop observed from simulation to reality shows potential for further improvement, including refining strategies to address the reality gap or exploring offline reinforcement learning with real flight data.

Exploiting thermal updrafts to gain altitude can significantly extend the endurance of fixed-wing aircraft, as has been demonstrated by human glider pilots for decades. In this work, we present a novel end-to-end deep learning approach for the simultaneous detection of multiple thermal updrafts and the estimation of their properties — a key capability to let autonomous unmanned aerial vehicles soar as well. In contrast to previous works, our approach does not require separate algorithms for the detection of individual updrafts. Instead, a sequence of sensor measurements from a time window of interest can be directly fed into our temporal convolutional network, which estimates the position, strength, and spread of the encountered updrafts. We demonstrated in simulations that our approach can reliably detect updrafts solely based on measurements of the aircraft’s position and the local vertical wind velocity. Nevertheless, our method can additionally make use of measurements of the roll moment induced by updrafts, which improves the precision further. Compared with a particle-filter-based method, we can determine the correct number of encountered updrafts with an accuracy of 99.99% instead of 79.50%, significantly improve the precision of strength as well as spread estimates, and reduce the computational demand.

3D instance segmentation is a fundamental and critical task for enabling robots to operate effectively in unstructured 3D environments. In order to address the challenges posed by the high demand for large-scale annotated data and the limited availability of such data in the context of 3D instance segmentation, we study semi-supervised 3D instance segmentation problem and propose a novel end-to-end framework based on the mean teacher paradigm, named PCTeacher. Our PCTeacher generates both point-level and cluster-level pseudo labels to harness knowledge from unlabeled data. It notably enhances the training stability through end-to-end training and improves pseudo-label quality. Specifically, for point-level pseudo labels, PCTeacher employs a multi-view fusion strategy to achieve higher precision and recall. Regarding cluster-level pseudo labels, it introduces a hybrid grouping strategy to generate more potential proposals and utilizes a point-cluster agreement-based thresholding (PCAT) mechanism to fully exploit cluster-level pseudo labels. By combining and strengthening both point-level and cluster-level pseudo labels, our PCTeacher achieves state-of-the-art performance on two benchmark datasets across multiple labeled data ratios with a more compact network compared to the existing method.

In an increasingly automated world – from ware-house robots to self-driving cars – streamlining the development and deployment process and operations of robotic applications becomes ever more important. Automated DevOps processes and microservice architectures have already proven successful in other domains such as large-scale customer-oriented web services (e.g., Netflix). We recommend to employ similar microservice architectures for the deployment of small- to large-scale robotic applications in order to accelerate development cycles, loosen functional dependence, and improve resiliency and elasticity. In order to facilitate involved DevOps processes, we present and release a tooling suite for automating the development of microservices for robotic applications based on the Robot Operating System (ROS). Our tooling suite covers the automated minimal containerization of ROS applications, a collection of useful machine learning-enabled base container images, as well as a CLI tool for simplified interaction with container images during the development phase. Within the scope of this paper, we embed our tooling suite into the overall context of streamlined robotics deployment and compare it to alternative solutions. We release our tools as open-source software at github.com/ika-rwth-aachen/dorotos.

An emerging trend in the field of human-robot collaboration is the disassembly of end-of-life products. Safety is a crucial requirement of the disassembly process since worn-out or damaged products could break, possibly resulting in dangerous behavior of the robot. To protect the user from such behavior, this work addresses this challenge through the implementation of an energy-aware Cartesian impedance controller, combined with virtual workspace restrictions. Hereby, the passivity of the robotic system is ensured. The paper proposed two approaches to ensure the passivity of the system when subjected to workspace restrictions due to unplanned interactions and contact loss. The first approach employs an augmented energy tank with restricted energy flow. The second approach monitors the overall energy flow, regulating and separating non-passive behavior, caused by workspace restrictions. The approaches are evaluated and compared with each other, by using a KUKA LBR iiwa robot. The results highlight the potential of virtual workspace restrictions in human-robot collaborative disassembly tasks.

Multi-robot collaboration in large-scale environments with limited-sized teams and without external infrastructure is challenging, since the software framework required to support complex tasks must be robust to unreliable and intermittent communication links. In this work, we present MOCHA (Multi-robot Opportunistic Communication for Heterogeneous Collaboration), a framework for resilient multi-robot collaboration that enables large-scale exploration in the absence of continuous communications. MOCHA is based on a gossip communication protocol that allows robots to interact opportunistically whenever communication links are available, propagating information on a peer-to-peer basis. We demonstrate the performance of MOCHA through real-world experiments with commercial-off-the-shelf (COTS) communication hardware. We further explore the system’s scalability in simulation, evaluating the performance of our approach as the number of robots increases and communication ranges vary. Finally, we demonstrate how MOCHA can be tightly integrated with the planning stack of autonomous robots. We show a communication-aware planning algorithm for a high-altitude aerial robot executing a collaborative task while maximizing the amount of information shared with ground robots.The source code for MOCHA and the high-altitude UAV planning system is available open source 1.

In recent years, significant progress has been made in the field of underwater image enhancement (UIE). However, its practical utility for high-level vision tasks, such as underwater object detection (UOD) in Autonomous Underwater Vehicles (AUVs), remains relatively unexplored. It may be attributed to several factors: (1) Existing methods typically employ UIE as a pre-processing step, which inevitably introduces considerable computational overhead and latency. (2) The process of enhancing images prior to training object detectors may not necessarily yield performance improvements. (3) The complex underwater environments can induce significant domain shifts across different scenarios, seriously deteriorating the UOD performance. To address these challenges, we introduce EnYOLO, an integrated real-time framework designed for simultaneous UIE and UOD with domain-adaptation capability. Specifically, both the UIE and UOD task heads share the same network backbone and utilize a lightweight design. Furthermore, to ensure balanced training for both tasks, we present a multi-stage training strategy aimed at consistently enhancing their performance. Additionally, we propose a novel domain-adaptation strategy to align feature embeddings originating from diverse underwater environments. Comprehensive experiments demonstrate that our framework not only achieves state-of-the-art (SOTA) performance in both UIE and UOD tasks, but also shows superior adaptability when applied to different underwater scenarios. Our efficiency analysis further highlights the substantial potential of our framework for onboard deployment.

Electroaerodynamic (EAD) propulsion, where thrust is produced by collisions between electrostatically-accelerated ions and neutral air, is a potentially transformative method for indoor flight owing to its silent and solid-state nature. Like rotors, EAD thrusters exhibit changes in performance based on proximity to surfaces. Unlike rotors, they have no fragile and quickly spinning parts that have to avoid those surfaces; taking advantage of the efficiency benefits from proximity effects may be a route towards longer-duration indoor operation of ion-propelled fliers. This work presents the first empirical study of ground proximity effects for EAD propulsors, both individually and as quad-thruster arrays. It focuses on multi-stage ducted centimeter-scale actuators suitable for use on small robots envisioned for deployment in human-proximal and indoor environments. Three specific effects (ground, suckdown, and fountain lift), each occurring with a different magnitude at a different spacing from the ground plane, are investigated and shown to have strong dependencies on geometric parameters including thruster-to-thruster spacing, thruster protrusion from the fuselage, and inclusion of flanges or strakes. Peak thrust enhancement ranging from 300 to 600% is found for certain configurations operated in close proximity (0.2 mm) to the ground plane and as much as a 20% increase is measured even when operated centimeters away.

Sensing plays a pivotal role in robotic manipulation, dictating the accuracy and versatility with which objects are handled. Vision-based sensing methods often suffer from fabrication complexity and low durability, while approaches that rely on direct measurements on the gripper often have limited resolution and are difficult to scale. Here, we present a soft robotic gripper made out of two cubic lattices that are sensorized by embedding air channels within the structure. The lattices are 3D printed from a single build material, simplifying the fabrication process. The flexibility of this approach offers significant control over sensor and lattice design, while the pressure-based internal sensing provides measurements with minimal disruption to the grasping surface. With only 12 sensors, 6 per lattice, this gripper can estimate an object’s weight and location and offer new insights into grasp parameters like friction coefficients and grasp force.

Six-axis force/moment (6-A F/M) sensors make surgical robots effectively sense intraoperative force feedback and drilling status information, reducing the operating challenges and psychological burden of doctors, which also improves the quality and safety of surgery. However, it is difficult for current commercial electrical 6-A F/M sensors to adapt the electromagnetic environment in the operating room, and status changes after installation can also reduce accuracy. At the same time, there is a strong vibration coupling of low-frequency force information, leading to low identification accuracy and slow response speed in the drilling and milling status. Aiming at these problems, an elliptical torus-based 6-A fiber optic F/M sensor and its in-situ calibration method for orthopedic surgical robot force sensing are proposed. Furthermore, combined with the multichannel one-dimensional convolutional gated recurrent unit (M1-DCGRU), a fast and accurate identification of seven drilling stages was realized. The final force sensing error is less than 7.1%, and the drilling state identification accuracy is at least 93.9%. The designed sensor has higher accuracy, is compatible with magnetic resonance imaging (MRI), and accurately identifies finer drilling stages without relying on other sensors.

Recent 3D object detectors typically utilize multi-sensor data and unify multi-modal features in the shared bird’s-eye view (BEV) representation space. However, our empirical findings indicate that previous methods have limitations in generating fusion BEV features free from cross-modal conflicts. These conflicts encompass extrinsic conflicts caused by BEV feature construction and inherent conflicts stemming from heterogeneous sensor signals. Therefore, we propose a novel Eliminating Conflicts Fusion (ECFusion) method to explicitly eliminate the extrinsic/inherent conflicts in BEV space and produce improved multi-modal BEV features. Specifically, we devise a Semantic-guided Flow-based Alignment (SFA) module to resolve extrinsic conflicts via unifying spatial distribution in BEV space before fusion. Moreover, we design a Dissolved Query Recovering (DQR) mechanism to remedy inherent conflicts by preserving objectness clues that are lost in the fusion BEV feature. In general, our method maximizes the effective information utilization of each modality and leverages inter-modal complementarity. Our method achieves state-of-the-art performance in the highly competitive nuScenes 3D object detection dataset. The code is released at https://github.com/fjhzhixi/ECFusion.

Some cable-driven parallel robots (CDPRs) can be rapidly deployed on-site. To achieve such deployability, the fixed frame is usually substituted by four masts. However, not having any rigid fixture between the masts reduces the overall stiffness of the CDPR. This paper introduces a CDPR called Rocaspect, that has four compliant masts. The robot behavior and accuracy is evaluated experimentally and three different mast models are proposed.

A robot’s ability to anticipate the 3D action target location of a hand’s movement from egocentric videos can greatly improve safety and efficiency in human-robot interaction (HRI). While previous research predominantly focused on semantic action classification or 2D target region prediction, we argue that predicting the action target’s 3D coordinate could pave the way for more versatile downstream robotics tasks, especially given the increasing prevalence of headset devices. This study expands EgoPAT3D, the sole dataset dedicated to egocentric 3D action target prediction. We augment both its size and diversity, enhancing its potential for generalization. Moreover, we substantially enhance the baseline algorithm by introducing a large pre-trained model and human prior knowledge. Remarkably, our novel algorithm can now achieve superior prediction outcomes using solely RGB images, eliminating the previous need for 3D point clouds and IMU input. Furthermore, we deploy our enhanced baseline algorithm on a real-world robotic platform to illustrate its practical utility in straightforward HRI tasks. The demonstrations showcase the real-world applicability of our advancements and may inspire more HRI use cases involving egocentric vision. All code and data are open-sourced and can be found on the project website.

Solving a Task-and-Motion Planning (TAMP) problem can be represented as a sequential (meta-) decision process, where early decisions concern the skeleton (sequence of logic actions) and later decisions concern what to compute for such skeletons (e.g., action parameters, bounds, RRT paths, or full optimal manipulation trajectories). We consider the general problem of how to schedule compute effort in such hierarchical solution processes. More specifically, we introduce infinite completion trees as a problem formalization, where before we can expand or evaluate a node, we have to solve a preemptible computational sub-problem of a priori unknown compute effort. Infinite branchings represent an infinite choice of random initializations of computational sub-problems. Decision making in such trees means to decide on where to invest compute or where to widen a branch. We propose a heuristic to balance branching width and compute depth using polynomial level sets. We show completeness of the resulting solver and that a round robin baseline strategy used previously for TAMP becomes a special case. Experiments confirm the robustness and efficiency of the method on problems including stochastic bandits and a suite of TAMP problems, and compare our approach to a round robin baseline. An appendix comparing the framework to bandit methods and proposing a corresponding tree policy version is found on the supplementary webpage 1.

Depth-aware panoptic segmentation (DPS) combines image segmentation and monocular depth estimation in a single model to achieve semantic and geometry perception simultaneously. DPS task has important applications in the robot area but the previous DPS models are too heavy to be applied. Thus, we propose EfficientDPS, an efficient, end-to-end, and unified model for DPS. In our method, query features extracted with convolution networks are used to represent things/stuff. In this way, different vision tasks such as classification, segmentation, and depth estimation can be realized in a unified manner, leading to a compact and efficient model. EfficientDPS can be trained and tested in an end-to-end manner via bipartite matching and complex post-process is not needed at inference. To enhance the supervision signal, group query representation is proposed, leading to better performance without affecting the inference speed. Extensive experiments on Cityscapes-DPS and SemKITTI-DPS show that EfficientDPS can achieve the best trade-off between speed and accuracy than the state-of-the-art methods.