ICRA 2024 Papers

Soil is the most important building element of agriculture and its analysis is crucial for healthy plants and a high crop yield. But apart from its importance, soil analysis is a tedious and time-consuming task. This paper presents EdgeSoil 2.0, a non-invasive, accurate, and real-time robotic system for soil pH prediction, a key parameter of soil status for farmers. The EdgeSoil 2.0 predicts the pH value of the soil in real-time, using a live video stream from a webcam with an average of 7 FPS. The method is suitable to be implemented on edge devices necessary for the application: we are using a mobile robot with the NVIDIA Jetson Nano module which is running a pH-estimator trained with a Convolutional Neural Network (CNN) on a novel dataset we built for this purpose. Predictions are performed while the robot is moving over the plowed field before the planting process starts. In order to achieve the best performance, we train the pH-estimator with different input modalities and validate each result using Mean Squared Error (MSE) and Standard Deviation (SD). We are able to achieve accurate results with the MSE value of 0.08, the SD value of 0.15, and with testing results from the field showing up to ± 0.3 deviation from the GT value during prediction, which is sufficient to comply with agricultural standards.

Efficient interest point detection and description in images play a crucial role in many tasks such as multi-robot SLAM and collaborative localization. To facilitate fast detection and generate compact descriptions on edge devices, we introduce EdgePoint, a lightweight neural network. We design a new detection loss UnfoldSoftmax to improve inference speed. Futhermore, we propose Ortho-Alignment loss combined with LocalPCA compression to learn compact 32-dimensional descriptors. To enable efficient storage or communication, we also quantize the generated descriptors into integral values. We perform EdgePoint on various datasets, and show that it surpasses SuperPoint in performance while utilizing only 1% of the parameters and achieving up to more than 10 times faster inference speed. By applying descriptor quantization, the requirements for storage and communication can be reduced by up to 97% without performance decreasing.

Planning informative trajectories while considering the spatial distribution of the information over the environment, as well as constraints such as the robot’s limited battery capacity, makes the long-time horizon persistent coverage problem complex. Ergodic search methods consider the spatial distribution of environmental information while optimizing robot trajectories; however, current methods lack the ability to construct the target information spatial distribution for environments that vary stochastically across space and time. Moreover, current coverage methods dealing with battery capacity constraints either assume simple robot and battery models or are computationally expensive. To address these problems, we propose a framework called Eclares, in which our contribution is two-fold. 1) First, we propose a method to construct the target information spatial distribution for ergodic trajectory optimization using clarity, an information measure bounded between [0, 1]. The clarity dynamics allow us to capture information decay due to a lack of measurements and to quantify the maximum attainable information in stochastic spatiotemporal environments. 2) Second, instead of directly tracking the ergodic trajectory, we introduce the energy-aware (eware) filter, which iteratively validates the ergodic trajectory to ensure that the robot has enough energy to return to the charging station when needed. The proposed eware filter is applicable to nonlinear robot models and is computationally lightweight. We demonstrate the working of the framework through a simulation case study. [Code]a[Video]b

Emergent-scene safety is the key milestone for fully autonomous driving, and reliable on-time prediction is essential to maintain safety in emergency scenarios. However, these emergency scenarios are long-tailed and hard to collect, which restricts the system from getting reliable predictions. In this paper, we build a new dataset, which aims at the longterm prediction with the inconspicuous state variation in history for the emergency event, named the Extro-Spective Prediction (ESP) problem. Based on the proposed dataset, a flexible feature encoder for ESP is introduced to various prediction methods as a seamless plug-in, and its consistent performance improvement underscores its efficacy. Furthermore, a new metric named clamped temporal error (CTE) is proposed to give a more comprehensive evaluation of prediction performance, especially in time-sensitive emergency events of subseconds. Interestingly, as our ESP features can be described in human-readable language naturally, the application of integrating into ChatGPT also shows huge potential. The ESP-dataset and all benchmarks are released at https://dingrui-wang.github.io/ESP-Dataset/.

Mapping plays a crucial role in location and navigation within automatic systems. However, the presence of dynamic objects in 3D point cloud maps generated from scan sensors can introduce map distortion and long traces, thereby posing challenges for accurate mapping and navigation. To address this issue, we propose ERASOR++, an enhanced approach based on the Egocentric Ratio of Pseudo Occupancy for effective dynamic object removal. To begin, we introduce the Height Coding Descriptor, which combines height difference and height layer information to encode the point cloud. Subsequently, we propose the Height Stack Test, Ground Layer Test, and Surrounding Point Test methods to precisely and efficiently identify the dynamic bins within point cloud bins, thus overcoming the limitations of prior approaches. Through extensive evaluation on open-source datasets, our approach demonstrates superior performance in terms of precision and efficiency compared to existing methods. Furthermore, the techniques described in our work hold promise for addressing various challenging tasks or aspects through subsequent migration.

In autonomous driving, cooperative perception makes use of multi-view cameras from both vehicles and infrastructure, providing a global vantage point with rich semantic context of road conditions beyond a single vehicle viewpoint. Currently, two major challenges persist in vehicle-infrastructure cooperative 3D (VIC3D) object detection: 1) inherent pose errors when fusing multi-view images, caused by time asynchrony across cameras; 2) information loss in transmission process resulted from limited communication bandwidth. To address these issues, we propose a novel camera-based 3D detection framework for VIC3D task, Enhanced Multi-scale Image Feature Fusion (EMIFF). To fully exploit holistic perspectives from both vehicles and infrastructure, we propose Multi-scale Cross Attention (MCA) and Camera-aware Channel Masking (CCM) modules to enhance infrastructure and vehicle features at scale, spatial, and channel levels to correct the pose error introduced by camera asynchrony. We also introduce a Feature Compression (FC) module with channel and spatial compression blocks for transmission efficiency. Experiments show that EMIFF achieves SOTA on DAIR-V2X-C datasets, significantly outperforming previous early-fusion and late-fusion methods with comparable transmission costs.

High-frequency, low-latency, 6-DoF object tracking is useful for grasping objects in motion, taking robots beyond pick-and-place tasks. We propose using an event-camera for tracking the objects to leverage the low-latency and continuous (i.e. not fixed-rate) data capture for high-frequency tracking. We propose the EDOPT algorithm, which maintains real-time operation with a variable event-rate (which occurs due to variation in camera velocity and scene texture) and avoids frame-jumps and motion-blur which are problematic in traditional computer vision solutions. EDOPT uses a strong object prior, leading to a novel solution possible only with the event-camera. To our knowledge, this is the first method for 6-DoF object pose estimation with only the event-camera. The proposed method achieves comparable results to a state-of-the-art DNN technique that fuses frames, depth, and events. We demonstrate smooth, online object pose tracking with a live camera feed at > 300 Hz.

Classical motion planning for robotic manipulation includes a set of general algorithms that aim to minimize a scene-specific cost of executing a given plan. This approach offers remarkable adaptability, as they can be directly used off-the-shelf for any new scene without needing specific training datasets. However, without a prior understanding of what diverse valid trajectories are and without specially designed cost functions for a given scene, the overall solutions tend to have low success rates within a certain time limit. While deep-learning-based algorithms tremendously improve success rates, they are much harder to adopt without specialized training datasets. We propose EDMP, an Ensemble-of-costs-guided Diffusion for Motion Planning that aims to combine the strengths of classical and deep-learning-based motion planning. Our diffusion-based network is trained on a set of diverse kinematically valid trajectories. Like classical planning, for any new scene at the time of inference, we compute scene-specific costs such as "collision cost" and guide the diffusion to generate valid trajectories that satisfy the scene-specific constraints. Further, instead of a single cost function that may be insufficient in capturing diversity across scenes, we use an ensemble of costs to guide the diffusion process, significantly improving the success rate compared to classical planners. EDMP performs comparably with SOTA deep-learning-based methods while retaining the generalization capabilities primarily associated with classical planners.

Robot designs can take many inspirations from nature, where there are many examples of highly resilient and fault-tolerant locomotion strategies to navigate complex terrains by using multi-functional appendages. For example, Chukar and Hoatzin birds can repurpose their wings for quadrupedal walking and wing-assisted incline running (WAIR) to climb steep surfaces. We took inspiration from nature and designed a morphing robot with multi-functional thruster-wheel appendages that allows the robot to change its mode of locomotion by transforming into a rover, quad-rotor, mobile inverted pendulum (MIP), and other modes. In this work, we derive a dynamic model and formulate a nonlinear model predictive controller to perform WAIR to showcase the unique capabilities of our robot. We implemented the model and controller in a numerical simulation and experiments to show their feasibility and the capabilities of our transforming multimodal robot.

We present and evaluate a novel suction-based gripper designed for fruit picking. This work is motivated by common problems observed in field trials of robotic harvesting: Calibration/perception errors, workspace obstacles, fruit swinging/moving when contacted, and varying stem and branch stiffnesses. The gripper consists of three suction-cups located on the palm, along with in-hand perception. To evaluate the gripper, we developed a physical proxy that approximates a realistic apple-stem-branch dynamic system. We performed 756 apple picks on the proxy with varying branch stiffness, stem strength and gripper pose (yaw, roll and offset w.r.t. the apple). Our results show that grasping performance improves when the gripper yaw w.r.t. the apple has two suction cups on the bottom of the apple and one suction cup on top. Even with ±15mm offset, at least two suction cups engaged with the apple 80% of the time, regardless of branch stiffness. Moreover, the gripper withstands ±20mm offset when it approaches the apple near its equator.

Maximizing the utility of limited Earth observing satellite resources is a difficult ongoing problem. Dynamic Targeting is an approach to this challenge that intelligently plans and executes primary sensor observations based on information from a look-ahead sensor. However, current implementations have failed to account for realistic satellite operational constraints and have used static utility for repeat observations of the same target. To address these limitations, we implement a more general Dynamic Targeting framework that comprises a physics-based slew model, a dynamic model of observation utility, and an algorithm for gathering high-utility observations. To demonstrate this framework, we also supply complex dynamic utility models that are applicable to many missions and new algorithms for intelligently scheduling observations with slewing restrictions and changing utility, including a greedy algorithm and a depth-first search algorithm. To evaluate these algorithms, we test their performance across simulated runs through two datasets and compare to the performance of an algorithm representative of most scheduling algorithms aboard Earth science missions today as well as an intractable upper bound. We show that our algorithms have great potential to improve science return from Earth science missions.

Dynamic Occupancy Grid Mapping is a technique used to generate a local map of the environment, containing both static and dynamic information. Typically, these maps are primarily generated using lidar measurements. However, with improvements in radar sensing, resulting in better accuracy and higher resolution, radar is emerging as a viable alternative to lidar as the primary sensor for mapping. In this paper, we propose a radar-centric dynamic occupancy grid mapping algorithm with adaptations to the state computation, inverse sensor model, and field-of-view computation tailored to the specifics of radar measurements. We extensively evaluate our approach with real data to demonstrate its effectiveness and establish the first benchmark for radar-based dynamic occupancy grid mapping using the publicly available Radarscenes dataset.

The reconfigurable unmanned aerial vehicle (RUAV) has the ability to create and break physical links to self-assemble and self-disassemble in midair. For the changes in task or environment, this system can dynamically disassemble the rectangular structure into multiple individual UAV modules or integrate these UAV modules into a whole. For practical applications, the R-UAV requires collaborative decision-making for autonomous navigation in complex environments. However, the navigation problem of the R-UAV has not been investigated. In this paper, we propose a dynamic multi-agent deep deterministic policy gradient (DMADDPG) algorithm for autonomous navigation of R-UAV. This algorithm introduces the leader agent assignment mechanism and a collaborative experience reward. The former deals with the action conflict problem caused by the disappearance of the UAV agent when multiple UAV modules are assembled. The latter provides guidance for the UAV agent to plan a collision-free and efficient trajectory. We validate our strategy in both simulation and practical scenarios, and experimental results demonstrate that the proposed scheme can generate reasonable and efficient paths for R-UAV in the presence of obstacles. The experiment video is available at https://youtu.be/mVm0qCvB7HY.

Legged mobile manipulators are receiving much more attention. Mobile platforms can infinitely expand the workspace of robotic arms, providing more possibilities for robot application scenarios. Compared with wheeled mobile manipulators, legged mobile manipulators have higher requirements for cooperative control of legged robots and robotic arms. This work decouples the control of the robotic arm and the legged robot. On the legged robot side, we explicitly estimate the wrench exerted by the robotic arm on the base and bring it into the legged robot’s dynamics, and then use a nonlinear model predictive controller (NMPC) to control the legged robot. On the robotics arm side, we adopt an impedance controller to realize the end-effector’s force control, and the introduction of impedance control has improved the safety and interactivity of legged mobile manipulators. We conducted experiments on physical robot to compare the differences between decoupled control and independent control, and the results show that the stability and robustness of robot systems have improved using decoupled control.

This paper tackles the dynamic crane scheduling problem in a steel coil warehouse, involving tasks such as coil storage, retrieval, and shuffling. Tasks arrive dynamically with precedence relations, while multiple cranes share a track, necessitating collision avoidance. We aim to minimize the average task waiting time by allocating tasks to cranes and optimizing their execution sequence. Unlike prior research focusing on static scenarios or rule-based heuristics, we introduce a real-time, reinforcement learning-based algorithm. We propose a policy network based on graph neural networks to effectively handle precedence relations and global information. Experimental results demonstrate its superiority over traditional heuristics, such as dispatching rules in dynamic scenarios.

Many multi-robot deployments, such as automated construction of buildings, distributed search, or cooperative mapping, often require agents to intelligently coordinate their trajectories and form coalition over a large domain, to complete spatially distributed tasks as quickly as possible. We focus on scenarios involving homogeneous robots, but where tasks vary in the number of agents required to start them. For example, construction robots may need to collaboratively air-lift heavy objects at different locations (e.g., prefabricated rooms, crates of material/equipment), where the weight of each payload defines the required coalition size. To balance the total travel time of the agents and their waiting time (before task initiation), agents need to carefully sequence tasks but also dynamically form/disband coalitions. While simpler problems can be approached using heuristics or optimization, these methods struggle with more complex instances involving large task-to-agent ratios, where frequent coalition changes are needed. In this work, we propose to let agents learn to iteratively build cooperative schedules to solve such problems, by casting the problem in the reinforcement learning framework. Our approach relies on an attention-based neural network, allowing agents to reason about the current state of the system to sequence movement decisions that optimize short-term coalition formation and long-term task scheduling. We further propose a novel leader-follower technique to boost cooperation learning and compare our performance to conventional baselines in a wide variety of scenarios. There, our method closely matches or outperforms the baselines; in particular, it yields higher-quality solutions and is at least 2 orders of magnitude faster than exact solver in cases where frequent coalition updates are required.

Optoelectronic tweezers (OET) has shown great promise in various applications, especially in the precise manipulation of microparticles and microorganisms on a micron and nanometer scale. This technology significantly enhances the efficiency of single-cell sorting and the development of antibody-based drugs. However, conventional OET platforms are limited by issues such as low autofocusing accuracy, restricted imaging field of view, and uneven illumination. To overcome these limitations, we have innovatively developed a dynamic adaptive imaging system. By incorporating peak-finding and in situ Gaussian blur compensation algorithms, we achieved rapid automatic focusing and illumination shadow compensation across an expanded field of view. At the same time, the system can also dynamically adjust compensation parameters under different lighting conditions. Our system has successfully completed comprehensive scanning of the optoelectronic tweezers chip, achieving a 60% reduction in autofocus time and a 15.8% improvement in lighting uniformity. Moreover, this imaging system demonstrates robust versatility and can serve as a reference for other optical systems.

Dynamic objects diversify the distribution of point cloud in the map, degrading the performance of the robotic downstream tasks. To address this problem, we present a novel real-time dynamic instance-aware static mapping framework called DynaInsRemover, which exploits the geometric discrepancies between instances to efficiently remove dynamic objects and preserve more details of static map. It contains the Instance Occupancy Check module for initial dynamic instance proposal and the Instance Belief Update module for reverting false positives. We quantitatively evaluate our approach performance on the SemanticKITTI dataset and validate it in a real-world environment. Experimental evaluations show that our method achieves very promising results in dynamic environments. The implementation of our method is available as open source at: https://github.com/Zhaohuanfeng/DynaInsRemover.git.

Modeling the interactions among agents based on their historical trajectories is key to precise multi-agent trajectory prediction. Hypergraph Convolutional Networks (HGCN) have become a proper choice for capturing high-order interactions among agents in this field. However, most existing works only consider static hypergraphs, and ignore that in a hypergraph, the power of influence varies between vertices (or hyperedges). Therefore, we propose DyHGDAT, a dynamic hypergraph dual attention network to capture the high-order interactions among agents, which not only models the evolution of hypergraph over time but also highlights the vertices and hyperedges with larger impacts. We apply DyHGDAT to a CVAE-based prediction system for predicting plausible trajectories. To validate the effectiveness of prediction, we evaluate our proposed method on two well-established trajectory prediction datasets: the ETH/UCY datasets and the Stanford Drone Dataset (SDD). The experimental results show that with DyHGDAT, the CVAE-based prediction system out-performs state-of-the-art methods by 12.5%/5.3% in ADE/FDE on ETH/UCY, and the improvement on SDD is 6.4%/7.4%.