ICRA 2024 论文列表

Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments. We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework. We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds and an object discovery and tracking engine. Our approaches can perform multiple challenging tasks including reasoning with occluded objects, novel objects appearance, and object reappearance. Throughout our extensive simulation and real-world experiments, we find that our approaches perform well in terms of different numbers of objects and different numbers of distractor actions. Furthermore, we show our approaches outperform an implicit memory baseline.

In this work, we achieved a self-takeoff of an eagle-scale flapping-wing robot for the first time. Inspired by the takeoff process of Ospreys, we propose a bio-inspired takeoff strategy, then discuss the dynamic model and the requirements for self-takeoff. Based on the requirements of flight strategy, we designed a system with two parts, including a flapping-wing aircraft with a wingspan of 1.8m and a take-off weight of 870g, and an auxiliary platform with an initial pitch angle adjustment function. In order to explore the differences in the take-off process under different conditions, we conduct the flight experiments under different time-averaged thrust-to-weight ratios (0.745-0.876) and launch angles (45°-90°). The results of flight experiments confirmed the theoretical analysis that the flapping-wing robot can achieve self-takeoff with no potential energy cost and maintain high maneuverability (The video shows a rapid climb immediately after takeoff) even when the time-averaged thrust-to-weight ratio is smaller than 1. This is significantly different from conventional rotary-wing and vertical take-off and landing (VTOL) UAVs. This work solves the challenge of self-takeoff for large-scale flapping-wing robots using a designable method and demonstrates the superior performance potential of flapping-wing robots compared to conventional UAVs.

3D scene graphs have recently emerged as a powerful and human-understandable way of representing complex 3D environments. These describe environments through a layered or hierarchical graph where nodes represent different spatial concepts (from low-level geometry to higher-level scene-scale reasoning) and the edges between them represent relationships. While these representations have shown great promise in indoor well-structured environments, their use in outdoor structured environments such as agricultural environments has been under-explored. A key challenge here is that concepts and structures often observed in urban indoor environments cannot be easily transferred to these novel scenes.Motivated by this challenge, this paper presents Osiris which is a 3D scene graph builder for agricultural environments. We first propose a structure of the hierarchical graph for agricultural environments consisting of rowed crops and through our proposed system Osiris incrementally construct a 3D scene graph of agricultural environments from data taken onboard a mobile robot. We validate and evaluate the performance of Osiris using real-world data collected at several farms and show that this system is able to accurately get to the underlying structure of these agricultural environments while presenting a metrically accurate and human-understandable representation.

Accurate identification of road intersections is the pivotal task for automatic construction of high-definition maps, particularly in unstructured scenes. Existing methods predominantly rely on single-modal data and thus show an obvious unimodal limitation, i.e., lack of contextual information. Moreover, these approaches overlook the benefits of leveraging multi-modal data fusion and representation learning that is crucial for generalizability. To this end, we propose a novel orientation-aware multi-modal learning paradigm, which formulates intersection identification as an oriented object detection task. Specifically, heterogeneous fusion is introduced to harmonize disparate data modalities, i.e., vector maps, point clouds, and vehicle trajectories, into a unified feature space. Concurrently, we present trigonometry-induced adaptive regression to elevate orientation estimation, while mitigating issues related to scale imbalance and boundary confusion through dual-objective matching with spatial adaptation. To evaluate our methodology, we assemble the first-of-its-kind multi-modal benchmark tailored for complex low-speed environments, complete with fine-grained semantic annotations for intersections. Comprehensive empirical analyses, including ablation studies, affirm both the superior performance of our proposed framework and the efficacy of its constituent modules.

In response to the need for sustainable energy solutions, solar panels have gained significant traction. One way to increase the energy capture of solar systems is through solar tracking, a means of reorienting solar panels throughout the day in order to face the sun. The energy consumption increase that comes with solar tracking often far outweighs the amount of energy required to move the panel, which makes it a compelling strategy for improving solar systems. Unfortunately, while solar trackers are commonly used in large solar farms, they are rarely used on rooftops, an area where solar panels are commonly installed. This is for two primary reasons: (1) most commercially available solar trackers are too large to be installed on roofs and (2) even if traditional solar trackers were made in a more compact form-factor it would be difficult to densely lay them out on a roof without the trackers substantially shading each other. In order to address these issues, we introduce OriTrack, a small three-degree-of-freedom (3 DOF) solar tracker which reduces the area of its shadow by reducing its height as it tracks the sun. In this paper we discuss the design, manufacturing, and control of OriTrack. We then compare OriTrack to a flat reference panel, the solar energy solution commonly used on roofs today, and find that OriTrack demonstrates 23% increased energy production. This result suggests OriTrack could be used as a future solution for solar tracking on rooftops.

Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present Orbit-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. Orbit-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. Orbit-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate Orbit-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot.Project website: orbit-surgical.github.io

Tactile and textile skin technologies have become increasingly important for enhancing human-robot interaction and allowing robots to adapt to different environments. Despite notable advancements, there are ongoing challenges in skin signal processing, particularly in achieving both accuracy and speed in dynamic touch sensing. This paper introduces a new framework that poses the touch sensing problem as an estimation problem of resistive sensory arrays. Utilizing a Regularized Least Squares objective function—which estimates the resistance distribution of the skin—we enhance the touch sensing accuracy and mitigate the ghosting effects, where false or misleading touches may be registered. Furthermore, our study presents a streamlined skin design that simplifies manufacturing processes without sacrificing performance. Experimental outcomes substantiate the effectiveness of our method, showing 26.9% improvement in multi-touch force-sensing accuracy for the tactile skin.

Industrial robots are designed as general-purpose hardware with limited ability to adapt to changing task requirements or environments. Modular robots, on the other hand, offer flexibility and can be easily customized to suit diverse needs. The morphology, i.e., the form and structure of a robot, significantly impacts the primary performance metrics acquisition cost, cycle time, and energy efficiency. However, identifying an optimal module composition for a specific task remains an open problem, presenting a substantial hurdle in developing task-tailored modular robots. Previous approaches either lack adequate exploration of the design space or the possibility to adapt to complex tasks. We propose combining a genetic algorithm with a lexicographic evaluation of solution candidates to overcome this problem and navigate search spaces exceeding those in prior work by magnitudes in the number of possible compositions. We demonstrate that our approach outperforms a state-of-the-art baseline and is able to synthesize modular robots for industrial tasks in cluttered environments.

Balancing oneself using the spine is a physiological alignment of the body posture in the most efficient manner by the muscular forces for mammals. For this reason, we can see many disabled quadruped animals can still stand or walk even with three limbs. This paper investigates the optimization of dynamic balance during trot gait based on the spatial relationship between the center of mass (CoM) and support area influenced by spinal flexion. During trotting, the robot balance is significantly influenced by the distance of the CoM to the support area formed by diagonal footholds. In this context, lateral spinal flexion, which is able to modify the position of footholds, holds promise for optimizing balance during trotting. This paper explores this phenomenon using a rat robot equipped with a soft actuated spine. Based on the lateral flexion of the spine, we establish a kinematic model to quantify the impact of spinal flexion on robot balance during trot gait. Subsequently, we develop an optimized controller for spinal flexion, designed to enhance balance without altering the leg locomotion. The effectiveness of our proposed controller is evaluated through extensive simulations and physical experiments conducted on a rat robot. Compared to both a non-spine based trot gait controller and a trot gait controller with lateral spinal flexion, our proposed optimized controller effectively improves the dynamic balance of the robot and retains the desired locomotion during trotting.

In recent years, bio-syncretic robots actuated by living materials have received widespread attention. Among the common living materials, engineered skeletal muscle tissue (eSKT) has been the focus of researchers due to its high contraction force and good controllability. However, the current performance of eSKT is far from that of natural skeletal muscle tissue. In this paper, an optimized design method for eSKTs has been proposed. By combining simulation analysis with experiments, the eSKTs with multiple strips have been developed. The results show that under a specific volume (250 μL), the optimized strip structures can enhance the stability of eSKT and facilitate the penetration of nutrients and oxygen, leading to improved fusion of myoblasts and the directional arrangement of myotubes, thus improving the performance of eSKT. The eSKT with multiple strips exhibits a significant contraction force and has been successfully utilized in a bio-syncretic robot to demonstrate its actuation capability. This work may provide insights into the development of the field of bio-syncretic robots and even tissue engineering.

This work presents a novel shape evaluation and optimization approach for shape sensing, specifically targeting the constrained, irregular, and intricate spatial shapes of flexible bronchoscopes (FB) in human bronchial tree. The proposed evaluation criteria and optimization methods combine clinical significance related to bronchial anatomical structures and address issues related to singular points and discontinuities in traditional shape reconstruction models. Three-dimensional experiments were conducted within eight spatial complex configurations printed from a proportional bronchial model. The 3D experiment results demonstrate an average reduction of approximately 34.1% in shape reconstruction errors across all eight airway models compared to the traditional model, validating the effectiveness and feasibility.

Robot swarms can be tasked with a variety of automated sensing and inspection applications in aerial, aquatic, and surface environments. In this paper, we study a simplified two-outcome surface inspection task. We task a group of robots to inspect and collectively classify a 2D surface section based on a binary pattern projected on the surface. We use a decentralized Bayesian decision-making algorithm and deploy a swarm of 3-cm sized wheeled robots to inspect a randomized black and white tiled surface section of size 1m×1m in simulation. We first describe the model parameters that characterize our simulated environment, the robot swarm, and the inspection algorithm. We then employ a noise-resistant heuristic optimization scheme based on the Particle Swarm Optimization (PSO) using a fitness evaluation that combines the swarm’s classification decision accuracy and decision time. We use our fitness measure definition to asses the optimized parameters through 100 randomized simulations that vary surface pattern and initial robot poses. The optimized algorithm parameters show up to 55% improvement in median of fitness evaluations against an empirically chosen parameter set.

Robot skateboarding is a novel and challenging task for legged robots. Accurately modeling the dynamics of dual floating bases and developing effective planning and control methods present significant complexities in accomplishing skateboarding behavior. This paper focuses on enabling the quadrupedal platform CyberDog2 to achieve dynamic balancing and acceleration on a skateboard. An optimization-based control pipeline is developed through careful derivation of the system’s equations of motion, considering both the robot and skateboard dynamics. By accounting for system physical constraints, an advanced offline trajectory optimization method is employed to generate various acceleration trajectories, creating a motion library for the system. An online linear model predictive control with whole body control framework is used to track the generated trajectories and stablize the system in real-time. To validate its effectiveness, we conducted experiments in various scenarios. The quadrupedal robot successfully performed acceleration from a static state to various velocities and demonstrated the ability to balance and steer the skateboard.

This paper presents a novel approach to optimal multi-robot task allocation in heterogeneous teams of robots. When robots have heterogeneous capabilities and there are diverse objectives and constraints to comply with, computing optimal plans can become especially hard. Moreover, we increase the problem complexity by: 1) considering battery-limited robots that need to schedule recharges; 2) tasks that can be decomposed into multiple fragments; and 3) multi-robot tasks that need to be executed by a coalition synchronously. We define a new problem for heterogeneous multi-robot task allocation and formulate it as a Mixed-Integer Linear Program that includes all the aforementioned features. Then we use an off-the-shelf solver to show the type of optimal solutions that our planner can produce and assess its performance in random scenarios. Our method, which is released as open-source code, represents a first step to formalize and analyze a complex problem that has not been solved in the state of the art.

Recent advances in metric, semantic, and topological mapping have equipped autonomous robots with concept grounding capabilities to interpret natural language tasks. Leveraging these capabilities, this work develops an efficient task planning algorithm for hierarchical metric-semantic models. We consider a scene graph model of the environment and utilize a large language model (LLM) to convert a natural language task into a linear temporal logic (LTL) automaton. Our main contribution is to enable optimal hierarchical LTL planning with LLM guidance over scene graphs. To achieve efficiency, we construct a hierarchical planning domain that captures the attributes and connectivity of the scene graph and the task automaton, and provide semantic guidance via an LLM heuristic function. To guarantee optimality, we design an LTL heuristic function that is provably consistent and supplements the potentially inadmissible LLM guidance in multi-heuristic planning. We demonstrate efficient planning of complex natural language tasks in scene graphs of virtualized real environments.

In this paper, we propose a reactive planning system for quadruped robots based on prescribed-time control. The navigation of the quadruped robot is fundamentally depicted as omnidirectional movements, while a feedback control law is formulated to address any deviations the robot may encounter. In particular, our proposed feedback control system is theoretically proven to achieve convergence within a predefined finite time that is specified by the user. To further compute the optimal convergent time and the local goal state, we present a high-level planning node encompassing terrain-aware kinodynamic search and spatiotemporal trajectory optimization, which can generate collision-free, smooth, and efficient trajectories. The effectiveness of our proposed framework is validated through both numerical simulation and real-robot experiments in indoor and outdoor environments, including scenarios with cluttered obstacles, slopes, and external disturbances.

This paper addresses the challenge of planning a sequence of tasks to be performed by multiple robots while minimizing the overall completion time subject to timing and precedence constraints. Our approach uses the Timed Partial Orders (TPO) model to specify these constraints. We translate this problem into a Traveling Salesman Problem (TSP) variant with timing and precedent constraints, and we solve it as a Mixed Integer Linear Programming (MILP) problem. Our contributions include a general planning framework for TPO specifications, a MILP formulation accommodating time windows and precedent constraints, its extension to multi-robot scenarios, and a method to quantify plan robustness. We demonstrate our framework on several case studies, including an aircraft turnaround task involving three Jackal robots, highlighting the approach’s potential applicability to important real-world problems. Our benchmark results show that our MILP method outperforms state-of-the-art open-source TSP solvers OR-Tools.

The driver warning system that alerts the human driver about potential risks during driving is a key feature of an advanced driver assistance system. Existing driver warning technologies, mainly the forward collision warning and unsafe lane change warning, can reduce the risk of collision caused by human errors. However, the current design methods have several major limitations. Firstly, the warnings are mainly generated in a one-shot manner without modeling the ego driver’s reactions and surrounding objects, which reduces the flexibility and generality of the system over different scenarios. Additionally, the triggering conditions of warning are mostly rule-based threshold-checking given the current state, which lacks the prediction of the potential risk in a sufficiently long future horizon. In this work, we study the problem of optimally generating driver warnings by considering the interactions among the generated warning, the driver behavior, and the states of ego and surrounding vehicles on a long horizon. The warning generation problem is formulated as a partially observed Markov decision process (POMDP). An optimal warning generation framework is proposed as a solution to the proposed POMDP. The simulation experiments demonstrate the superiority of the proposed solution to the existing warning generation methods.

The control of granular materials, which are found in many industrial applications, is a challenging open research problem. Granular material systems are complex-behavior (as they could have solid-, fluid-, and gas-like behaviors) and high-dimensional (as they could have many grains/particles with at least 3 DOF in 3D) systems. Recently, a machine learning-based Graph Neural Network (GNN) simulator has been proposed to learn the underlying dynamics. In this paper, we perform optimal control of a rigid body-driven granular material system whose dynamics is learned by a GNN model trained by reduced data generated via a physics-based simulator and Principal Component Analysis (PCA). We use Differential Dynamic Programming (DDP) to obtain optimal control commands that can form granular particles into a target shape. The model and results are shown to be relatively fast and accurate. The control commands are also applied to the ground truth model, i.e., physics-based simulator, to further validate the approach.