ICRA 2024 论文列表

In this paper, we present a novel point generation model, referred to as Pillar-based Point Generation Network (PillarGen), which facilitates the transformation of point clouds from one domain into another. PillarGen can produce synthetic point clouds with enhanced density and quality based on the provided input point clouds. The PillarGen model performs the following three steps: 1) pillar encoding, 2) Occupied Pillar Prediction (OPP), and 3) Pillar to Point Generation (PPG). The input point clouds are encoded using a pillar grid structure to generate pillar features. Then, OPP determines the active pillars used for point generation and predicts the center of points and the number of points to be generated for each active pillar. PPG generates the synthetic points for each active pillar based on the information provided by OPP. We evaluate the performance of PillarGen using our proprietary radar dataset, focusing on enhancing the density and quality of short-range radar data using the long-range radar data as supervision. Our experiments demonstrate that PillarGen outperforms traditional point upsampling methods in quantitative and qualitative measures. We also confirm that when PillarGen is incorporated into bird’s eye view object detection, a significant improvement in detection accuracy is achieved.

Constrained Motion Planning (CMP) aims to find a collision-free path between the given start and goal configurations on the kinematic constraint manifolds. These problems appear in various scenarios ranging from object manipulation to legged-robot locomotion. However, the zero-volume nature of manifolds makes the CMP problem challenging, and the state-of-the-art methods still take several seconds to find a path and require a computationally expansive path dataset for imitation learning. Recently, physics-informed motion planning methods have emerged that directly solve the Eikonal equation through neural networks for motion planning and do not require expert demonstrations for learning. Inspired by these approaches, we propose the first physics-informed CMP framework that solves the Eikonal equation on the constraint manifolds and trains neural function for CMP without expert data. Our results show that the proposed approach efficiently solves various CMP problems in both simulation and real-world, including object manipulation under orientation constraints and door opening with a high-dimensional 6-DOF robot manipulator. In these complex settings, our method exhibits high success rates and finds paths in sub-seconds, which is many times faster than the state-of-the-art CMP methods.

Sophisticated models can accurately describe deformations of continuum robots while being computationally demanding, which limits their application. Especially when considering sampling-based path planning, the model has to be evaluated frequently, which can lead to substantially increased computation times. We present a new approach to compute the entire shape of a tendon-driven continuum robot by a physics-informed neural network (PINN). The underlying physics is modelled with the Cosserat rod theory and incorporated into the PINN’s loss function. The boundary values for the training are obtained from a reference model, solved by the shooting method. Our approach allows for a computation of the learned Cosserat rod model multiple orders of magnitude faster than a publicly available reference model. The median position deviation from the reference model lies below 1mm (0.5% of the simulated robot length) for each of the robot’s 20 disks.

Recent advances in aerial robotics have enabled the use of multirotor vehicles for autonomous payload transportation. Resorting only to classical methods to reliably model a quadrotor carrying a cable-slung load poses significant challenges. On the other hand, purely data-driven learning methods do not comply by design with the problem’s physical constraints, especially in states that are not densely represented in training data. In this work, we explore the use of physics-informed neural networks to learn an end-to-end model of the multirotor-slung-load system and, at a given time, estimate a sequence of the future system states. An LSTM encoder-decoder with an attention mechanism is used to capture the dynamics of the system. To guarantee the cohesiveness between the multiple predicted states of the system, we propose the use of a physics-based term in the loss function, which includes a discretized physical model derived from first principles together with slack variables that allow for a small mismatch between expected and predicted values. To train the model, a dataset using a real-world quadrotor carrying a slung load was curated and is made available. Prediction results are presented and corroborate the feasibility of the approach. The proposed method outperforms both the first principles physical model and a comparable neural network model trained without the physics regularization proposed.

In robotics, it’s crucial to understand object deformation during tactile interactions. A precise understanding of deformation can elevate robotic simulations and have broad implications across different industries. We introduce a method using Physics-Encoded Graph Neural Networks (GNNs) for such predictions. Similar to robotic grasping and manipulation scenarios, we focus on modeling the dynamics between a rigid mesh contacting a deformable mesh under external forces. Our approach represents both the soft body and the rigid body within graph structures, where nodes hold the physical states of the meshes. We also incorporate cross-attention mechanisms to capture the interplay between the objects. By jointly learning geometry and physics, our model reconstructs consistent and detailed deformations. We’ve made our code and dataset public to advance research in robotic simulation and grasping.†

Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PHYSOBJECTS, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.

Grasp Quality Networks are important components of grasping-capable autonomous robots, as they allow them to evaluate grasp candidates and select the one with highest chance of success. The widespread use of pick-and-place robots and Grasp Quality Networks raises the question of whether such systems are vulnerable to adversarial attacks, as that could lead to large economic damage. In this paper we propose two kinds of attacks on Grasp Quality Networks, one assuming physical access to the workspace (to place or attach a new object) and another assuming digital access to the camera software (to inject a pixel-intensity change on a single pixel). We then use evolutionary optimization to obtain attacks that simultaneously minimize the noticeability of the attacks and the chance that selected grasps are successful. Our experiments show that both kinds of attack lead to drastic drops in algorithm performance, thus making them important attacks to consider in the cybersecurity of grasping robots. Source code can be found at https://github.com/Naif-W-Alharthi/Physical-and-Digital-Attacks-on-Grasping-Networks

3D neural implicit representations play a significant component in many robotic applications. However, reconstructing neural radiance fields (NeRF) from realistic event data remains a challenge due to the sparsities and the lack of information when only event streams are available. In this paper, we utilize motion, geometry, and density priors behind event data to impose strong physical constraints to augment NeRF training. The proposed novel pipeline can directly benefit from those priors to reconstruct 3D scenes without additional inputs. Moreover, we present a novel density-guided patch-based sampling strategy for robust and efficient learning, which not only accelerates training procedures but also conduces to expressions of local geometries. More importantly, we establish the first large dataset for event-based 3D reconstruction, which contains 101 objects with various materials and geometries, along with the groundtruth of images and depth maps for all camera viewpoints, which significantly facilitates other research in the related fields. The code and dataset will be publicly available at https://github.com/Mercerai/PAEv3d.

Reviewing the previous work of diversity Reinforcement Learning, diversity is often obtained via an augmented loss function, which requires a balance between reward and diversity. Generally, diversity optimization algorithms use Multi-armed Bandits algorithms to select the coefficient in the pre-defined space. However, the dynamic distribution of reward signals for MABs or the conflict between quality and diversity limits the performance of these methods. We introduce the Phasic Diversity Optimization (PDO) algorithm, a Population-Based Training framework that separates reward and diversity training into distinct phases instead of optimizing a multi-objective function. In the auxiliary phase, agents with poor performance diversified via determinants will not replace the better agents in the archive. The decoupling of reward and diversity allows us to use an aggressive diversity optimization in the auxiliary phase without performance degradation. Furthermore, we construct a dogfight scenario for aerial agents to demonstrate the practicality of the PDO algorithm. We introduce two implementations of PDO archive and conduct tests in the newly proposed adversarial dogfight and MuJoCo simulations. The results show that our proposed algorithm achieves better performance than baselines.

Retractable worm robots possess hyper-flexibility, allowing them to work in confined spaces that are difficult for humans. However, the spatial locomotion control of these robots remains challenging due to the robots’ large degrees of freedom. To address this challenge, we propose a phase synthesis (PS) scheme for retractable worm robots. The scheme combines an undulating gait inspired by caterpillars with three-dimensional movement commands. We first introduce the kinematics model and real-world prototype of our retractable worm robot, called RW-Robot, and then we introduce footstep phases to express the timing of segments’ spatial movement. According to the length of movement periods, we classify the movement into short-term movements and long-term movements and compress their patterns in the frequency domain. Our PS scheme aligns the patterns according to the footstep phases to generate new gaits of spatial locomotion. We evaluate the scheme in real-world experiments, including steering and climbing a slope. The experimental results indicate that our scheme allows the RW-Robot to perform flexible spatial locomotion from simple user input.

The synergic orchestration of the cognitive and psychological dimensions characterizes human intelligence. Accordingly, carefully designing this mechanism in artificial intelligence can be a successful strategy to increase human likeness in a robot, enhancing mutual understanding and building a more natural and intuitive interaction. For this purpose, the main contribution of this work is a psychological and cognitive architecture tailored for HRI based on the interplay between robotic personality and memory-based cognitive processes. Indeed, the artificial personality manifests itself not only in various aspects of the behavior but also within the action selection process, which is closely intertwined with personality-dependent hedonic experiences linked to memories. Within this paper, we propose a task- and platform-independent framework, evaluated in a multiparty collaborative scenario. Obtained results show that a robot connected to our proposed framework is perceived as a cognitive agent capable of manifesting perceivable and distinguishable personality traits.

This paper proposes a novel decision-making framework for planning "when" and "where" to deploy robots based on prior data with the goal of persistently monitoring a spatio-temporal phenomenon in an environment. We specifically focus on large lake monitoring, where remote sensors, such as satellites, can provide a snapshot of the target phenomenon at regular cycles. Between these cycles, Autonomous Surface Vehicles (ASVs) can be deployed to maintain an up-to-date model of the phenomenon. However, deploying ASVs has a significant logistical overhead in terms of time and cost. It requires a team of people to go on site and spend typically a day to monitor the deployment. It is vital to not only be intentional about where to sample in the environment on a given day, but also determine the worth of deploying the ASVs that day at all. Therefore, we propose a persistent monitoring strategy that provides the days and locations of when and where to sample with the robots by leveraging Gaussian Process model estimates of future trends based on collected remote sensing and point measurement data. Our approach minimizes the number of days and locations for sampling, while preserving the quality of estimates. Through simulation experiments using realistic spatio-temporal datasets, we demonstrate the benefits of our approach over traditional deployment strategies, including significant savings on the effort and operational cost of deploying the ASVs.

Accurately assessing the potential value of new sensor observations is a critical aspect of planning for active perception. This task is particularly challenging when reasoning about high-level scene understanding using measurements from vision-based neural networks. Due to appearance-based reasoning, the measurements are susceptible to several environmental effects such as the presence of occluders, variations in lighting conditions, and redundancy of information due to similarity in appearance between nearby viewpoints. To address this, we propose a new active perception framework incorporating an arbitrary number of perceptual effects in planning and fusion. Our method models the correlation with the environment by a set of general functions termed perceptual factors to construct a perceptual map, which quantifies the aggregated influence of the environment on candidate viewpoints. This information is seamlessly incorporated into the planning and fusion processes by adjusting the uncertainty associated with measurements to weigh their contributions. We evaluate our perceptual maps in a simulated environment that reproduces environmental conditions common in robotics applications. Our results show that, by accounting for environmental effects within our perceptual maps, we improve the state estimation by correctly selecting the viewpoints and considering the measurement noise correctly when affected by environmental factors. We furthermore deploy our approach on a ground robot to showcase its applicability for real-world active perception missions.

Global navigation satellite systems (GNSS) denied environments/conditions require unmanned aerial vehicles (UAVs) to energy-efficiently and reliably fly. To this end, this study presents perception-and-energy-aware motion planning for UAVs in GNSS-denied environments. The proposed planner solves the trajectory planning problem by optimizing a cost function consisting of two indices: the total energy consumption of a UAV and the perception quality of light detection and ranging (LiDAR) sensor mounted on the UAV. Before online navigation, a high-fidelity simulator acquires a flight dataset to learn energy consumption for the UAV and heteroscedastic uncertainty associated with LiDAR measurements, both as functions of the horizontal velocity of the UAV. The learned models enable the online planner to estimate energy consumption and perception quality, reducing UAV battery usage and localization errors. Simulation experiments in a photorealistic environment confirm that the proposed planner can address the trade-off between energy efficiency and perception quality under heteroscedastic uncertainty. The open-source code is released at https://gitlab.com/ReI08/perception-energy-planner.

This paper presents the design and analysis of Pegasus, a quadrupedal wheeled robot grounded in biomimicry principles. Pegasus offers two distinct motion modes, including a wheeled motion and a hybrid wheeled-legged motion, enabling adaptability across various tasks and environmental conditions. The robot draws inspiration from the joint structures of quadruped animals and incorporates biomimetic features. At the robot’s ankle joint, we imitate the articulation of a radiusulna joint to enhance the wheeled motion’s agility. Additionally, we establish comprehensive mathematical models for adaptive dynamics model, providing a robust theoretical foundation for subsequent motion planning and high-precision control. A novel telescopic vehicle mode is also proposed for complex wheel-leg hybrid motion, offering optimized solutions for intricate robot locomotion. Furthermore, we employ parallel underactuated MPC controllers for each leg at the control level, contributing to heightened motion precision and stability. Extensive validation through physical platform experiments highlights the effectiveness and feasibility of the proposed controllers, offering substantial support for real-world applications in robotics.

Legged robots have the potential to become vital in maintenance, home support, and exploration scenarios. In order to interact with and manipulate their environments, most legged robots are equipped with a dedicated robot arm, which means additional mass and mechanical complexity compared to standard legged robots. In this work, we explore pedipulation - using the legs of a legged robot for manipulation. By training a reinforcement learning policy that tracks position targets for one foot, we enable a dedicated pedipulation controller that is robust to disturbances, has a large workspace through whole-body behaviors, and can reach far-away targets with gait emergence, enabling loco-pedipulation. By deploying our controller on a quadrupedal robot using teleoperation, we demonstrate various real-world tasks such as door opening, sample collection, and pushing obstacles. We demonstrate load carrying of more than 2.0 kg at the foot. Additionally, the controller is robust to interaction forces at the foot, disturbances at the base, and slippery contact surfaces. Videos of the experiments are available at https://sites.google.com/leggedrobotics.com/pedipulate.

Pedestrian trajectory prediction plays an important role in autonomous driving systems and robotics. Recent work utilizing prominent deep learning models for pedestrian motion prediction makes limited a priori assumptions about human movements, resulting in a lack of explainability and explicit constraints enforced on predicted trajectories. We present a dynamics-based deep learning framework with a novel asymptotically stable dynamical system integrated into a Transformer-based model. We use an asymptotically stable dynamical system to model human goal-targeted motion by enforcing the human walking trajectory, which converges to a predicted goal position, and to provide the Transformer model with prior knowledge and explainability. Our framework features the Transformer model that works with a goal estimator and dynamical system to learn features from pedestrian motion history. The results show that our framework outperforms prominent models using five benchmark human motion datasets.

RGB-D cameras are crucial in robotic perception, given their ability to produce images augmented with depth data. However, their limited field of view (FOV) often requires multiple cameras to cover a broader area. In multi-camera RGB-D setups, the goal is typically to reduce camera overlap, optimizing spatial coverage with as few cameras as possible. The extrinsic calibration of these systems introduces additional complexities. Existing methods for extrinsic calibration either necessitate specific tools or highly depend on the accuracy of camera motion estimation. To address these issues, we present PeLiCal, a novel line-based calibration approach for RGB-D camera systems exhibiting limited overlap. Our method leverages long line features from surroundings, and filters out outliers with a novel convergence voting algorithm, achieving targetless, real-time, and outlier-robust performance compared to existing methods. We open source our implementation on https://github.com/joomeok/PeLiCal.git.