ICRA 2024 论文列表

This paper primarily focuses on evaluating and benchmarking the robustness of visual representations in the context of object assembly tasks. Specifically, it investigates the alignment and insertion of objects with geometrical extrusions, commonly referred to as a peg-in-hole task. The accuracy required to detect and orient the peg and the hole geometry in SE(3) space for successful assembly poses significant challenges. Addressing this, we employ a general framework in visuomotor policy learning that utilizes visual pretraining models as vision encoders. Our study investigates the robustness of this framework when applied to a dual-arm manipulation setup, specifically to the grasp variations. Our quantitative analysis shows that existing pretrained models fail to capture the essential visual features necessary for this task: a visual encoder trained from scratch consistently outperforms the frozen pre-trained models. Moreover, we discuss rotation representations and associated loss functions that substantially improve policy learning. We present a novel task scenario designed to evaluate the progress in visuomotor policy learning, with a specific focus on improving the robustness of intricate assembly tasks that require both geometrical and spatial reasoning. Videos, additional experiments, dataset, and code are available at https://sites.google.com/view/geometric-peg-in-hole.

Ethically compliant autonomous systems (ECAS) are the prevailing approach to building robotic systems that perform sequential decision making subject to ethical theories in fully observable environments. However, in real-world robotics settings, these systems often operate under partial observability because of sensor limitations, environmental conditions, or limited inference due to bounded computational resources. Therefore, this paper proposes a partially observable ECAS (PO-ECAS), bringing this work one step closer to being a practical and useful tool for roboticists. First, we formally introduce the PO-ECAS framework and a MILP-based solution method for approximating an optimal ethically compliant policy. Next, we extend an existing ethical framework for prima facie duties to belief space and offer an ethical framework for virtue ethics inspired by Aristotle’s Doctrine of the Mean. Finally, we demonstrate that our approach is effective in a simulated campus patrol robot domain.

Robots need to estimate the material and dynamic properties of objects from observations in order to simulate them accurately. We present a Bayesian optimization approach to identifying the material property parameters of objects based on a set of observations. Our focus is on estimating these properties based on observations of scenes with different sets of interacting objects. We propose an approach that exploits the structure of the reward function by modeling the reward for each observation separately and using only the parameters of the objects in that scene as inputs. The resulting lower-dimensional models generalize better over the parameter space, which in turn results in a faster optimization. To speed up the optimization process further, and reduce the number of simulation runs needed to find good parameter values, we also propose partial evaluations of the reward function, wherein the selected parameters are only evaluated on a subset of real world evaluations. The approach was successfully evaluated on a set of scenes with a wide range of object interactions, and we showed that our method can effectively perform incremental learning without resetting the rewards of the gathered observations.

Current methods based on Neural Radiance Fields (NeRF) significantly lack the capacity to quantify uncertainty in their predictions, particularly on the unseen space including the occluded and outside scene content. This limitation hinders their extensive applications in robotics, where the reliability of model predictions has to be considered for tasks such as robotic exploration and planning in unknown environments. To address this, we propose a novel approach to estimate a 3D Uncertainty Field based on the learned incomplete scene geometry, which explicitly identifies these unseen regions. By considering the accumulated transmittance along each camera ray, our Uncertainty Field infers 2D pixel-wise uncertainty, exhibiting high values for rays directly casting towards occluded or outside the scene content. To quantify the uncertainty on the learned surface, we model a stochastic radiance field. Our experiments demonstrate that our approach is the only one that can explicitly reason about high uncertainty both on 3D unseen regions and its involved 2D rendered pixels, compared with recent methods. Furthermore, we illustrate that our designed uncertainty field is ideally suited for real-world robotics tasks, such as next-best-view selection.

If a robot masters folding a kitchen towel, we would expect it to master folding a large beach towel. However, existing policy learning methods that rely on data augmentation still don’t guarantee such generalization. Our insight is to add equivariance to both the visual object representation and policy architecture. We propose EquivAct which utilizes SIM(3)-equivariant network structures that guarantee generalization across all possible object translations, 3D rotations, and scales by construction. EquivAct is trained in two phases. We first pre-train a SIM(3)-equivariant visual representation on simulated scene point clouds. Then, we learn a SIM(3)-equivariant visuomotor policy using a small amount of source task demonstrations. We show that the learned policy directly transfers to objects that substantially differ from demonstrations in scale, position, and orientation. We evaluate our method in three manipulation tasks involving deformable and articulated objects, going beyond typical rigid object manipulation tasks considered in prior work. We conduct experiments both in simulation and in reality. For real robot experiments, our method uses 20 human demonstrations of a tabletop task and transfers zero-shot to a mobile manipulation task in a much larger setup. Experiments confirm that our contrastive pre-training procedure and equivariant architecture offer significant improvements over prior work. Project website: equivact.github.io

The fifth generation (5G) cellular network technology is mature and increasingly utilized in many industrial and robotics applications, while an important functionality is the advanced Quality of Service (QoS) features. Despite the prevalence of 5G QoS discussions in the related literature, there is a notable absence of real-life implementations and studies concerning their application in time-critical robotics scenarios. This article considers the operation of time-critical applications for 5G-enabled unmanned aerial vehicles (UAVs) and how their operation can be improved by the possibility to dynamically switch between QoS data flows with different priorities. As such, we introduce a robotics oriented analysis on the impact of the 5G QoS functionality on the performance of 5G-enabled UAVs. Furthermore, we introduce a novel framework for the dynamic selection of distinct 5G QoS data flows that is autonomously managed by the 5G-enabled UAV. This problem is addressed in a novel feedback loop fashion utilizing a probabilistic finite state machine (PFSM). Finally, the efficacy of the proposed scheme is experimentally validated with a 5G-enabled UAV in a real-world 5G stand-alone (SA) network. https://www.youtube.com/watch?v=lWtMOlVMEFI&t=1s

While many inkjet printers employ only four types of ink (i.e. CKMY) to produce a wide range of colors, numerous technical challenges still exist for contemporary 3D printers to fabricate various materials and generate composite products such as electric devices. Conversely, there have been attempts and endeavors to make things through self-assembly of parts, analogous to the autonomous and decentralized development process of the human body from just 20 types of amino acids. In our previous work, we proposed a method for the rapid production of 3D objects using the centimeter-sized modules (referred to as Roblets) capable of generating a 2D structure and subsequently self-folding themselves into a 3D configuration, akin to origami. To further leverage the capability of generating a wide variety of different types of structures by combining different modules, this research studies a method of automatically selecting and supplying modules using environmental cues. More precisely, we developed a mechanism to couple different modules corresponding to three different environments (on a flat surface, on low-dense saline, and on saturated saline) and yielded different module configurations. The process of self-assembly necessitated the application of perturbation, which was realized by imparting magnetic torque originating from an external magnetic field onto the magnets embedded in the modules.

Planning in learned latent spaces helps to decrease the dimensionality of raw observations. In this work, we propose to leverage the ensemble paradigm to enhance the robustness of latent planning systems. We rely on our Latent Space Roadmap (LSR) framework, which builds a graph in a learned structured latent space to perform planning. Given multiple LSR framework instances, that differ either on their latent spaces or on the parameters for constructing the graph, we use the action information as well as the embedded nodes of the produced plans to define similarity measures. These are then utilized to select the most promising plans. We validate the performance of our Ensemble LSR (ENS-LSR) on simulated box stacking and grape harvesting tasks as well as on a real-world robotic T-shirt folding experiment.

This paper addresses the migration of high-frequency robot controllers to remote computing services, which are connected via a communication channel prone to delays and packet loss. The stability of the networked system is guaranteed by ensuring passivity of each subcomponent in the interconnection, as well as the Time-Domain-Passivity-Approach (TDPA) for the communication channel. We reduce conservatism of the TDPA using the model knowledge on both sides of the communication system to identify passivity excesses. This is further used to avoid over-dissipation of energy in the passivity controller by augmentation of a tolerable passivity-shortage. Tracking offsets are eliminated with a position drift compensation algorithm, for which convergence guarantees are provided. The experimental validation of the results conducted on a 7-DoF Franka Research 3 robot demonstrates a substantial enhancement in tracking performance due to the proposed modifications, particularly in scenarios with high communication delays.

Trajectory segmentation refers to dividing a trajectory into meaningful consecutive sub-trajectories. This paper focuses on trajectory segmentation for 3D rigid-body motions. Most segmentation approaches in the literature represent the body’s trajectory as a point trajectory, considering only its translation and neglecting its rotation. We propose a novel trajectory representation for rigid-body motions that incorporates both translation and rotation, and additionally exhibits several invariant properties. This representation consists of a geometric progress rate and a third-order trajectory-shape descriptor. Concepts from screw theory were used to make this representation time-invariant and also invariant to the choice of body reference point. This new representation is validated for a self-supervised segmentation approach, both in simulation and using real recordings of human-demonstrated pouring motions. The results show a more robust detection of consecutive sub-motions with distinct features and a more consistent segmentation compared to conventional representations. We believe that other existing segmentation methods may benefit from using this trajectory representation to improve their invariance.

Complementary to prevalent LiDAR and camera systems, millimeter-wave (mmWave) radar is robust to adverse weather conditions like fog, rainstorms, and blizzards but offers sparse point clouds. Current techniques enhance the point cloud by the supervision of LiDAR’s data. However, high-performance LiDAR is notably expensive and is not commonly available on vehicles. This paper presents mmEMP, a supervised learning approach that enhances radar point clouds using a low-cost camera and an inertial measurement unit (IMU), enabling crowd-sourcing training data from commercial vehicles. Bringing the visual-inertial (VI) supervision is challenging due to the spatial agnostic of dynamic objects. Moreover, spurious radar points from the curse of RF multipath make robots misunderstand the scene. mmEMP first devises a dynamic 3D reconstruction algorithm that restores the 3D positions of dynamic features. Then, we design a neural network that densifies radar data and eliminates spurious radar points. We build a new dataset in the real world. Extensive experiments show that mmEMP achieves competitive performance compared with the SOTA approach training by LiDAR’s data. In addition, we use the enhanced point cloud to perform object detection, localization, and mapping to demonstrate mmEMP’s effectiveness.

Visual place recognition(VPR) is a crucial technology for autonomous driving and robotic navigation. However, severe appearance and perspective changes often lead to degradation of algorithm performance. Current methods mainly utilize single-modality RGB images, which are sensitive to environmental changes. To address this challenge, we propose a novel multi-modal visual place recognition method by incorporating depth information as auxiliary data to enhance the robustness of the VPR algorithm. The pipeline involves dual-branch feature extraction and shared multi-modal feature fusion based on transformer(SFFM) to enable full interaction between semantic and structural information. Furthermore, we introduces a time-constrained graph attention aggregation(TC-GAT) that propagates node information across time and space to deal with perceptual aliasing. Extensive experiments on the Oxford Robotcar and MSLS datasets demonstrate that the proposed algorithm is not only effective in appearance changes but also competitive in opposing viewpoints.

This paper presents an extension to visual inertial odometry (VIO) by introducing tightly-coupled fusion of magnetometer measurements. A sliding window of keyframes is optimized by minimizing re-projection errors, relative inertial errors, and relative magnetometer orientation errors. The results of IMU orientation propagation are used to efficiently transform magnetometer measurements between frames producing relative orientation constraints between consecutive frames. The soft and hard iron effects are calibrated using an ellipsoid fitting algorithm. The introduction of magnetometer data results in significant reductions in the orientation error and also in recovery of the true yaw orientation with respect to the magnetic north. The proposed framework operates in all environments with slow-varying magnetic fields, mainly outdoors and underwater. We have focused our work on the underwater domain, especially in underwater caves, as the narrow passage and turbulent flow make it difficult to perform loop closures and reset the localization drift. The underwater caves present challenges to VIO due to the absence of ambient light and the confined nature of the environment, while also being a crucial source of fresh water and providing valuable historical records. Experimental results from underwater caves demonstrate the improvements in accuracy and robustness introduced by the proposed VIO extension.

In contact-rich tasks, the hybrid, multi-modal nature of contact dynamics poses great challenges in model representation, planning, and control. Recent efforts have attempted to address these challenges via data-driven methods, learning dynamical models in combination with model predictive control. Those methods, while effective, rely solely on minimizing forward prediction errors to hope for better task performance with MPC controllers. This weak correlation can result in data inefficiency as well as limitations to overall performance. In response, we propose a novel strategy: using a policy gradient algorithm to find a simplified dynamics model that explicitly maximizes task performance. Specifically, we parameterize the stochastic policy as the perturbed output of the MPC controller, thus, the learned model representation can directly associate with the policy or task performance. We apply the proposed method to contact-rich tasks where a three-fingered robotic hand manipulates previously unknown objects. Our method significantly enhances task success rate by up to 15% in manipulating diverse objects compared to the existing method while sustaining data efficiency. Our method can solve some tasks with success rates of 70% or higher using under 30 minutes of data. All videos and codes are available at https://sites.google.com/view/lcs-rl.

Electrical Impedance Tomography (EIT)-based tactile sensors offer durability, scalability, and cost-effective manufacturing. However, simultaneously reconstructing force and shape from boundary measurements remains challenging due to EIT’s inherent location dependencies and image artifacts. This study presents a model-driven multimodal convolutional neural network (MM-CNN) for joint EIT-based force and shape sensing. The hybrid approach combines physics-inspired voltage preprocessing with an attention-based network to overcome EIT’s limitations. The preprocessing network applies a linearized one-step inverse solution with Tikhonov regularization to convert raw boundary voltage into a noise-reduced 2D image. The image reconstruction network uses an attention mechanism to focus on salient features, addressing location dependency issues. Quantitative metrics show that MM-CNN outperforms traditional EIT algorithms like NOSER and TV, reducing location dependency and improving shape discrimination. MM-CNN enables unified force and shape modalities, validated through real-contact experiments, enhancing EIT tactile systems for human-robot interaction by incorporating physical knowledge with deep learning.

Autonomous navigation on inland waters requires an accurate understanding of the environment in order to react to possible obstacles. Deep learning is a promising technique to detect obstacles robustly. However, supervised deep learning models require large data-sets to adjust their weights and to generalize to unseen data. Therefore, we equipped our research vessel with a laser scanner and a stereo camera to record a novel obstacle detection data-set for inland waters. We annotated 1974 stereo images and lidar point clouds with 3d bounding boxes. Furthermore, we provide an initial approach and a suitable metric2 to compare the results on the test data-set. The data-set is publicly available3 and seeks to make a contribution towards increasing the safety on inland waters.

Automation application in crop harvesting has increased in the past decades. Various types of harvesting robots are emerging in both commercial and research areas. One of the main challenges is the precision alignment of the gripper and the target crop. An undesired dislocation can harm both the gripper and the crop, which is mainly caused by uncertainties from the sensors and the manipulator. To solve the problem, the dual-camera setup is designed and implemented on a self-built robot. The perception of the tomato is done by a fixed depth camera and a camera without depth on the gripper. The proposed dual-camera image-based visual servoing (IBVS) controller is designed to deal with the image feedback from both cameras and the proof of asymptotically convergence is provided. Furthermore, the cumulative error compensation reduces the time for the harvesting process. The experiments were conducted in the greenhouse and tested under various conditions. The time cost is formulated as a function and the success picking rate of tomatoes is 68.4%.

Exploration missions on extra terrestrial celestial bodies are to date performed by complex and heavy robotic systems. The trend is towards lighter modular systems that can be (re)configured in situ according to mission specific requirements. To facilitate flexible configurability, a multifunctional interconnect is used to mechanically couple the involved systems while providing electrical power and data transmission. The paper presents the further development of the reliable electromechanical interface (EMI) from the TransTerrA project, which has been proven in several field tests and reached TRL 4. Docking under loads of up to 550 N has been successfully tested with the new design. The experiments presented include undocking at various inclinations with different loads expected for the application scenario. The maximum determined static load that can be carried by the further developed EMI is 2000 N. In further experiments, new contact blocks responsible for the transfer of electrical power and data were tested for water resistance and resilience to environmental factors, as well as power and data transfer. The obtained results will be helpful in the development of a multi-functional interface suitable for lunar applications and missions having similar challenging environmental conditions.