IROS 2024 论文列表

6D object pose estimation holds essential roles in various fields, particularly in the grasping of industrial workpieces. Given challenges like rust, high reflectivity, and absent textures, this paper introduces a point cloud based pose estimation framework (PS6D). PS6D centers on slender and multi-symmetric objects. It extracts multi-scale features through an attention-guided feature extraction module, designs a symmetry-aware rotation loss and a center distance sensitive translation loss to regress the pose of each point to the centroid of the instance, and then uses a two-stage clustering method to complete instance segmentation and pose estimation. Objects from the Siléane and IPA datasets and typical workpieces from industrial practice are used to generate data and evaluate the algorithm. In comparison to the state-of-the-art approach, PS6D demonstrates an 11.5% improvement in {{\text{F}}_{{1_{inst}}}}{{\text{F}}_{{1_{inst}}}} and a 14.8% improvement in Recall. The main part of PS6D has been deployed to the software of Mech-Mind, and achieves a 91.7% success rate in bin-picking experiments, marking its application in industrial pose estimation tasks.

Prior structural reference like floor plan is readily accessible in indoor scene, which exhibits the potential of improving localization quality without the requirements of a previously-built high-precision map. This paper introduces a novel optimal transport-based framework for prior structural reference-based localization, aiming to improve the robustness for the robot localization. Leveraging the spacial relations of structures, a matching method based on optimal transport theory is proposed and it improves the robustness of matching results in dynamic scene and rapid rotation conditions. Additionally, this paper handles metric inaccuracies in the known structural reference by implementing an prior guided plane adjustment-based updating strategy. This strategy combines prior and observational information to jointly optimize the structural information within a sliding window. The performance of the framework is validated through real-world experiments, demonstrating superior accuracy and robustness to disturbances from dynamic occlusion and rapid rotation compared to common state-of-the-art SLAM and localization methods.

Near Infrared (NIR) spectroscopy is widely used in industrial quality control and automation to test the purity and grade of items. In this research, we propose a novel sensorized end effector and acquisition strategy to capture spectral signatures from objects and register them with a 3D point cloud. Our methodology first takes a 3D scan of an object generated by a time-of-flight depth camera and decomposes the object into a series of planned viewpoints covering the surface. We generate motion plans for a robot manipulator and end-effector to visit these viewpoints while maintaining a fixed distance and surface normal. This process is enabled by the spherical motion of the end-effector and ensures maximal spectral signal quality. By continuously acquiring surface reflectance values as the end-effector scans the target object, the autonomous system develops a four-dimensional model of the target object: position in an R3 coordinate frame, and a reflectance vector denoting the associated spectral signature. We demonstrate this system in building spectral-spatial object profiles of increasingly complex geometries. We show the proposed system and spectral acquisition planning produce more consistent spectral signals than naïve point scanning strategies. Our work represents a significant step towards high-resolution spectral-spatial sensor fusion for automated quality assessment.

Personalized motion planning holds significant importance within urban automated driving, catering to the unique requirements of individual users. Nevertheless, prior endeavors have frequently encountered difficulties in simultaneously addressing two crucial aspects: personalized planning within intricate urban settings and enhancing planning performance through data utilization. The challenge arises from the expensive and limited nature of user data, coupled with the scene state space tending towards infinity. These factors contribute to overfitting and poor generalization problems during model training. Henceforth, we propose an instance-based transfer imitation learning approach. This method facilitates knowledge transfer from extensive expert domain data to the user domain, presenting a resolution to these issues. We initially train a pre-trained model using large-scale expert data. Subsequently, during the fine-tuning phase, we feed the batch data, which comprises expert and user data. Employing the inverse reinforcement learning technique, we extract the style feature distribution from user demonstrations, constructing the regularization term for the approximation of user style. In our experiments, we conducted extensive evaluations of the proposed method. Compared to the baseline methods, our approach mitigates the overfitting issue caused by sparse user data. Furthermore, we discovered that integrating the driving model with a differentiable nonlinear optimizer as a safety protection layer for end-to-end personalized fine-tuning results in superior planning performance. The code will be available at https://github.com/LinFunster/PP-TIL.

Modelling complex deformation for soft robotics provides a guideline to understand their behaviour, leading to safe interaction with the environment. However, building a surrogate model with high accuracy and fast inference speed can be challenging for soft robotics due to the nonlinearity from complex geometry, large deformation, material nonlinearity etc. The reality gap from surrogate models also prevents their further deployment in the soft robotics domain.In this study, we proposed a physics-informed Neural Networks (PINNs) named PINN-Ray to model complex deformation for a Fin Ray soft robotic gripper, which embeds the minimum potential energy principle from elastic mechanics and additional high-fidelity experimental data into the loss function of neural network for training. This method is significant in terms of its generalisation to complex geometry and robust to data scarcity as compared to other data-driven neural networks. Furthermore, it has been extensively evaluated to model the deformation of the Fin Ray finger under external actuation. PINN-Ray demonstrates improved accuracy as compared with Finite element modelling (FEM) after applying the data assimilation scheme to treat the sim-to-real gap. Additionally, we introduced our automated framework to design, fabricate soft robotic fingers, and characterise their deformation by visual tracking, which provides a guideline for the fast prototype of soft robotics.

Robotics collaborative drawing involves the inter-action between humans and robots to create of visual art using a variety of tools and materials, serving various functions such as communication, narration, and emotional representation. A creative technique within the human natural drawing process is known as inpainting, which involves reconstructing or editing elements in a drawing. This paper introduces PICaSo (Physical Inpainting on Canvas Solution), a robotic drawing system that enables multiple users to collaboratively create artwork on a canvas by integrating the inpainting process. PICaSo utilizes a fine-tuned text-to-image model to interpret natural language prompts into artistic renderings on canvas. Users guide the process by simple descriptive text and specifying desired drawing placement, empowering the robotic arm to autonomously translate these instructions into physical artworks. Our system’s innovation lies in its effective translation of digital inpainting processes into physical actions. By leveraging our erasing capability that enables selective removal of specific parts on the canvas without impacting neighboring areas, facilitating the creation of sequential drawings. This paper comprehensively outlines the capabilities of the proposed system, explores potential applications across various domains, and addresses technical challenges encountered during its development. Project website: shadynasrat.github.io/PICaSo

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that comprehend and grasp objects based on natural language instructions. While the ability to understand personal objects like my wallet facilitates more natural interaction with human users, current LCRG systems only allow generic language instructions, e.g., the black-colored wallet next to the laptop. To this end, we introduce a task scenario GraspMine alongside a novel dataset aimed at pinpointing and grasping personal objects given personal indicators via learning from a single human-robot interaction, rather than a large labeled dataset. Our proposed method, Personalized Grasping Agent (PGA), addresses GraspMine by leveraging the unlabeled image data of the user’s environment, called Reminiscence. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. This results in significant efficiency while previous LCRG systems rely on resource-intensive human annotations—necessitating hundreds of labeled data to learn my wallet. Moreover, PGA outperforms baseline methods across all metrics and even shows comparable performance compared to the fully-supervised method, which learns from 9k annotated data samples. We further validate PGA’s real-world applicability by employing a physical robot to execute GrsapMine. Code and data are publicly available at https://github.com/JHKim-snu/PGA.

We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DoF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. PEGASUS allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that capture images from both the object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.

Networked robotic systems balance compute, power, and latency constraints in applications such as self-driving vehicles, drone swarms, and teleoperated surgery. A core problem in this domain is deciding when to offload a computationally expensive task to the cloud, a remote server, at the cost of communication latency. Task offloading algorithms often rely on precise knowledge of system-specific performance metrics, such as sensor data rates, network bandwidth, and machine learning model latency. While these metrics can be modeled during system design, uncertainties in connection quality, server load, and hardware conditions introduce real-time performance variations, hindering overall performance. We introduce PEERNet, an end-to-end and real-time profiling tool for cloud robotics. PEERNet enables performance monitoring on heterogeneous hardware through targeted yet adaptive profiling of system components such as sensors, networks, deep-learning pipelines, and devices. We showcase PEERNet’s capabilities through networked robotics tasks, such as image-based teleoperation of a Franka Emika Panda arm and querying vision language models using an Nvidia Jetson Orin. PEERNet reveals non-intuitive behavior in robotic systems, such as asymmetric network transmission and bimodal language model output. Our evaluation underscores the effectiveness and importance of benchmarking in networked robotics, demonstrating PEERNet’s adaptability. Our code is open-source and available at github.com/UTAustin-SwarmLab/PEERNet.

Event cameras can record scene dynamics with high temporal resolution, providing rich scene details for monocular depth estimation (MDE) even at low-level illumination. Therefore, existing complementary learning approaches for MDE fuse intensity information from images and scene details from event data for better scene understanding. However, most methods directly fuse two modalities at pixel level, ignoring that the attractive complementarity mainly impacts high-level patterns that only occupy a few pixels. For example, event data is likely to complement contours of scene objects. In this paper, we discretize the scene into a set of high-level patterns to explore the complementarity and propose a Pattern-based Complementary learning architecture for monocular Depth estimation (PCDepth). Concretely, PCDepth comprises two primary components: a complementary visual representation learning module for discretizing the scene into high-level patterns and integrating complementary patterns across modalities and a refined depth estimator aimed at scene reconstruction and depth prediction while maintaining an efficiency-accuracy balance. Through pattern-based complementary learning, PCDepth fully exploits two modalities and achieves more accurate predictions than existing methods, especially in challenging nighttime scenarios. Extensive experiments on MVSEC and DSEC datasets verify the effectiveness and superiority of our PCDepth. Remarkably, compared with state-of-the-art, PCDepth achieves a 37.9% improvement in accuracy in MVSEC nighttime scenarios.

Identifying traversable areas is a critical task for unmanned vehicles exploring safely through unstructured environments. In practice, the ambiguity in perceiving terrain traversability usually brings great challenges for autonomous exploration in unknown and uneven terrain, which often leads to conservative strategies or potential risk of vehicle damage, resulting in many unexplored areas in the environment. To that end, this paper proposes a plane-assisted autonomous robot exploration framework (PARE) to achieve maximum volume and safe autonomous exploration. The process is carried out by a three-step dual-layer framework: constructing a local tree using Plane-Assisted RRT* (PA-RRT*), calculating exploration gain based on terrain information, and maintaining a global search graph. Firstly, the planar feature metrics (flatness, sparsity, elevation variation, slope and slope variation) are introduced to determine the terrain traversability. Secondly, to completely explore the rugged environment, we propose a dual-layer exploration framework comprising local and global strategies. A local planner based on PA-RRT* is proposed to find the best path by evaluating the planar information and the volumetric gain within the local exploration tree. Meanwhile, a global planner constructed by graph is proposed to record unexplored nodes with high exploration gain from the local tree to ensure a high level of exploration volume. Extensive simulation and real-world experiments demonstrate that our method significantly outperforms existing frameworks, with an average improvement of more than 12% in exploration volume.

In this paper, we introduce the concept of using passive arm structures with intrinsic impedance for robot-robot and human-robot collaborative carrying with quadruped robots. The concept is meant for a leader-follower task and takes a minimalist approach that focuses on exploiting the robots’ payload capabilities and reducing energy consumption, without compromising the robot locomotion capabilities. We introduce a preliminary arm mechanical design and describe how to use its joint displacements to guide the robot’s motion. To control the robot’s locomotion, we propose a decentralized Model Predictive Controller that incorporates an approximation of the arm dynamics and the estimation of the external forces from the collaborative carrying. We validate the overall system experimentally by performing both robot-robot and human-robot collaborative carrying on a stair-like obstacle and on rough terrain.

Optimization methods for long-horizon, dynamically feasible motion planning in robotics tackle challenging nonconvex and discontinuous optimization problems. Traditional methods often falter due to the nonlinear characteristics of these problems. We introduce a technique that utilizes learned representations of the system, known as Polytopic Action Sets, to efficiently compute long-horizon trajectories. By employing a suitable sequence of Polytopic Action Sets, we transform the long-horizon dynamically feasible motion planning problem into a Linear Program. This reformulation enables us to address motion planning as a Mixed Integer Linear Program (MILP). We demonstrate the effectiveness of a Polytopic Action-Set and Motion Planning (PAAMP) approach by identifying swing-up motions for a torque-constrained pendulum as fast as 0.75 milliseconds. This approach is well-suited for solving complex motion planning and long-horizon Constraint Satisfaction Problems (CSPs) in dynamic and underactuated systems such as legged and aerial robots.

Locomotion control is still a challenging task for quadruped robots traversing diverse terrains amidst unforeseen disturbances. Recently, privileged learning has been employed to learn reliable and robust quadrupedal locomotion over various terrains based on a teacher-student architecture. However, its one-encoder structure is not adequate in addressing external force perturbations. The student policy would experience inevitable performance degradation due to the feature embedding discrepancy between the feature encoder of the teacher policy and the one of the student policy. Hence, this paper presents a privileged learning framework with multiple feature encoders and a residual policy network for robust and reliable quadruped locomotion subject to various external perturbations. The multi-encoder structure can decouple latent features from different privileged information, ultimately leading to enhanced performance of the learned policy in terms of robustness, stability, and reliability. The efficiency of the proposed feature encoding module is analyzed in depth using extensive simulation data. The introduction of the residual policy network helps mitigate the performance degradation experienced by the student policy that attempts to clone the behaviors of a teacher policy. The proposed framework is evaluated on a Unitree GO1 robot, showcasing its performance enhancement over the state-of-the-art privileged learning algorithm through extensive experiments conducted on diverse terrains. Ablation studies are conducted to illustrate the efficiency of the residual policy network.

In recent years, multi-agent path finding (MAPF) has attracted widespread attention in the fields of artificial intelligence and robotics. Its main goal is to find paths for multiple agents, each having specified start and end locations on a grid, without collisions, while minimizing the total travel time. In this work, we present a new algorithm called P4 (Pruning and Prediction-based Priority Planning), designed to accommodate large numbers of agents with enhanced scalability. The P4 method combines three key components: Point-to-Point (PnP) algorithm, dynamic window approach, and path direction prediction. In this way we reduce the search space and increase the speed of the computation. Our experiments show that P4 consistently achieves shorter execution times and produces solutions that are close to optimal. For example, for 200 agents and real map orz900d, P4 is 4× faster than optimal algorithm CBSH while the sum of delays is within 15% of optimal. The P4 method outperforms other existing suboptimal methods in both performance and solution quality. We also show that our approach exceeds existing methods in success rate under time constraints. As time limit is increased from 0.1 to 100 seconds, success rate of P4 increases from 50% to 100%. On the other hand, the success rate for alternative sub-optimal methods is less than that of P4.

Geometric perception problems are fundamental tasks in robotics and computer vision. In real-world applications, they often encounter the inevitable issue of outliers, preventing traditional algorithms from making correct estimates. In this paper, we present a novel general-purpose robust estimator TIVM (Thresholding with Intra-class Variance Maximization) that can collaborate with standard non-minimal solvers to efficiently reject outliers for geometric perception problems. First, we introduce the technique of intra-class variance maximization to design a dynamic 2-group thresholding method on the measurement residuals, aiming to distinctively separate inliers from outliers. Then, we develop an iterative framework that robustly optimizes the model by approaching the pure-inlier group using a multi-layered dynamic thresholding strategy as subroutine, in which a self-adaptive mechanism for layer-number tuning is further employed to minimize the user-defined parameters. We validate the proposed estimator on 3 classic geometric perception problems: rotation averaging, point cloud registration and category-level perception, and experiments show that it is robust against 7090% of outliers and can converge typically in only 315 iterations, much faster than state-of-the-art robust solvers such as RANSAC, GNC and ADAPT. Furthermore, another highlight is that: our estimator can retain approximately the same level of robustness even when the inlier-noise statistics of the problem are fully unknown.

This paper presents a soft actuator inspired by origami and a tunable strain limiting layer, which is proposed for reconfigurable soft robotic grasping. Main structure of the actuator is based on Miura origami which generates extension under pressurized air while a limiting layer with tunable length enables the actuator with different motion patterns. By driving the limiting layer through a servo motor, the range of motion and trajectory of the actuator can be pre-programed and the gripper’s grasping range will be affected accordingly. This paper discusses the design, fabrication, analysis and experimental verification of the actuator. Then grasping performance of the gripper under objects of different shapes, sizes, and weights is experimentally evaluated. The reconfigurable soft gripper can be applied as an end-effector to accomplish adaptive grasping tasks with various targets.

This study addresses the challenge of deploying robotic software with Quality of Service (QoS) constraints in Edge-Cloud computing clusters. The paper introduces HEFT4K, an event-driven scheduling method tailored for Kubernetes-managed systems based on the Heterogeneous Early Finish Time (HEFT) algorithm. This algorithm reduces software execution time (makespan) and facilitates re-mapping in case of node failures, involving only essential containers to maintain uninterrupted robot functionality. Experimental results, conducted on a real-world robot and synthetic benchmarks, show a 75% speedup in makespan compared to the standard Kubernetes scheduler, enhancing the efficiency of QoS-focused scheduling for robotic applications in distributed systems.

Markerless motion capture devices such as the Leap Motion Controller (LMC) have been extensively used for tracking hand, wrist, and forearm positions as an alternative to Marker-based Motion Capture (MMC). However, previous studies have highlighted the subpar performance of LMC in reliably recording hand kinematics. In this study, we employ four LMC devices to optimize their collective tracking volume, aiming to enhance the accuracy and precision of hand kinematics. Through Monte Carlo simulation, we determine an optimized layout for the four LMC devices and subsequently conduct reliability and validity experiments encompassing 1560 trials across ten subjects. The combined tracking volume is validated against an MMC system, particularly for kinematic movements involving wrist, index, and thumb flexion. Utilizing calculation resources in one computer, our result of the optimized configuration has a better visibility rate with a value of 0.05 ± 0.55 compared to the initial configuration with -0.07 ± 0.40. Multiple Leap Motion Controllers (LMCs) have proven to increase the interaction space of capture volume but are still unable to give agreeable measurements from dynamic movement.