ICRA 2024 论文列表

In this work, we propose a singularity-robust whole-body control framework that ensures smooth task transitions while maintaining strict priorities. The weighted generalized inverse is adopted to derive a hierarchical control law compatible with singular and redundant tasks. Moreover, a smooth activation matrix is proposed to continuously shape both null-space projectors and task-level control actions. Validation has been conducted in MATLAB/Simulink and MuJoCo simulations with Rollin’ Justin.

Unlike parallel robots, for which hundreds of different architectures have been proposed, the vast majority of six-degree-of-freedom (DOF) serial robots have one of two simple architectures. In both architectures, the inverse kinematics can be solved in closed form and the singularities described by trivial geometric and algebraic conditions. These conditions can be readily obtained by analyzing the determinant of the robot’s Jacobian matrix, and provide an in-depth understanding of the robot’s singularities, which is essential for its optimal use. However, for various reasons, robot arms with unorthodox architectures are occasionally designed. Such arms do not have closed-form inverse kinematics and little insight into their singularities can be gained by analyzing the determinant of their Jacobian. One such robot arm for which the conventional singularity analysis approach fails is the new Link 6 collaborative robot by Kinova. In this paper, we study the complex singularities of Link 6 by investigating all possibilities for screw dependencies, deriving a simple equation for each case, and then describing each singularity type using Grassmann line geometry. Twelve different singularity configurations are identified and described with seven relatively simple geometric conditions. Our approach is general and can be applied to other robot arms.

Mutual localization stands as a foundational component within various domains of multi-robot systems. Nevertheless, in relative pose estimation, time synchronization is usually underappreciated and rarely addressed, although it significantly influences estimation accuracy. In this paper, we introduce time synchronization into mutual localization to recover the time offset and relative poses between robots simultaneously. Under a constant velocity assumption in a short time, we fuse time offset estimation with our previous bearing-based mutual localization by a novel error representation. Based on the error model, we formulate a joint optimization problem and utilize semi-definite relaxation (SDR) to furnish a lossless relaxation. By solving the relaxed problem, time synchronization and relative pose estimation can be achieved when time drift between robots is limited. To enhance the application range of time offset estimation, we further propose an iterative method to recover the time offset from coarse to fine. Comparisons between the proposed method and the existing ones through extensive simulation tests present prominent benefits of time synchronization on mutual localization. Moreover, real-world experiments are conducted to show the practicality and robustness.

Recently, fiber optic sensors such as fiber Bragg gratings (FBGs) have been widely investigated for shape reconstruction and force estimation of flexible surgical robots. However, most existing approaches need precise model parameters of FBGs inside the fiber and their alignments with the flexible robots for accurate sensing results. Another challenge lies in online acquiring external forces at arbitrary locations along the flexible robots, which is highly required when with large deflections in robotic surgery. In this paper, we propose a novel data-driven paradigm for simultaneous estimation of shape and force along highly deformable flexible robots by using sparse strain measurement from a single-core FBG fiber. A thin-walled soft sensing tube helically embedded with FBG sensors is designed for a robotic-assisted flexible ureteroscope with large deflection up to 270° and a bend radius under 10 mm. We introduce and study three learning models by incorporating spatial strain encoders, and compare their performances in both free space without interactions as well as constrained environments with contact forces at different locations. The experimental results in terms of dynamic shape-force sensing accuracy demonstrate the effectiveness and superiority of the proposed methods.

This paper details the simulation and experimental validation of an autonomous perching and take-off method for a multirotor unmanned aerial vehicle (UAV) using a suction cup perching mechanism on vertical surfaces. The suction cup interaction with different surface types is characterized with experimental tests to accurately model the perching manoeuvre. The resulting model is used to develop a realistic hardware-in-the-loop (HIL) simulation of the perching and take-off manoeuvre of the UAV in Gazebo. A control method is developed to automate the perching and take-off manoeuvre. The method is tested in simulation and is experimentally validated with flight tests. Comparisons between simulation and experimental data demonstrate that the simulation is accurate and can be used to continue the development of autonomous perching methods.

Simulation is a key technology in robotics as it enables the generation of environmental data and testing scenarios for development and maintenance purposes. However, simulations are an imperfect representation of the real-world and the so-called sim-to-real gap between simulation and reality hinders the deployment of virtual developed solutions without additional effort. Modeling complex systems like highly dynamic and holonomic mobile robots presents additional complexities in simulation. This paper addresses these challenges through a case study on creating a model for a highly dynamic logistics robot. The study considers the modeling of the entire system down to creating suitable colliders for the rollers of a Mecanum wheel. Additionally, the impact of significant physics parameters is presented. To bridge the sim-to-real gap, a pipeline is developed that utilizes a Motion Capture system to compare the behavior of a real robot with its simulated counterpart across various motions. By leveraging expert knowledge gained from the real-world data, the simulation model is manually tuned to replicate complex system behaviors, such as sliding effects.

Using tactile sensors for manipulation remains one of the most challenging problems in robotics. At the heart of these challenges is generalization: How can we train a tactile-based policy that can manipulate unseen and diverse objects? In this paper, we propose to perform Reinforcement Learning with only visual tactile sensing inputs on diverse objects in a physical simulator. By training with diverse objects in simulation, it enables the policy to generalize to unseen objects. However, leveraging simulation introduces the Sim2Real transfer problem. To mitigate this problem, we study different tactile representations and evaluate how each affects real-robot manipulation results after transfer. We conduct our experiments on diverse real-world objects and show significant improvements over baselines. Our project page is available at https://tactilerl.github.io/.

In this paper, we address the Sim2Real gap in the field of vision-based tactile sensors for classifying object surfaces. We train a Diffusion Model to bridge this gap using a relatively small dataset of real-world images randomly collected from unlabeled everyday objects via the DIGIT sensor. Subsequently, we employ a simulator to generate images by uniformly sampling the surface of objects from the YCB Model Set. These simulated images are then translated into the real domain using the Diffusion Model and automatically labeled to train a classifier. During this training, we further align features of the two domains using an adversarial procedure. Our evaluation is conducted on a dataset of tactile images obtained from a set of ten 3D-printed YCB objects. The results reveal a total accuracy of 81.9%, a significant improvement compared to the 34.7% achieved by the classifier trained solely on simulated images. This demonstrates the effectiveness of our approach. We further validate our approach using the classifier on a 6D object pose estimation task from tactile data.

In industry, random bin picking is a complex and difficult task where instance segmentation and object pose estimation based on point clouds are key processes. Recently, learning-based segmentation and pose estimation methods for 3D point clouds have been proposed. However, many of them require supervised learning with datasets with annotations of objects. Since it is difficult to annotate all stacked instances in bin picking dataset, learning without real-world datasets has become a major interest. In this paper, we introduce an instance-level object pose estimation method for bin picking, which is trained using only simulated data and seamlessly applied to real-world scenarios without additional adaptation. To enable this, we introduce a method for generating a comprehensive synthetic dataset using a physics simulator, which incorporates 3D CAD models of objects and automatically generates annotations for both segmentation and pose estimation. Our experiments, conducted on synthetic datasets, highlight the competitive performance of our method in terms of recall and accuracy. Furthermore, we demonstrate the successful integration of our approach with real robot random bin picking, resulting in significantly improved picking success rates.

Robotic sketching in real-world scenarios poses a challenging problem with diverse applications in art, robotics, and digital design. We present a novel approach that bridges the gap between digital and robotic sketching, leveraging behavior cloning and reinforcement learning techniques. This paper introduces an approach aimed at bringing the gap between simulated and real-world robotic sketching closer together through the integration of behavior cloning and reinforcement learning techniques. Our approach trains painting policies that operate effectively in both virtual environments and real-world robotic sketching systems. We have implemented a robotic sketching system featuring an UltraArm robot equipped with a RealSense D415 camera, closely emulating the MyPaint virtual environment. Our system can perceive its environment and adapt painting policies to natural painting media. Our results highlight the effectiveness of our agent in terms of acquiring policies for high-dimensional continuous action spaces, enabling the seamless transfer of brush manipulation techniques from simulation to practical robotic sketching. Furthermore, we demonstrate our robotic sketching system’s capability to generate complex images and strokes using various configurations. https://sites.google.com/view/sketchingrobot

In this work we propose a learning-based approach to box loco-manipulation for a humanoid robot. This is a particularly challenging problem due to the need for whole-body coordination in order to lift boxes of varying weight, position, and orientation while maintaining balance. To address this challenge, we present a sim-to-real reinforcement learning approach for training general box pickup and carrying skills for the bipedal robot Digit. Our reward functions are designed to produce the desired interactions with the box while also valuing balance and gait quality. We combine the learned skills into a full system for box loco-manipulation to achieve the task of moving boxes from one table to another with a variety of sizes, weights, and initial configurations. In addition to quantitative simulation results, we demonstrate successful sim-to-real transfer on the humanoid robot Digit. To our knowledge this is the first demonstration of a learned controller for such a task on real world hardware.

This paper focuses on the sim-to-real issue of RGB-D grasp detection and formulates it as a domain adaptation problem. In this case, we present a global-to-local method to address hybrid domain gaps in RGB and depth data and insufficient multi-modal feature alignment. First, a self-supervised rotation pre-training strategy is adopted to deliver robust initialization for RGB and depth networks. We then propose a global-to-local alignment pipeline with individual global domain classifiers for scene features of RGB and depth images as well as a local one specifically working for grasp features in the two modalities. In particular, we propose a grasp prototype adaptation module, which aims to facilitate fine-grained local feature alignment by dynamically updating and matching the grasp prototypes from the simulation and real-world scenarios throughout the training process. Due to such designs, the proposed method substantially reduces the domain shift and thus leads to consistent performance improvements. Extensive experiments are conducted on the GraspNet-Planar benchmark and physical environment, and superior results are achieved which demonstrate the effectiveness of our method. Code is available at https://github.com/mahaoxiang822/GL-MSDA.

We present a neural-field-based large-scale reconstruction system that fuses lidar and vision data to generate high-quality reconstructions that are geometrically accurate and capture photo-realistic textures. This system adapts the state-of-the-art neural radiance field (NeRF) representation to also incorporate lidar data which adds strong geometric constraints on the depth and surface normals. We exploit the trajectory from a real-time lidar SLAM system to bootstrap a Structure-from-Motion (SfM) procedure to both significantly reduce the computation time and to provide metric scale which is crucial for lidar depth loss. We use submapping to scale the system to large-scale environments captured over long trajectories. We demonstrate the reconstruction system with data from a multi-camera, lidar sensor suite onboard a legged robot, hand-held while scanning building scenes for 600 metres, and onboard an aerial robot surveying a multi-storey mock disaster site-building. Website: https://ori-drs.github.io/projects/silvr/

Exoskeleton technologies have numerous potential applications, ranging from improving human motor skills to aiding individuals in their daily activities. While exoskeletons are increasingly viewed, for example, as promising tools in industrial ergonomics, the effect of using them on human motor control, particularly on inter-joint coordination, remains relatively uncharted. This paper investigates the effects of generic low-amplitude force fields applied by an exoskeleton on motor strategies in asymptomatic users. The force fields mimic common perturbations encountered in exoskeletons, such as residual friction, over/under-tuned assistance, or structural elasticity. Fifty-five participants performed reaching tasks while connected to an arm exoskeleton, experiencing one of five tested force fields. Their movements before and after exposure to the exoskeleton force field were compared. The study focuses both on spatial and temporal changes in coordination using specific metrics. The results reveal that even brief exposure to a low- amplitude force field, or to uncompensated residual friction and dynamic forces, applied at the joint level can alter the interjoint coordination, while task performance remains unaffected. The tested force fields induced varying degrees of changes in joint contributions and synchronization. This study highlights the importance of monitoring coordination changes to fully understand the impact of exoskeletons on human motor control and thus enable safe and widespread adoption of those devices.

This article introduces a framework for complex human-robot collaboration tasks, such as the co-manufacturing of furniture. For these tasks, it is essential to encode tasks from human demonstration and reproduce these skills in a compliant and safe manner. Therefore, two key components are addressed in this work: motion generation and shared autonomy. We propose a motion generator based on a time-invariant potential field, capable of encoding wrench profiles, complex and closed-loop trajectories, and additionally incorporates obstacle avoidance. Additionally, the paper addresses shared autonomy (SA) which enables synergetic collaboration between human operators and robots by dynamically allocating authority. Variable impedance control (VIC) and force control are employed, where impedance and wrench are adapted based on the human-robot autonomy factor derived from interaction forces. System passivity is ensured by an energy-tank based task passivation strategy. The framework’s efficacy is validated through simulations and an experimental study employing a Franka Emika Research 3 robot.

In this paper, building on recent advances in the fields of gaming AI and social robotics, we present a new approach to facilitate the social robot Haru to imitate game strategies from human players’ demonstrated trajectories and evaluative feedback in a real-time two-player game. Our research shows that Haru is able to learn and imitate human different game strategies from human players in a human time scale. In addition, our results show that human evaluative feedback plays an important role in allowing Haru to obtain a better performance via our method than human player’s demonstrations. Finally, results of our user study indicate that Haru imitating human player’s game strategies via our method is perceived to be more human-like and have better game performance and experience than self-learning from pre-defined reward functions via traditional deep reinforcement learning.

In ophthalmic surgeries, such as vitreoretinal operations, surgeons rely on imaging systems, primarily microscopes, for real-time instrument monitoring and motion planning. However, novice surgeons struggle to extract 3D instrument positions from 2D microscope frames, necessitating extensive trial-and-error experience with the background that additional imaging modalities such as iOCT remain inaccessible in most operating rooms. Targeting intraocular assessment within the current surgical setup, this paper presents an imagebased pose estimation method to obtain real-time instrument tip positions in a standard 12mm-radius spherical eyeball model, which links floating instruments with on-the-retinal objects based on the intraocular shadowing principle. We validate this estimation method in a Unity simulator and verify its depth estimation capability using a specially designed eyeball phantom. Both simulator and phantom experiments demonstrate an average needle-tip estimation error within [1.0, 2.0] mm using only 2D microscope frames.

In this paper, we present a novel trajectory planning method for externally-actuated modular manipulators (EAMMs), consisting of multiple rotor-actuated links with joints that can be either locked or unlocked. This joint-locking feature allows effective balancing of the payload capacity and dexterity of the robot but significantly complicates the planning problem by introducing binary decision variables. To address this challenge, we leverage the problem’s intrinsic structure, i.e., the payload at the end-effector being enhanced by merely locking its immediate connected links; this allows us to break down the complex planning problem into a series of manageable subproblems and solve them sequentially. Our approach significantly reduces the problem’s complexity: in a serial n-link EAMM with m joint-lock mechanisms, where there could potentially be 2m distinct configurational dynamics, we require solving only n + 1 trajectory optimization problems for single rigid body dynamics sequentially, thereby rendering the problem tractable. We substantiate the efficacy of our method through various simulation and experimental studies, covering ground-free and ground-bound configurations as well as both motion-only and manipulation tasks.

Robotic manipulation of deformable linear objects (DLOs) is an active area of research, though emerging applications, like automotive wire harness installation, introduce constraints that have not been considered in prior work. Confined workspaces and limited visibility complicate prior assumptions of multi-robot manipulation and direct measurement of DLO configuration (state). This work focuses on single-arm manipulation of stiff DLOs (StDLOs) connected to form a DLO network (DLON), for which the measurements (output) are the endpoint poses of the DLON, which are subject to unknown dynamics during manipulation. To demonstrate feasibility of output-based control without state estimation, direct input-output dynamics are shown to exist by training neural network models on simulated trajectories. Output dynamics are then approximated with polynomials and found to contain well-known rigid body dynamics terms. A composite model consisting of a rigid body model and an online data-driven residual is developed, which predicts output dynamics more accurately than either model alone, and without prior experience with the system. An adaptive model predictive controller is developed with the composite model for DLON manipulation, which completes DLON installation tasks, both in simulation and with a physical automotive wire harness.