ICRA 2024 Papers

Physical human-robot interaction has been an area of interest for decades. Collaborative tasks, such as joint compliance, demand high-quality joint torque sensing. While external torque sensors are reliable, they come with the drawbacks of being expensive and vulnerable to impacts. To address these issues, studies have been conducted to estimate external torques using only internal signals, such as joint states and current measurements. However, insufficient attention has been given to friction hysteresis approximation, which is crucial for tasks involving extensive dynamic to static state transitions. In this paper, we propose a deep-learning-based method that leverages a novel long-term memory scheme to achieve dynamics identification, accurately approximating the static hysteresis. We also introduce modifications to the well-known Residual Learning architecture, retaining high accuracy while reducing inference time. The robustness of the proposed method is illustrated through a joint compliance and task compliance experiment.

Conventional industrial robots often use two-fingered grippers or suction cups to manipulate objects or interact with the world. Because of their simplified design, they are unable to reproduce the dexterity of human hands when manipulating a wide range of objects. While the control of humanoid hands evolved greatly, hardware platforms still lack capabilities, particularly in tactile sensing and providing soft contact surfaces. In this work, we present a method that equips the skeleton of a tendon-driven humanoid hand with a soft and sensorized tactile skin. Multi-material 3D printing allows us to iteratively approach a cast skin design which preserves the robot’s dexterity in terms of range of motion and speed. We demonstrate that a soft skin enables firmer grasps and piezoresistive sensor integration enhances the hand’s tactile sensing capabilities.

Multi-robot systems have increasingly become instrumental in tackling coverage problems. However, the challenge of optimizing task efficiency without compromising task success still persists, particularly in expansive, unstructured scenarios with dense obstacles. This paper presents an innovative, decentralized Voronoi-based coverage control approach to reactively navigate these complexities while guaranteeing safety. This approach leverages the active sensing capabilities of multi-robot systems to supplement GIS (Geographic Information System), offering a more comprehensive and real-time understanding of environments like post-disaster. Based on point cloud data, which is inherently non-convex and unstructured, this method efficiently generates collision-free Voronoi regions using only local sensing information through spatial decomposition and spherical mirroring techniques. Then, deadlock-aware guided map integrated with a gradient-optimized, centroid Voronoi-based coverage control policy, is constructed to improve efficiency by avoiding exhaustive searches and local sensing pitfalls. The effectiveness of our algorithm has been validated through extensive numerical simulations in high-fidelity environments, demonstrating significant improvements in task success rate, coverage ratio, and task execution time compared with others.

Given the patchy nature of gas plumes and the slow response of conventional gas sensors, the use of mobile robots for Gas Source Localization (GSL) tasks presents significant challenges. These aspects increase the difficulties in obtaining gas measurements, encompassing both qualitative and quantitative aspects. Most existing model-based GSL algorithms rely on lengthy stops at each sampling point to ensure accurate gas measurements. However, this approach not only prolongs the time required for a single measurement but also hinders sampling during robot motion, thus exacerbating the scarcity of available gas measurements. In this work, our goal is to push the boundaries in terms of continuity in sampling to enhance system efficiency. Firstly, we decouple and comprehensively evaluate the impact of both plume dynamics and gas sensor properties on the GSL performance. Secondly, we demonstrate that adopting a continuous sampling strategy, which has been generally overlooked in prior research, markedly enhances the system efficiency by obviating the prolonged measurement pauses and leveraging all the data gathered during the robot motion. Thirdly, we further expand the capabilities of the continuous sampling by introducing a novel informative path-planning strategy, which takes into account all the information gathered along the robot's movement. The proposed method is evaluated in both simulation and reality under different scenarios emulating indoor environmental conditions.

Despite the potential of mobile manipulators and applications where robots require a force-controlled physical interaction with the environment, the majority of robot automation nowadays is still based on fixed manipulators for free-motion tasks (e.g. welding, pick and place, or painting). In this work, we propose a control solution for omnidirectional mobile manipulators in force-tracking tasks, interacting with unknown surface geometries and with a human teleoperator in the control loop. Keeping a teleoperator in the loop makes the system widely applicable to unstructured environments. With little effort, a human can take care of the mobile base navigation, self-collisions, and collisions with the environment, as well as selecting the area of the asset surface to process. The teleoperator interfaces with the robot platform by commanding motion in the mobile base to increase the arm’s workspace and manoeuvrability. The operator can also command the movement of the end-effector, sliding on the surface geometry to process a specific area. Alternatively, he can let the controller execute a parametric trajectory (spiral or raster) for an autonomous area coverage and meanwhile telecommand the base in order to keep the arm in configurations with good dexterity. The autonomous controller, on the other hand, takes responsibility for following the unknown contour on the manipulated surface by only taking observations from a force/torque sensor attached to the arm’s wrist, exerting a prescribed force, and handling the motion control in the base and the arm so that both can follow their respective task requests. Overall, we have developed a user-friendly control scheme, where an operator with little training and using a joystick, can guide the robot system to perform a physically interactive task on the surface of an asset.

Visual bird’s eye view (BEV) semantic segmentation helps autonomous vehicles understand the surrounding environment only from front-view (FV) images, including static elements (e.g., roads) and dynamic elements (e.g., vehicles, pedestrians). However, the high cost of annotation procedures of full-supervised methods limits the capability of the visual BEV semantic segmentation, which usually needs HD maps, 3D object bounding boxes, and camera extrinsic matrixes. In this paper, we present a novel semi-supervised framework for visual BEV semantic segmentation to boost performance by exploiting unlabeled images during the training. A consistency loss that makes full use of unlabeled data is then proposed to constrain the model on not only semantic prediction but also the BEV feature. Furthermore, we propose a novel and effective data augmentation method named conjoint rotation which reasonably augments the dataset while maintaining the geometric relationship between the FV images and the BEV semantic segmentation. Extensive experiments on the nuScenes dataset show that our semi-supervised framework can effectively improve prediction accuracy. To the best of our knowledge, this is the first work that explores improving visual BEV semantic segmentation performance using unlabeled data. The code is available at https://github.com/Junyu-Z/Semi-BEVseg.

We introduce a new algorithm that utilizes semantic information to enhance feature matching in visual SLAM pipelines. The proposed method constructs a high-dimensional semantic descriptor for each detected ORB feature. When integrated with traditional visual ones, these descriptors aid in establishing accurate tentative point correspondences between consecutive frames. Additionally, our semantic descriptors enrich 3D map points, enhancing loop closure detection by providing deeper insights into the underlying map regions. Experiments on public large-scale datasets demonstrate that our technique surpasses the accuracy of established methods. Importantly, given its detector-agnostic nature, our algorithm also amplifies the efficacy of modern keypoint detectors, such as SuperPoint. The implementation of our algorithm can be found on Github 3.

Visual Place Recognition (VPR) is critical for navigation and loop closure in autonomous driving tasks, mitigating the impact of shift errors caused by dynamic changes in the environment. Due to the limited ability of backbone networks and extreme environmental changes, current methods fail to capture foundational semantic details that include the distinctive attributes for unique place identification. To address this problem, we propose a new visual token-guided VPR framework that contains a semantic-focused patch tokenizer and a multi-branch Mixer. To mitigate the inference from place-unrelated objects, the semantic-focused patch tokenizer exploits attention-based channel selection and spatial partition, which efficiently captures important semantic information within the channels and preserve spatial relationships among the backbone features. To extract abstract features with spatial structure information, the multi-branch Mixer utilizes a multi-branch structure to aggregate local and global position information, improving the robustness of global representations to environmental changes. Experimental results demonstrate that our method outperforms state-of-the-art methods, achieving 85.3% Recall@1 on the MSLS val dataset and 59.1% Recall@1 on the Nordland dataset when using ResNet18 as the backbone.

For the task of simultaneous monocular depth and visual odometry estimation, we propose learning self-supervised transformer-based models in two steps. Our first step consists in a generic pretraining to learn 3D geometry, using cross-view completion objective (CroCo), followed by self-supervised finetuning on non-annotated videos. We show that our self-supervised models can reach state-of-the-art performance ’without bells and whistles’ using standard components such as visual transformers, dense prediction transformers and adapters. We demonstrate the effectiveness of our proposed method by running evaluations on six benchmark datasets, both static and dynamic, indoor and outdoor, with synthetic and real images. For all datasets, our method outperforms state-of-the-art methods, in particular for depth prediction task.

Robotic systems often face challenges when attempting to grasp a target object due to interference from surrounding items. We propose a Deep Reinforcement Learning (DRL) method that develops joint policies for grasping and pushing, enabling effective manipulation of target objects within untrained, densely cluttered environments. In particular, a dual RL model is introduced, which presents high resilience in handling complicated scenes, reaching an average of 98% task completion in simulation and real-world scenes. To evaluate the proposed method, we conduct comprehensive simulation experiments in three distinct environments: densely packed building blocks, randomly positioned building blocks, and common household objects. Further, real-world tests are conducted using actual robots to confirm the robustness of our approach in various untrained and highly cluttered environments. The results from experiments underscore the superior efficacy of our method in both simulated and real-world scenarios, outperforming recent state-of-the-art methods. To ensure reproducibility and further the academic discourse, we make available a demonstration video, the trained models, and the source code for public access. https://sites.google.com/view/pushandgrasp/home.

Most existing 6-DoF robot grasping solutions depend on strong supervision on grasp pose to ensure satisfactory performance, which could be laborious and impractical when the robot works in some restricted area. To this end, we propose a self-supervised 6-DoF grasp pose detection framework via an Augmented Reality (AR) teleoperation system that can efficiently learn human demonstrations and provide 6-DoF grasp poses without grasp pose annotations. Specifically, the system collects the human demonstration from the AR environment and contrastively learns the grasping strategy from the demonstration. For the real-world experiment, the proposed system leads to satisfactory grasping abilities and learning to grasp unknown objects within three demonstrations.

The inherent scale ambiguity issue greatly limits the performance of monocular visual odometry. In recent years, a variety of methods have been proposed for self-supervised learning of ego-motion and depth estimation, incorporating specifically designed scale-consistency constraints that utilize estimated depth as a reference. However, these existing methods neglect the influence of the depth uncertainty introduced by the dominant photometric loss, which leads to unreliable depth estimation in difficult regions and detrimentally affects scale alignment. To solve these problems, we introduces a feature-based visual odometry learning system with an effective scale recovery strategy in this paper. Additionally, we propose a learning method to estimate the photometric-sensitive depth uncertainty for guiding the scale recovery. The proposed method is evaluated on KITTI odometry, and the experimental results demonstrate that our system can predict scale-consistent trajectories from monocular videos and achieves state-of-the-art performance. Moreover, the proposed method achieves competitive performance on KITTI depth estimation.

The human shoulder, with its glenohumeral joint, tendons, ligaments, and muscles, allows for the execution of complex tasks with precision and efficiency. However, current robotic shoulder designs lack the compliance and compactness inherent in their biological counterparts. A major limitation of these designs is their reliance on external sensors like rotary encoders, which restrict mechanical joint design and introduce bulk to the system. To address this constraint, we present a bio-inspired antagonistic robotic shoulder with two degrees of freedom powered by self-sensing hydraulically amplified self-healing electrostatic actuators. Our artificial muscle design decouples the high-voltage electrostatic actuation from the pair of low-voltage self-sensing electrodes. This approach allows for proprioceptive feedback control of trajectories in the task space while eliminating the necessity for any additional sensors. We assess the platform’s efficacy by comparing it to a feedback control based on position data provided by a motion capture system. The study demonstrates closed-loop controllable robotic manipulators based on an inherent self-sensing capability of electrohydraulic actuators. The proposed architecture can serve as a basis for complex musculoskeletal joint arrangements.

Decimeter scale robots in human environments are small relative to obstacles they encounter, making them prone to flipping over and needing to self-right. We present a multifaceted shell that by its geometry alone enables the hexapedal robot MediumANT to passively self-right without the need for additional sensory feedback. We designed the shell by specifying the cross-sectional geometry in the yz and xy planes such that the robot returns to an upright position by rolling around the longitudinal (x) axis, and then tweaked the design to reduce the number of faces. We then attached the shell to the robot by modifying some of its chassis structural plates to extend to and support the shell. We evaluated the effectiveness of the shell in two experimental scenarios: passive righting – balancing the robot on each face of the shell before releasing the robot – and an intentional fall – walking the robot off a ledge at various approach angles. As intended by our design, the robot recovered the upright orientation from all starting faces in the passive righting test and righted itself and continued walking in all falling trials. This work presents an example of using biologically inspired simplicity to solve what would otherwise be a technically challenging problem.

A general-purpose service robot (GPSR), which can execute diverse tasks in various environments, requires a system with high generalizability and adaptability to tasks and environments. In this paper, we first developed a top-level GPSR system for worldwide competition (RoboCup@Home2023) based on multiple foundation models. This system is both generalizable to variations and adaptive by prompting each model. Then, by analyzing the performance of the developed system, we found three types of failure in more realistic GPSR application settings: insufficient information, incorrect plan generation, and plan execution failure. We then propose the self-recovery prompting pipeline, which explores the necessary information and modifies its prompts to recover from failure. We experimentally confirm that the system with the self-recovery mechanism can accomplish tasks by resolving various failure cases. https://sites.google.com/view/srgpsr

We present a robotic system for the assembly of 3D discrete lattice structures in which the robots are able to self-reproduce, such that the assembly system may scale its own parallelization. Robots and structures are made from a set of compatible building blocks, or voxels, which can be assembled and reassembled into more complex structures. Robotic modules are made by combining actuators with a functional voxel, which routes electrical power and signals. Robotic modules then assemble into reconfigurable robots via a reversible solder joint. The robot assembles higher performance structures using a set of construction voxels, which do not contain electrical features. This paper describes the design, development, and evaluation of this assembly system, including the robotic hardware, lattice material, and planning and controls methods. We demonstrate the system through a set of fundamental assembly tasks: the robot assembling another robot, and the two robots collaborating to assemble a small structure.

Modeling and control of the human musculoskele-tal system is important for understanding human motor functions, developing embodied intelligence, and optimizing human-robot interaction systems. However, current human musculoskeletal models are restricted to a limited range of body parts and often with a reduced number of muscles. There is also a lack of algorithms capable of controlling over 600 muscles to generate reasonable human movements. To fill this gap, we build a musculoskeletal model (MS-HUMAN-700) with 90 body segments, 206 joints, and 700 muscle-tendon units, allowing simulation of full-body dynamics and interaction with various devices. We develop a new algorithm using low-dimensional representation and hierarchical deep reinforcement learning to achieve state-of-the-art full-body control. We validate the effectiveness of our model and algorithm in simulations with real human locomotion data. The musculoskeletal model, along with its control algorithm, will be made available to the research community to promote a deeper understanding of human motion control and better design of interactive robots.Project page: https://lnsgroup.cc/research/MS-Human-700

Equipping multi-fingered robots with tactile sensing is crucial for achieving the precise, contact-rich, and dexterous manipulation that humans excel at. However, relying solely on tactile sensing fails to provide adequate cues for reasoning about objects’ spatial configurations, limiting the ability to correct errors and adapt to changing situations. In this paper, we present Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances tactile-based dexterity by optimizing dexterous policies using vision-based rewards. First, we use a contrastive-based objective to learn visual representations. Next, we construct a reward function using these visual representations through optimal-transport based matching on one human demonstration. Finally, we use online reinforcement learning on our robot to optimize tactile-based policies that maximize the visual reward. On six challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping slender objects, TAVI achieves a success rate of 73% using our four-fingered Allegro robot hand. The increase in performance is 108% higher than policies using tactile and vision-based rewards and 135% higher than policies without tactile observational input. Robot videos are best viewed on our project website: https://see-to-touch.github.io/.

Efficiently performing intervention tasks underwater is crucial in various commercial and scientific sectors; however, propeller-driven vehicles face limitations due to their floating nature. In Remotely Operated Vehicles (ROVs) operations, this can be compensated by the ability of the operator, but they come with high operational costs. Instead, Autonomous Underwater Vehicles (AUVs) have shown promise, but demonstrated intervention tasks are limited to controlled environments or docked. To address these limitations, we focused on the use of Underwater Legged Robots (ULRs), which offer greater stability and agile seabed mobility thanks to their legged propulsion system. This paper presents the field demonstration of teleoperated pick-and-place tasks using the ULR SILVER2 for which a novel stance control, Graphic User Interface (GUI), and tendon-driven gripper have been developed based on the lessons learned through several hours of field use. The methodology is validated through four field trials, including missions in both shallow water and open sea environments. The trials involve picking and placing various objects, such as plastic bottles, bags, and cans. The results demonstrate successful teleoperated object grasping and manipulation in real-world conditions, with collection times ranging from a few minutes to around ten minutes. Overall, this research contributes to advancing the capabilities of ULRs and lays the foundation for future underwater intervention missions in various scientific and industrial applications, aligning with the goals of the Decade of Ocean Science for Sustainable Development.