IROS 2024 Papers

Embodied navigation requires robots to understand and interact with the environment based on given tasks. Vision-Language Navigation (VLN) is an embodied navigation task, where a robot navigates within a previously seen and unseen environment, based on linguistic instruction and visual inputs. VLN agents need access to both local and global action spaces; former for immediate decision making and the latter for recovering from navigational mistakes. Prior VLN agents rely only on instruction-viewpoint alignment for local and global decision making and back-track to a previously visited viewpoint, if the instruction and its current viewpoint mismatches. These methods are prone to mistakes, due to the complexity of the instruction and partial observability of the environment. We posit that, back-tracking is sub-optimal and agent that is aware of its mistakes can recover efficiently. For optimal recovery, exploration should be extended to unexplored viewpoints (or frontiers). The optimal frontier is a recently observed but unexplored viewpoint that aligns with the instruction and is novel. We introduce a memory-based and mistake-aware path planning strategy for VLN agents, called StratXplore, that presents global and local action planning to select the optimal frontier for path correction. The proposed method collects all past actions and viewpoint features during navigation and then selects the optimal frontier suitable for recovery. Experimental results show this simple yet effective strategy improves the success rate on two VLN datasets with different task complexities.

Rod-driven soft robots (RDSR) with a well-balanced performance in terms of perception, precision, and intelligence have a great potential for application. Mathematical description and predicted sensing of deformable soft bodies are crucial to achieve controllable and intelligent behaviors of these robots. In this work, we propose a kinetostatic model for RDSR embedded with co-located sensors based on the Geometric Variable Strain (GVS) approach where local deformations, actuation lengths and external interactions are included. This approach allows us to estimate the shape of RDSR and predict the strain variation of soft bodies under internal and external interactions. Simulations and experimental results show that tip position errors are not greater than 1.8% with respect to the whole body length under different loads (0, 100, 200, 300 gf). The maximum error of predicted sensor length change is up to 2 mm and its percentage relative to the actual length does not exceed 4%. The results demonstrate the accuracy and effectiveness of the proposed model.

In-hand manipulation is challenging in robotics due to the intricate contact dynamics and high degrees of control freedom. Precise manipulation with high accuracy often requires tactile perception, which adds further complexity to the system. Despite the challenges in perception and control, the rolling stick problem is an essential and practical motion primitive with many demanding industrial applications. This work aims to learn the high-resolution tactile dynamics of the rolling stick. Specifically, we try manipulating a small stick using the Allegro hand equipped with the Digit vision-based tactile sensor. The learning framework includes an action filtering module, tactile perception module, and learning with uncertainty module, all designed to operate in low data regimes. With only 2.3% amount of data and 5.7% model complexity of previous similar work, our learned contact dynamics model achieves better grasp stability, sub-millimeter precision, and promising zero-shot generalizability across novel objects. The proposed framework demonstrates the potential for precise in-hand manipulation with tactile feedback on real hardware. The project source code is available at: https://github.com/duyipai/Allegro_Digit. A video presentation is available here.

Visual navigation has received significant attention recently. Most of the prior works focus on predicting navigation actions based on semantic features extracted from visual encoders. However, these approaches often rely on large datasets and exhibit limited generalizability. In contrast, our approach draws inspiration from traditional navigation planners that operate on geometric representations, such as occupancy maps. We propose StereoNavNet (SNN), a novel visual navigation approach employing a modular learning framework comprising perception and policy modules. Within the perception module, we estimate an auxiliary 3D voxel occupancy grid from stereo RGB images and extract geometric features from it. These features, along with user-defined goals, are utilized by the policy module to predict navigation actions. Through extensive empirical evaluation, we demonstrate that SNN outperforms baseline approaches in terms of success rates, success weighted by path length, and navigation error. Furthermore, SNN exhibits better generalizability, characterized by maintaining leading performance when navigating across previously unseen environments.

Probabilistic Movement Primitives (ProMPs) and their variants are powerful methods for enabling robots to learn complex tasks from human demonstrations, where motion trajectories are represented as stochastic processes with Gaussian assumptions. However, despite their computational efficiency, these methods have limited expressiveness in capturing the diversity found in human demonstrations, which are typically characterized by the multi-modality of motions. For example, when picking up an object partially obscured by an obstacle, some individuals may opt to go to the right, while others may choose the left side of the object. In this paper, we introduce Stein Movement Primitives (SMPs), a novel approach to probabilistic movement primitives. We formulate motion primitive adaptation as a non-parametric probabilistic inference using Stein Variational Gradient Descent (SVGD), thus avoiding any explicit posterior distribution assumptions and enabling the direct representation of the multi-modality in human demonstrations. We illustrate how our method can adapt robot motion to different scenarios while maintaining high similarity to the original demonstrations, even when the demonstrations are multi-modal. Experimentally, we demonstrate our approach to several domain adaptation problems using the LASA dataset and with a real robotic arm.

Decision Transformers (DTs) have been highly effective for offline reinforcement learning (RL) tasks, successfully modeling the sequences of actions in a given set of demonstrations. However, DTs may perform poorly in stochastic environments, which are prevalent in robotics scenarios. In this paper, we identify that the root cause of this performance degradation is the growing variance of returns-to-go, the signal used by DTs to predict actions, accumulated over the horizon. Building upon this insight, we propose an extension to DTs that allows them to be steered toward high-reward regions, where the expected returns are estimated using temporal difference learning. This way, we not only mitigate the growing variance problem but also eliminate the need for DTs to have access to returns-to-go during evaluation and deployment phases. We show that our method outperforms state-of-the-art offline RL methods in both simulated and real-world robotic arm environments.

Rolling element eccentric drives promise to be an easy-to-manufacture and performant gear system for robotic actuators. They share characteristics with other eccentric drives, such as strain wave and cycloidal drives, but use rolling elements instead of an eccentric gear. They offer reduced manufacturing complexity and costs by using readily available standard parts. Little research into rolling element eccentric drives is available, and their characteristics are still underexplored. This work uses a contact-based model to investigate the previously unknown stiffness of rolling element eccentric drives. Such calculation methods are well established for structurally similar components, such as cycloidal drives and roller bearings, and provide a high-level and computationally efficient model. Good stiffness models are critical for accurately predicting robotic actuator behavior and enabling better control of robotic systems. Additionally, the proposed model is used to calculate the contact forces under load occurring in rolling element eccentric drives. Contact forces are critical to calculating a drive’s load capacity, lifetime, and efficiency and serve as the foundation for further research. The mathematical description of the proposed model is derived, and the stiffness of a representative rolling element eccentric drive is calculated. Different manufacturing techniques, characterized by tolerance levels and material choices, are compared. Irrespective of manufacturing precision, similar stiffness curves result for drives made of steel, but higher contact forces result from less precise manufacturing. The stiffness of drives made from 3D printed plastic is considerably lower than that of drives made from steel. Additionally, the stiffness of rolling element eccentric drives is compared to similar eccentric drives, and a comparable twist-over-torque curve is shown.

Adaptive grippers enable easy and robust grasping of diverse objects by adapting to their shapes and enclosing them. However, determining the exact state of the hand remains challenging. This is not always straightforward but is often necessary to assess grip success, quality, or the pose of the object. In this work, we present two deep learning approaches using recurrent neural networks to successfully estimate the joint states of the Robotiq 3-Finger Adaptive Gripper. The models are compared with an existing analytical approach, which does not distinguish between the fingers of the hand and calculates the three angles of their respective joints using joint limits, contact information, and motor position in a transition model. We test the differences in accuracy with our networks by not distinguishing between the fingers as the analytical approach for the first model and by looking at the entire hand in the second model. Our experiments demonstrate that the model considering the entire hand outperforms the other two approaches, is more robust against object movements and achieves an average joint position accuracy of 2.29 degrees.

We propose a state estimation method that can accurately predict the robot’s privileged states to push the limits of quadruped robots in executing advanced skills such as jumping in the wild. In particular, we present the State Estimation Transformers (SET), an architecture that casts the state estimation problem as conditional sequence modeling. SET outputs the robot states that are hard to obtain directly in the real world, such as the body height and velocities, by leveraging a causally masked Transformer. By conditioning an autoregressive model on the robot’s past states, our SET model can predict these privileged observations accurately even in highly dynamic locomotions. We evaluate our methods on three tasks — running jumping, running backflipping, and running sideslipping — on a low-cost quadruped robot, Cyberdog2. Results show that SET can outperform other methods in estimation accuracy and transferability in the simulation as well as success rates of jumping and triggering a recovery controller in the real world, suggesting the superiority of such a Transformer-based explicit state estimator in highly dynamic locomotion tasks.

We introduce StaccaToe, a human-scale, electric motor-powered single-leg robot designed to rival the agility of human locomotion through two distinctive attributes: an actuated toe and a co-actuation configuration inspired by the human leg. Leveraging the foundational design of HyperLeg’s lower leg mechanism, we develop a stand-alone robot by incorporating new link designs, custom-designed power electronics, and a refined control system. Unlike previous jumping robots that rely on either special mechanisms (e.g., springs and clutches) or hydraulic/pneumatic actuators, StaccaToe employs electric motors without energy storage mechanisms. This choice underscores our ultimate goal of developing a practical, high-performance humanoid robot capable of human-like, stable walking as well as explosive dynamic movements. In this paper, we aim to empirically evaluate the balance capability and the exertion of explosive ground reaction forces of our toe and co-actuation mechanisms. Throughout extensive hardware and controller development, StaccaToe showcases its control fidelity by demonstrating a balanced tip-toe stance and dynamic jump. This study is significant for three key reasons: 1) StaccaToe represents the first human-scale, electric motor-driven single-leg robot to execute dynamic maneuvers without relying on specialized mechanisms; 2) our research provides empirical evidence of the benefits of replicating critical human leg attributes in robotic design; and 3) we explain the design process for creating agile legged robots, the details that have been scantily covered in academic literature.

This paper introduces a novel X-shaped walker with telescopic legs and investigates its control method with the aim of generating a stable wheel gait on a horizontal plane without including zero dynamics which is essentially unstable and difficult to stabilize. First, we outline a planar 6-DOF robot model with three control inputs, and describe the equations of motion and inelastic collision. Second, we design an output-following control system that smoothly controls the extension/contraction lengths of the legs and relative hip-joint angle to their target terminal values, and creates an asymmetric impact posture in the anteroposterior direction so that the robot can easily overcome the next potential barrier. The coefficients of the desired-time trajectory for each control output are updated with the position and velocity values immediately after each impact as the target initial values, so the generated leg motion and control inputs exhibit smooth time variation. The validity of the proposed gait generation method and the change trend of fundamental motion characteristics with respect to control parameters are investigated through numerical simulations.

Ensuring stable object placement is crucial to prevent objects from toppling over, breaking, or causing spills. When an object makes initial contact to a surface, and some force is exerted, the moment of rotation caused by the instability of the object’s placing can cause the object to rotate in a certain direction (henceforth referred to as direction of corrective rotation). Existing methods often employ a Force/Torque (F/T) sensor to estimate the direction of corrective rotation by detecting the moment of rotation as a torque. However, its effectiveness may be hampered by sensor noise and the tension of the external wiring of robot cables. To address these issues, we propose a method for stable object placing using GelSights, vision-based tactile sensors, as an alternative to F/T sensors. Our method estimates the direction of corrective rotation of objects using the displacement of the black dot pattern on the elastomeric surface of GelSight. We calculate the Curl from vector analysis, indicative of the rotational field magnitude and direction of the displacement of the black dots pattern. Simultaneously, we calculate the difference (Diff) of displacement between the left and right fingers’ GelSight’s black dots. Then, the robot can manipulate the objects’ pose using Curl and Diff features, facilitating stable placing. Across experiments, handling 18 differently characterized objects, our method achieves precise placing accuracy (less than 1-degree error) in nearly 100% of cases.

In the absence of a tether attachment mechanism that can provide infinitely large tension to the tethered robots moving on extreme planetary terrains, there is a limit on how much tension can be realistically generated or supported by the tether. In this paper, we consider a team of two robots tethered together moving on extreme terrains. The traction on the wheels of the robot and the friction between the tether and the tether attachment surfaces/objects (e.g., rocks) is the only way to support the tether tension. Given a path for the robots to navigate, we provide a systematic algorithm to check if the robots will be stable along the given path while considering the maximum constraints on the tension generated or supported by the tether. The results are validated via simulation experiments.

Spiking Neural Networks (SNNs) have gained attention for their apparent energy efficiency and significant biological interpretability, although they also face significant challenges such as prolonged latency and suboptimal tracking accuracy. Recent studies have explored the application of SNNs in object tracking tasks. Dynamic visual sensors (DVS) have become a popular way to implement SNN-based object tracking due to their asynchronous and spiking characteristics similar to SNNs. However, challenges such as the high cost of DVS cameras and the lack of object surface texture information hinder the utility and performance of DVS trackers. In contrast, RGB information has inherent advantages, including low acquisition cost and comprehensive object surface texture representation. However, RGB information is prone to excessive image blurring in low-light conditions or in fast-motion scenes. To address these challenges, we propose the “Motion Feature Extractor” and the "RGB-DVS Fusion Module". The “Motion Feature Extractor” can replace the DVS camera at a very low cost, and the "RGB-DVS Fusion Module" can deeply fuse the feature information of the two to make up for their respective deficiencies. In addition, we adopt a conversion method to obtain a lossless SNN version of the model. Through experiments, our model achieves a 13.6% improvement in the expected average overlap (EAO) index using only 1.47% of the energy consumption of SiamRPN (VOT2016 dataset). In addition, we deployed the model to a robot and then conducted tracking experiments, which confirmed that the model can operate on the robot losslessly with satisfactory results.

Recent advances in AI have led to significant results in robotic learning, including natural language-conditioned planning and efficient optimization of controllers using generative models. However, the interaction data remains the bottleneck for generalization. Getting data for grasping is a critical challenge, as this skill is required to complete many manipulation tasks. Quality-Diversity (QD) algorithms optimize a set of solutions to get diverse, high-performing solutions to a given problem. This paper investigates how QD can be combined with priors to speed up the generation of diverse grasps poses in simulation compared to standard 6-DoF grasp sampling schemes. Experiments conducted on 4 grippers with 2-to-5 fingers on standard objects show that QD outperforms commonly used methods by a large margin. Further experiments show that QD optimization automatically finds some efficient priors that are usually hard coded. The deployment of generated grasps on a 2-finger gripper and an Allegro hand shows that the diversity produced maintains sim-to-real transferability. We believe these results to be a significant step toward the generation of large datasets that can lead to robust and generalizing robotic grasping policies.

In this paper, we address a method that integrates reinforcement learning into the Monte Carlo tree search to boost online path planning under fully observable environments for automated parking tasks. Sampling-based planning methods under high-dimensional space can be computationally expensive and time-consuming. State evaluation methods are useful by leveraging the prior knowledge into the search steps, making the process faster in a real-time system. Given the fact that automated parking tasks are often executed under complex environments, a solid but lightweight heuristic guidance is challenging to compose in a traditional analytical way. To overcome this limitation, we propose a reinforcement learning pipeline with a Monte Carlo tree search under the path planning framework. By iteratively learning the value of a state and the best action among samples from its previous cycle’s outcomes, we are able to model a value estimator and a policy generator for given states. By doing that, we build up a balancing mechanism between exploration and exploitation, speeding up the path planning process while maintaining its quality without using human expert driver data.

The increase in non-biodegradable waste is a worldwide concern. Recycling facilities play a crucial role, but their automation is hindered by the complex characteristics of waste recycling lines like clutter or object deformation. In addition, the lack of publicly available labeled data for these environments makes developing robust perception systems challenging. Our work explores the benefits of multimodal perception for object segmentation in real waste management scenarios. First, we present SpectralWaste, the first dataset collected from an operational plastic waste sorting facility that provides synchronized hyperspectral and conventional RGB images. This dataset contains labels for several categories of objects that commonly appear in sorting plants and need to be detected and separated from the main trash flow for several reasons, such as security in the management line or reuse. Additionally, we propose a pipeline employing different object segmentation architectures and evaluate the alternatives on our dataset, conducting an extensive analysis for both multimodal and unimodal alternatives. Our evaluation pays special attention to efficiency and suitability for real-time processing and demonstrates how hyperspectral imaging can bring a boost to RGB-only perception in these realistic industrial settings without much computational overhead.

This paper introduces an innovative planner for prioritized multi-agent motion planning, employing a sequential integration of spatial and temporal designs. The planner initiates a smooth trajectory in space for each agent, ignoring the presence of other agents. Subsequently, by treating spatial collisions as 2D obstacles from a temporal perspective, the planner dynamically fine-tunes the trajectory-tracking speed of agents to avoid collisions, ensuring optimal time consumption for the last agent to reach the target as well. Additionally, the proposed approach systematically coordinates priority for each agent. The efficacy of the approach is validated through both simulations and comparative experiments with a recent planner.

Tracking changes in growing plants is important for automating phenotyping and robots managing crops. In this paper, we propose a system that uses a 3D model of plants along crop rows to enable a robotic platform to localize itself even in the presence of heavy changes and deforming the model to adapt the scene description to the new measurements. In particular, we focus on consumer RGB-D cameras due to their cost-effectiveness and ease of deployment on real platforms. Our approach exploits modern deep-learning-based feature descriptors and geometric information to obtain matches between 3D points corresponding to temporally distant sessions. We then use the associations in a non-rigid registration pipeline to obtain the final result, an updated representation of the 3D model that reflects plant changes. Using a standard RGB-D sensor, we validate our approach on a real-world dataset recorded in a glasshouse. We obtain accurate 4D models of the plants and track the plant traits’ evolution over time. We show, through experiments, that our method is applicable to interpolate plant organs’ evolution, a helpful result for phenotypic trait measurement. We see our approach as a relevant step toward 4D reconstruction for robotic agriculture in the wild.