IROS 2024 Papers

Humanoid robots have potential applications across diverse sectors, including education, healthcare, and customer service. This paper presents a project on designing and building a low-cost humanoid robot equipped with a flex sensor-based movement mechanism, highlighting its compatibility with Raspberry Pi and microcontrollers such as Arduino Uno and Nano. The project aims to investigate the robot's relevance and effectiveness within educational settings to showcase how a low-cost humanoid robot can potentially support the United Nations' fourth Sustainable Development Goal (UN SDG4) by improving access to quality education through innovative robotics solutions. The robot was tested in a cycle two school (covering Grades 5 to 8 (ages 10 to 13)) in Dubai, United Arab Emirates. It was integrated into math, science, and design technology classes to assess its functionality and efficiency. Surveys conducted among students and teachers showed a high level of acceptance towards the robot, with over 85% of respondents expressing positive attitudes about its presence and interaction in the classroom. However, teachers and students provided feedback concerning the robot's shape, capabilities, and movement mechanism. Teachers also appreciated the robot's alignment with the UN SDG4, stating its capability to support students learning and engagement. The authors highlighted the robot's potential to assist students with sensory challenges, such as hearing and vision impairments, and learning difficulties like dyslexia while emphasizing their commitment to enhancing its accessibility features for a more inclusive learning environment.

In this paper, with the goal of enhancing the safety of current colonoscopic procedures and providing the pressure and location of the contact between the colonoscope and the colon’s surface, we propose design and development of a unique Soft Pressure Sensing Sleeve (SPSS). SPSS can seamlessly be integrated with the existing colonoscopic devices and would not change the existing diagnosis workflow. The pressure sensing of SPSS is performed based on the resistance change of a liquid metal (i.e., Gallium) embedded into several micro-channels located within SPSS’s deformable sleeve when it interacts with the colon surface. To demonstrate functionality of the SPSS, without loss of generality, in this paper, we designed and fabricated a SPSS with 4 sensing regions. We also proposed and experimentally evaluated an empirical calibration function for this sensor. Results demonstrate high accuracy (RMSE=2.45 and mean absolute error <3%) of the proposed calibration function compared with the evaluation experiments.

Depth estimation is a cornerstone for autonomous driving, yet acquiring per-pixel depth ground truth for supervised learning is challenging. Self-Supervised Surround Depth Estimation (SSSDE) from consecutive images offers an economical alternative. While previous SSSDE methods have proposed different mechanisms to fuse information across images, few of them explicitly consider the cross-view constraints, leading to inferior performance, particularly in overlapping regions. This paper proposes an efficient and consistent pose estimation design and two loss functions to enhance cross-view consistency for SSSDE. For pose estimation, we propose to use only front-view images to reduce training memory and sustain pose estimation consistency. The first loss function is the dense depth consistency loss, which penalizes the difference between predicted depths in overlapping regions. The second one is the multi-view reconstruction consistency loss, which aims to maintain consistency between reconstruction from spatial and spatial-temporal contexts. Additionally, we introduce a novel flipping augmentation to improve the performance further. Our techniques enable a simple neural model to achieve state-of-the-art performance on the DDAD and nuScenes datasets. Last but not least, our proposed techniques can be easily applied to other methods. The code is available at https://github.com/denyingmxd/CVCDepth.

This paper enhances model-based offline inverse reinforcement learning (IRL) by refining conservative Markov decision process (MDP) frameworks, traditionally employing uncertainty penalties to deter exploitation in uncertain areas. Existing methods, dependent on neural network ensembles to model MDP dynamics and quantify uncertainty through ensemble prediction heuristics, face limitations: they presume Gaussian-distributed state transitions, leading to simplified environmental representations. Additionally, ensemble modeling often results in high variance, indicating potential overfitting and a lack of generalizability. Moreover, the heuristic reliance for uncertainty quantification struggles to fully grasp environmental complexities, offering an incomplete foundation for informed decisions. Maintaining multiple models also demands substantial computational resources. Addressing these shortcomings, we propose leveraging score-based diffusion generative models for dynamic modeling. This method significantly broadens the scope of representable target distributions, surpassing Gaussian constraints. It not only improves the accuracy of transition modeling but also roots uncertainty quantification in diffusion models’ theoretical underpinnings, enabling more precise and dependable reward regularization. We further innovate by incorporating a transition stability regularizer (TSR) into the reward estimation. This novel element embeds stability into the reward learning process, diminishing the influence of transition variability and promoting more consistent policy optimization. Our empirical studies on diverse Mujoco robotic control tasks demonstrate that our diffusion-based methodology not only furnishes more accurate transition estimations but also surpasses conventional ensemble approaches in policy effectiveness. The addition of the TSR marks a distinctive advancement in offline IRL by enhancing the reward and policy learning efficacy. Code: https://github.com/Gabriel... Show More

Accurate state estimation plays a critical role in ensuring the robust control of humanoid robots, particularly in the context of learning-based control policies for legged robots. However, there is a notable gap in analytical research concerning estimations. Therefore, we endeavor to further understand how various types of estimations influence the decision-making processes of policies. In this paper, we provide quantitative insight into the effectiveness of learned state estimations, employing saliency analysis to identify key estimation variables and optimize their combination for humanoid locomotion tasks. Evaluations assessing tracking precision and robustness are conducted on comparative groups of policies with varying estimation combinations in both simulated and real-world environments. Results validated that the proposed policy is capable of crossing the sim-to-real gap and demonstrating superior performance relative to alternative policy configurations.

Robotic weed flaming is a new and environmentally friendly approach to weed removal in the agricultural field. Using a mobile manipulator equipped with a blowtorch, we design a new system and algorithm to enable effective weed flaming, which requires robotic manipulation with a soft and deformable end effector, as the thermal coverage of the flame is affected by dynamic or unknown environmental factors such as gravity, wind, atmospheric pressure, fuel tank pressure, and pose of the nozzle. System development includes overall design, hardware integration, and software pipeline. To enable precise weed removal, the greatest challenge is to detect and predict dynamic flame coverage in real time before motion planning, which is quite different from a conventional rigid gripper in grasping or a spray gun in painting. Based on the images from two onboard infrared cameras and the pose information of the blowtorch nozzle on a mobile manipulator, we propose a new dynamic flame coverage model. The flame model uses a center-arc curve with a Gaussian cross-section model to describe the flame coverage in real time. The experiments have demonstrated the working system and shown that our model and algorithm can achieve a mean average precision (mAP) of more than 76% in the reprojected images during online prediction.

Honeybees are one of the most important pollinators in the ecosystem. Unfortunately, the dynamics of living honeybee colonies are not well understood due to their complexity and difficulty of observation. In our project “RoboRoyale”, we build and operate a robot to be a part of a bio-hybrid system, which currently observes the honeybee queen in the colony and physically tracks it with a camera. Apart from tracking and observing the queen, the system needs to monitor the state of the honeybee comb which is most of the time occluded by workerbees. This introduces a necessary tradeoff between tracking the queen and visiting the rest of the hive to create a daily map. We aim to collect the necessary data more effectively. We evaluate several mapping methods that consider the previous observations and forecasted densities of bees occluding the view. To predict the presence of bees, we use previously established maps of dynamics developed for autonomy in human-populated environments. Using data from the last observational season, we show significant improvement of the informed comb mapping methods over our current system. This will allow us to use our resources more effectively in the upcoming season.

Detecting the micromovements of eyes that reflect a person’s inner state can be an essential step in many applications, but most eye trackers need to fix the subject’s head using a chin rest to obtain sufficient data quality. We propose a stand-alone eye tracker that utilizes two 500 fps cameras, a pair of rotating mirrors for gaze control, a liquid lens for focus control and an intensity-controllable light source, and describe how the proposed system works in real-time. The experimental results show that our system covers more than twice as wide a measurement range in the depth direction as the conventional eye tracker while achieving sufficient data quality to analyze microsaccades with an amplitude of down to 0.2 deg. We also examine the practical use of the proposed system for microsaccade detection.

Model-based controllers using a linearized model around the system’s equilibrium point is a common approach in the control of a wheeled humanoid due to their less computational load and ease of stability analysis. However, controlling a wheeled humanoid robot while it lifts an unknown object presents significant challenges, primarily due to the lack of knowledge in object dynamics. This paper presents a framework designed for predicting the new equilibrium point explicitly to control a wheeled-legged robot with unknown dynamics. We estimated the total mass and center of mass of the system from its response to initially unknown dynamics, then calculated the new equilibrium point accordingly. To avoid using additional sensors (e.g., force torque sensor) and reduce the effort of obtaining expensive real data, a data-driven approach is utilized with a novel real-to-sim adaptation. A more accurate nonlinear dynamics model, offering a closer representation of real-world physics, is injected into a rigid-body simulation for real-to-sim adaptation. The nonlinear dynamics model parameters were optimized using Particle Swarm Optimization. The efficacy of this framework was validated on a physical wheeled inverted pendulum, a simplified model of a wheeled-legged robot. The experimental results indicate that employing a more precise analytical model with optimized parameters significantly reduces the gap between simulation and reality, thus improving the efficiency of a model-based controller in controlling a wheeled robot with unknown dynamics.

Conventional approaches to grasp planning re- quire perfect knowledge of an object’s pose and geometry. Uncertainties in these quantities induce uncertainties in the quality of planned grasps, which can lead to failure. Classically, grasp robustness refers to the ability to resist external disturbances after grasping an object. In contrast, this work studies robustness to intrinsic sources of uncertainty like object pose or geometry affecting grasp planning before execution. To do so, we develop a novel analytic theory of grasping that reasons about this intrinsic robustness by characterizing the effect of friction cone uncertainty on a grasp’s force closure status. We apply this result in two ways. First, we analyze the theoretical guarantees on intrinsic robustness of two grasp metrics in the literature, the classical Ferrari-Canny metric and more recent min-weight metric. We validate these results with hardware trials that compare grasps synthesized with and without robustness guarantees, showing a clear improvement in success rates. Second, we use our theory to develop a novel analytic notion of probabilistic force closure, which we show can generate unique, uncertainty-aware grasps in simulation.

In this work, we propose a novel method to supervise 3D Gaussian Splatting (3DGS) scenes using optical tactile sensors. Optical tactile sensors have become widespread in their use in robotics for manipulation and object representation; however, raw optical tactile sensor data is unsuitable to directly supervise a 3DGS scene. Our representation leverages a Gaussian Process Implicit Surface to implicitly represent the object, combining many touches into a unified representation with uncertainty. We merge this model with a monocular depth estimation network, which is aligned in a two stage process, coarsely aligning with a depth camera and then finely adjusting to match our touch data. For every training image, our method produces a corresponding fused depth and uncertainty map. Utilizing this additional information, we propose a new loss function, variance-weighted depth supervised loss, for training the 3DGS scene model. We leverage the DenseTact optical tactile sensor and RealSense RGB-D camera to show that combining touch and vision in this manner leads to quantitatively and qualitatively better results than vision or touch alone in few-view scene synthesis on opaque, reflective and transparent objects. Please see our project page at armlabstanford.github.io/touchgs.

Torque ripple reduction in quasi-direct drive (QDD) motors is crucial in their robotic applications for dynamic locomotion and dexterous manipulation. In this paper, we present a novel approach for reducing torque ripples of QDD motors, which integrates an angle-based repetitive learning observer (ARLO) and a model predictive control-based field-oriented controller (MPC-FOC). The proposed method successfully improves the torque loop control bandwidth and surpasses conventional proportional-integral (PI) controllers owing to the integrated physical constraints inside MPC. Additionally, the ARLO portion is able to mitigate ripple caused by the inherent cogging torque in brushless motors and also the periodic friction torque from the planetary gearboxes in QDD systems. The effectiveness of the proposed method is demonstrated through both simulation of a single QDD motor and experiments on a two-degree-of-freedom robotic leg, where the performance improvement can be 72.7% in speed tracking and 58.5% in trajectory tracking. The proposed method shows great potential in facilitating smooth motion and precise force control in future robotic applications.

Autonomous robots exploring unknown environments face a significant challenge: navigating effectively without prior maps and with limited external feedback. This challenge intensifies in sparse reward environments, where traditional exploration techniques often fail. In this paper, we present TopoNav, a novel topological navigation framework that integrates active mapping, hierarchical reinforcement learning, and intrinsic motivation to enable efficient goal-oriented exploration and navigation in sparse-reward settings. TopoNav dynamically constructs a topological map of the environment, capturing key locations and pathways. A two-level hierarchical policy architecture, comprising a high-level graph traversal policy and low-level motion control policies, enables effective navigation and obstacle avoidance while maintaining focus on the overall goal. Additionally, TopoNav incorporates intrinsic motivation to guide exploration towards relevant regions and frontier nodes in the topological map, addressing the challenges of sparse extrinsic rewards. We evaluate TopoNav both in the simulated and real-world off-road environments using a Clearpath Jackal robot, across three challenging navigation scenarios: goal-reaching, feature-based navigation, and navigation in complex terrains. We observe an increase in exploration coverage by 7-20%, in success rates by 9-19%, and reductions in navigation times by 15-36% across various scenarios, compared to state-of-the-art methods.

The exploration of robotic dexterous hands utilizing tools has recently attracted considerable attention. A significant challenge in this field is the precise awareness of a tool’s pose when grasped, as occlusion by the hand often degrades the quality of the estimation. Additionally, the tool’s overall pose often fails to accurately represent the contact interaction, thereby limiting the effectiveness of vision-guided, contact-dependent activities. To overcome this limitation, we present the innovative TOOLEE dataset, which, to the best of our knowledge, is the first to feature affordance segmentation of a tool’s end-effector (EE) along with its defined 6D pose based on its usage. Furthermore, we propose the ToolEENet framework for accurate 6D pose estimation of the tool’s EE. This framework begins by segmenting the tool’s EE from raw RGB-D data, then uses a diffusion model-based pose estimator for 6D pose estimation at a category-specific level. Addressing the issue of symmetry in pose estimation, we introduce a symmetry-aware pose representation that enhances the consistency of pose estimation. Our approach excels in this field, demonstrating high levels of precision and generalization. Furthermore, it shows great promise for application in contact-based manipulation scenarios. All data and codes are available on the project website: https://tooleenet-iros2024.github.io/

How can a robot provide unobtrusive physical support within a group of humans? We present Attentive Support, a novel interaction concept for robots to support a group of humans. It combines scene perception, dialogue acquisition, situation understanding, and behavior generation with the common-sense reasoning capabilities of Large Language Models (LLMs). In addition to following user instructions, Attentive Support is capable of deciding when and how to support the humans, and when to remain silent to not disturb the group. With a diverse set of scenarios, we show and evaluate the robot’s attentive behavior, which supports and helps the humans when required, while not disturbing if no help is needed.

Previous attempts to integrate Neural Radiance Fields (NeRF) into the Simultaneous Localization and Mapping (SLAM) framework either rely on the assumption of static scenes or require the ground truth camera poses, which impedes their application in real-world scenarios. This paper proposes a time-varying representation to track and reconstruct the dynamic scenes. Firstly, two processes, a tracking process and a mapping process, are maintained simultaneously in our framework. In the tracking process, all input images are uniformly sampled and then progressively trained in a self-supervised paradigm. In the mapping process, we leverage motion masks to distinguish dynamic objects from the static background, and sample more pixels from dynamic areas. Secondly, the parameter optimization for both processes is comprised of two stages: the first stage associates time with 3D positions to convert the deformation field to the canonical field. The second stage associates time with the embeddings of the canonical field to obtain colors and a Signed Distance Function (SDF). Lastly, we propose a novel keyframe selection strategy based on the overlapping rate. Our approach is evaluated on two synthetic datasets and one real-world dataset, and the experiments validate that our method achieves competitive results in both tracking and mapping when compared to existing state-of-the-art NeRF-based dynamic SLAM systems.

Prior research has demonstrated the effectiveness of end-to-end deep learning for robotic navigation, where the control signals are directly derived from raw sensory data. However, the majority of existing end-to-end navigation solutions are predominantly camera-based. In this paper, we introduce TinyLidarNet, a lightweight 2D LiDAR-based end-to-end deep learning model for autonomous racing. An F1TENTH vehicle using TinyLidarNet won 3rd place in the 12th F1TENTH Autonomous Grand Prix competition, demonstrating its competitive performance. We systematically analyze its performance on untrained tracks and computing requirements for real-time processing. We find that TinyLidarNet’s 1D Convolutional Neural Network (CNN) based architecture significantly outperforms widely used Multi-Layer Perceptron (MLP) based architecture. In addition, we show that it can be processed in real-time on low-end micro-controller units (MCUs).

In this article, we present a control barrier function (CBF)-based control strategy for safe and precise landing of an unmanned aerial vehicle (UAV) on a moving target. The CBF is time-varying, as it depends on the velocity of the landing platform and captures three crucial safety constraints: (a) collision avoidance with the landing platform, (b) precise vertical descent on a narrow landing platform, and (c) ground clearance throughout the landing maneuver. The proposed CBF’s parameters can be adjusted to set the desired width and height of the descending cone. A quadratic programbased CBF safety filter is designed, which takes a nominal position tracking control input and yields a minimally invasive control input that enforces the safety constraints throughout the landing maneuver. The controller’s feasibility is analyzed and its performance is validated through multiple experiments using a quadrotor UAV and an unmanned ground vehicle.

Resource sharing is a crucial part of a multi-robot system. We propose a Boolean satisfiability problem (SAT) based approach to resource sharing. Our key contributions are an algorithm for converting any constrained assignment to a weighted-SAT based optimization. We propose a theorem that allows optimal resource assignment problems to be solved via repeated application of a SAT solver. Additionally we show a way to encode continuous time ordering constraints using Conjunctive Normal Form (CNF). We benchmark our new algorithms and show that they can be used in an ad-hoc setting. We test our algorithms on a fleet of simulated and real world robots and show that the algorithms are able to handle real world situations. Our algorithms and test harnesses are open source and build on Open-RMF’s fleet management system.