IROS 2024 Papers

In this work, we address the challenge of deploying Neural Radiance Field (NeRFs) in Simultaneous Localization and Mapping (SLAM) under the condition of lacking depth information, relying solely on RGB inputs. The key to unlocking the full potential of NeRF in such a challenging context lies in the integration of real-world priors. A crucial prior we explore is the binary opacity prior of 3D space with opaque objects. To effectively incorporate this prior into the NeRF framework, we introduce a ternary-type opacity (TT) model instead, which categorizes points on a ray intersecting a surface into three regions: before, on, and behind the surface. This enables a more accurate rendering of depth, subsequently improving the performance of image warping techniques. Therefore, we further propose a novel hybrid odometry (HO) scheme that merges bundle adjustment and warping-based localization. Our integrated approach of TT and HO achieves state-of-the-art performance on synthetic and real-world datasets, in terms of both speed and accuracy. This breakthrough underscores the potential of NeRF-SLAM in navigating complex environments with high fidelity.

Musculoskeletal quadruped robots driven by pneumatic artificial muscles (PAMs) have great softness. Due to the softness, the proprioceptive information of PAMs (e.g. tension) reflects the environmental information. However, how to utilize this information for stable quadrupedal gait has been rarely explored. In this work, we utilized PAM tension for stable locomotion control over uneven terrain. We newly developed a durable tension sensor and proposed tension feedback control for quadruped locomotion over uneven terrain. Our proposed controller stabilizes the trunk posture by modulating the phase of the leg. To verify the effectiveness of the proposed controller, we implement it in a simple quadrupedal model and a musculoskeletal quadruped robot driven by PAMs. Through experiments, with tension feedback, the trunk posture oscillated more stably than that without the feedback. Furthermore, over uneven terrain, the running velocity with tension feedback was higher than that without the feedback in the robot experiment. These successful results will lead to more robust musculoskeletal quadruped robots that can be employed in the real-world environment.

Conducting visual assessments under the canopy using mobile robots is an emerging task in smart farming and forestry. However, it is challenging to register images across different data-collection days, especially across seasons, due to the self-occluding geometry and temporal dynamics in forests and orchards. This paper proposes a new approach for registering under-canopy image sequences in general and in these situations. Our methodology leverages standard GPS data and deep-learning-based perspective to bird’s-eye view conversion to provide an initial estimation of the positions of the trees in images and their association across datasets. Furthermore, it introduces an innovative strategy for extracting tree trunks and clean ground surfaces from noisy and sparse 3D reconstructions created from the image sequences, utilizing these features to achieve precise alignment. Our robust alignment method effectively mitigates position and scale drift, which may arise from GPS inaccuracies and Sparse Structure from Motion (SfM) limitations. We evaluate our approach on three challenging real-world datasets, demonstrating that our method outperforms ICP-based methods on average by 50%, and surpasses FGR and TEASER++ by over 90% in alignment accuracy. These results highlight our method’s cost efficiency and robustness, even in the presence of severe outliers and sparsity. https://github.com/VIS4ROB-lab/bev_undercanopy_registration

This paper introduces a novel approach to enhancing cross-view localization, focusing on the fine-grained, sequential localization of street-view images within a single known satellite image patch, a significant departure from traditional one-to-one image retrieval methods. By expanding to sequential image fine-grained localization, our model, equipped with a novel Temporal Attention Module (TAM), leverages contextual information to significantly improve sequential image localization accuracy. Our method shows substantial reductions in both mean and median localization errors on the Cross-View Image Sequence (CVIS) dataset, outperforming current state-of-the-art single-image localization techniques. Additionally, by adapting the KITTI-CVL dataset into sequential image sets, we not only offer a more realistic dataset for future research but also demonstrate our model’s robust generalization capabilities across varying times and areas, evidenced by a 75.3% reduction in mean distance error in cross-view sequential image localization.

Autonomous driving requires an accurate representation of the environment. A strategy toward high accuracy is to fuse data from several sensors. Learned Bird’s-Eye View (BEV) encoders can achieve this by mapping data from individual sensors into one joint latent space. For cost-efficient camera-only systems, this provides an effective mechanism to fuse data from multiple cameras with different views. Accuracy can further be improved by aggregating sensor information over time. This is especially important in monocular camera systems to account for the lack of explicit depth and velocity measurements, such that decision-critical information about distance distances and motions of other objects is easily accessible. Thereby, the effectiveness of developed BEV encoders crucially depends on the operators used to aggregate temporal information and on the used latent representation spaces.We analyze BEV encoders proposed in the literature and compare their effectiveness, quantifying the effects of aggregation operators and latent representations. While most existing approaches aggregate temporal information either in image or in BEV latent space, our analyses and performance comparisons suggest that these latent representations exhibit complementary strengths. Therefore, we develop a novel temporal BEV encoder, TempBEV, which integrates aggregated temporal information from both latent spaces. We consider subsequent image frames as stereo through time and leverage methods from optical flow estimation for temporal stereo encoding.Empirical evaluation on the NuScenes dataset shows a significant improvement by TempBEV over the baseline for 3D object detection and BEV segmentation. The ablation uncovers a strong synergy of joint temporal aggregation in the image and BEV latent space. These results indicate the overall effectiveness of our approach and make a strong case for aggregating temporal information in both image and BEV latent spaces. Show More

Team Coordination on Graphs with Risky Edges (TCGRE) is a recently emerged problem, in which a robot team collectively reduces graph traversal cost through support from one robot to another when the latter traverses a risky edge. Resembling the traditional Multi-Agent Path Finding (MAPF) problem, both classical and learning-based methods have been proposed to solve TCGRE, however, they lacked either computational efficiency or optimality assurance. In this paper, we reformulate TCGRE as a constrained optimization problem and perform a rigorous mathematical analysis. Our theoretical analysis shows the NP-hardness of TCGRE by reduction from the Maximum 3D Matching problem and that efficient decomposition is a key to tackle this combinatorial optimization problem. Furthermore, we design three classes of algorithms to solve TCGRE, i.e., Joint State Graph (JSG) based, coordination based, and receding-horizon sub-team based solutions. Each of these proposed algorithms enjoys different provable optimality and efficiency characteristics that are demonstrated in our extensive experiments.

This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spatial Instruction Maps (SIMs) are learned upon them. These maps can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator’s reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified diagrammatic instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster runtime.

This manuscript primarily aims to enhance the performance of whole-body controllers(WBC) for underactuated legged locomotion. We introduce a systematic parameter design mechanism for the floating-base feedback control within the WBC. The proposed approach involves utilizing the linearized model of unactuated dynamics to formulate a Linear Quadratic Regulator(LQR) and solving a Riccati gain while accounting for potential physical constraints through a second-order approximation of the log-barrier function. And then the user-tuned feedback gain for the floating base task is replaced by a new one constructed from the solved Riccati gain. Extensive simulations conducted in MuJoCo with a point bipedal robot, as well as real-world experiments performed on a quadruped robot, demonstrate the effectiveness of the proposed method. In the different bipedal locomotion tasks, compared with the user-tuned method, the proposed approach is at least 12% better and up to 50% better at linear velocity tracking, and at least 7% better and up to 47% better at angular velocity tracking. In the quadruped experiment, linear velocity tracking is improved by at least 3% and angular velocity tracking is improved by at least 23% using the proposed method.

This work tackles the problem of task-oriented dexterous hand pose synthesis, which involves generating a static hand pose capable of applying a task-specific set of wrenches to manipulate objects. Unlike previous approaches that focus solely on force-closure grasps, which are unsuitable for non-prehensile manipulation tasks (e.g., turning a knob or pressing a button), we introduce a unified framework covering force-closure grasps, non-force-closure grasps, and a variety of non-prehensile poses. Our key idea is a novel optimization objective quantifying the disparity between the Task Wrench Space (TWS, the desired wrenches predefined as a task prior) and the Grasp Wrench Space (GWS, the achievable wrenches computed from the current hand pose). By minimizing this objective, gradient-based optimization algorithms can synthe-size task-oriented hand poses without additional human demonstrations. Our specific contributions include 1) a fast, accurate, and differentiable technique for estimating the GWS boundary; 2) a task-oriented objective function based on the disparity between the estimated GWS boundary and the provided TWS boundary; and 3) an efficient implementation of the synthesis pipeline that leverages CUDA accelerations and supports large-scale parallelization. Experimental results on 10 diverse tasks demonstrate a 72.6% success rate in simulation. Furthermore, real-world validation for 4 tasks confirms the effectiveness of synthesized poses for manipulation. Notably, despite being primarily tailored for task-oriented hand pose synthesis, our pipeline can generate force-closure grasps 50 times faster than DexGraspNet while maintaining comparable grasp quality. Project page: https://pku-epic.github.io/TaskDexGrasp/.

This paper discusses a novel task-oriented design method for wire-driven flexible hands. For a monolithic hand fabricated using 3D printing, an analytical design method is proposed to enable it to perform the given tasks. First, the wiring-synergy equation, which relates the parameters of the hand mechanism, the wire tension, and the generated posture is derived based on an analytical model of a hand with wire drive systems. Next, the posture-synergy equation is derived, using principal component analysis for multiple desired postures given to perform a task. Based on the isomorphism of the mathematical structure in the two synergy equations, a method for designing a hand is developed. By quantitatively evaluating the posture reproducibility with respect to the number of wire drive systems, this method can analytically determine the mechanism parameters and wire tension for the desired postures. Subsequently, the proposed method is validated through case studies. Finally, a hand for an in-hand manipulation task is developed, and the feasibility of the proposed method is validated experimentally. The method potentially contributes to expediting the design procedure, increasing the accuracy of the posture reproduction, and reducing the number of actuators.

ReachBot, a proposed robotic platform, employs extendable booms as limbs for mobility in challenging environments, such as martian caves. When attached to the environment, ReachBot acts as a parallel robot, with reconfiguration driven by the ability to detach and re-place the booms. This ability enables manipulation-focused scientific objectives: for instance, through operating tools, or handling and transporting samples. To achieve these capabilities, we develop a two-part solution, optimizing for robustness against task uncertainty and stochastic failure modes. First, we present a mixed-integer stance planner to determine the positioning of ReachBot’s booms to maximize the task wrench space about the nominal point(s). Second, we present a convex tension planner to determine boom tensions for the desired task wrenches, accounting for the probabilistic nature of microspine grasping. We demonstrate improvements in key robustness metrics from the field of dexterous manipulation, and show a large increase in the volume of the manipulation workspace. Finally, we employ Monte-Carlo simulation to validate the robustness of these methods, demonstrating good performance across a range of randomized tasks and environments, and generalization to cable-driven morphologies. We make our code available at our project webpage, https://stanfordasl.github.io/reachbot_manipulation/

Modular reconfigurable manipulators enable quick adaptation and versatility to address different application environments and tailor to the specific requirements of the tasks. Task performance significantly depends on the manipulator’s mounted pose and morphology design, therefore posing the need of methodologies for selecting suitable modular robot configurations and mounted pose that can address the specific task requirements and required performance. Morphological changes in modular robots can be derived through a discrete optimization process that involves the selective addition or removal of modules. In contrast, the adjustment of the mounted pose operates within a continuous space, allowing for smooth and precise alterations in both orientation and position. This work introduces a computational framework that simultaneously optimizes the pose and morphology mounted on modular manipulators. The core of the work is that we design a mapping function that implicitly captures the morphological state of manipulators in the continuous space. This transformation function unifies the optimization of mounted pose and morphology within a continuous space. Furthermore, our optimization framework incorporates a array of performance metrics, such as minimum joint effort and maximum manipulability, and considerations for trajectory execution error and physical and collision constraints. To highlight our method’s benefits, we compare it with previous methods that framed such problems as a combinatorial optimization problem and demonstrate its practicality in selecting the modular robot configuration for executing a drilling task with the CONCERT modular robotic platform.

Underactuated manipulators reduce the number of bulky motors, thereby enabling compact and mechanically robust designs. However, fewer actuators than joints means that the manipulator can only access a specific manifold within the joint space, which is particular to a given hardware configuration and can be low-dimensional and/or discontinuous. Determining an appropriate set of hardware parameters for this class of mechanisms, therefore, is difficult - even for traditional task-based co-optimization methods. In this paper, our goal is to implement a task-based design and policy co-optimization method for underactuated, tendon-driven manipulators. We first formulate a general model for an underactuated, tendon-driven transmission. We then use this model to co-optimize a three-link, two-actuator kinematic chain using reinforcement learning. We demonstrate that our optimized tendon transmission and control policy can be transferred reliably to physical hardware with real-world reaching experiments.

Deep reinforcement learning (DRL) has shown remarkable success in simulation domains, yet its application in designing robot controllers remains limited, due to its singletask orientation and insufficient adaptability to environmental changes. To overcome these limitations, we present a novel adaptive agent that leverages transfer learning techniques to dynamically adapt policy in response to different tasks and environmental conditions. The approach is validated through the blimp control challenge, where multitasking capabilities and environmental adaptability are essential. The agent is trained using a custom, highly parallelized simulator built on IsaacGym. We perform zero-shot transfer to fly the blimp in the real world to solve various tasks. We share our code at https://github.com/robot-perception-group/adaptive_agent/.

In the field of aviation, the Detect and Avoid (DAA) problem deals with incorporating collision avoidance capabilities into current autopilot navigation systems. As an application of the Small Object Detection (SOD) problem, DAA presents the difficulties of a low signal-to-noise ratio and far range detection. Visual DAA is also susceptible to changing weather and lighting conditions at deployment. While current literature has presented many solutions for this, prior work has yet to study the robustness of the learning-based models for DAA. In this work, we show that standard techniques for improving robustness for object detection do not produce the desired results for DAA given the SOD constraints. We present targeted transformations, a zero-shot technique that can significantly improve robustness with minimal impact on accuracy. We demonstrate how to construct these transformations and evaluate our method on the current SOTA model for DAA, showing a 53.6% increase in recall. This makes our pipeline more robust to changes in lighting and environmental factors, and better able to detect potential threats. In the future, we hope to automate the transformation selection process, making it easier to adopt in different use cases.

Aerial robots are well-established technologies in environments characterized by reliable GNSS signals and favorable conditions for navigation based on cameras or LiDARs. However, their robustness is significantly challenged whenever ambient lighting is insufficient, GNSS signals are blocked, and range measurements are corrupted, for example, in underground, dark, or foggy environments. There, conventional navigation methods solely based on computer vision are very limited. This work proposes a completely novel approach to Aerial Tactile Odometry for pose estimation of aerial robots exploiting contact to precisely determine the system’s pose. By employing a compliant end-effector design with onboard tactile information by means of a trackball, we infer the complete UAV’s pose with respect to the environment, and the path traveled during contact. Through a large set of experiments, the proposed method shows centimeter-level accuracy for various relative orientations between the environment and the robot as well as for different trajectories. Akin to conventional dead-reckoning odometry methods in wheeled robotics, this method provides a valuable additional source of pose estimation, increasing the robustness of aerial robots – especially aerial manipulators – in the real world.

Robotic manipulation holds the potential to replace humans in the execution of tedious or dangerous tasks. However, control-based approaches are not suitable due to the difficulty of formally describing open-world manipulation in reality, and the inefficiency of existing learning methods. Therefore, applying manipulation in a wide range of scenarios presents significant challenges. In this study, we propose a novel framework for skill learning in robotic manipulation called Tactile Active Inference Reinforcement Learning (TactileAIRL), aimed at achieving efficient learning. To enhance the performance of reinforcement learning (RL), we introduce active inference, which integrates model-based techniques and intrinsic curiosity into the RL process. This integration improves the algorithm’s training efficiency and adaptability to sparse rewards. Additionally, we have designed universal tactile static and dynamic features based on vision-based tactile sensors, making our framework scalable to many manipulation tasks learning involving tactile feedback. Simulation results demonstrate that our method achieves significantly high training efficiency in objects pushing tasks. It enables agents to excel in both dense and sparse reward tasks with just few interaction episodes, surpassing the SAC baseline. Furthermore, we conduct physical experiments on a gripper screwing task using our method, which showcases the algorithm’s rapid learning capability and its potential for practical applications.

Recent developments in vision-based tactile sensing offer a simple means to enable robots to perceive touch interactions. However, existing sensors are primarily designed for small-scale applications like robotic hands, lacking research on their integration for large-sized robot bodies that can be leveraged for safe human-robot interactions. This paper explores the utilization of the previously-developed vision-based tactile sensing link (called TacLink) with soft skin as a safety control mechanism, which can serve as an alternative to conventional rigid robot links and impact observers. We characterize the behavior of a robot integrated with the soft TacLink in response to collisions, particularly employing a reactive control strategy. The controller is primarily driven by tactile force information acquired from the soft TacLink sensor through a data-driven sim2real learning method. Compared with a standard rigid link, the results obtained from collision experiments also confirm the advantages of our "soft" solution in impact resilience and in facilitating controls that are difficult to achieve with a stiff robot body. This study can act as a benchmark for assessing the efficiency of soft tactile-sensitive skins in reactive collision responses and open new safety standards for soft skin-based collaborative robots in human-robot interaction scenarios.

Significant work has been done on advancing localization and mapping in underwater environments. Still, state-of-the-art methods are challenged by low-texture environments, which is common for underwater settings. This makes it difficult to use existing methods in diverse, real-world scenes. In this paper, we present TURTLMap, a novel solution that focuses on textureless underwater environments through a real-time localization and mapping method. We show that this method is low-cost, and capable of tracking the robot accurately, while constructing a dense map of a low-textured environment in real-time. We evaluate the proposed method using real-world data collected in an indoor water tank with a motion capture system and ground truth reference map. Qualitative and quantitative results validate the proposed system achieves accurate and robust localization and precise dense mapping, even when subject to wave conditions. The project page for TURTLMap is https://umfieldrobotics.github.io/TURTLMap.