IROS 2024 Papers

Reconstructing compositional 3D representations of scenes, where each object is represented with its own 3D model, is a highly desirable capability in robotics and augmented reality. However, most existing methods rely heavily on strong appearance priors for object discovery, therefore only working on those classes of objects on which the method has been trained, or do not allow for object manipulation, which is necessary to scan objects fully and to guide object discovery in challenging scenarios. We address these limitations with a novel interaction-guided and class-agnostic method based on object displacements that allows a user to move around a scene with an RGB-D camera, hold up objects, and finally outputs one 3D model per held-up object. Our main contribution to this end is a novel approach to detecting user-object interactions and extracting the masks of manipulated objects. On a custom-captured dataset, our pipeline discovers manipulated objects with 78.3% precision at 100% recall and reconstructs them with a mean chamfer distance of 0.90cm. Compared to Co-Fusion, the only comparable interaction-based and class-agnostic baseline, this corresponds to a reduction in chamfer distance of 73% while detecting 99% fewer false positives.

Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency significantly affects the reliability of existing PBA methods. To solve this problem, we propose a novel physically-based PBA method. Specifically, we introduce the physically-based weights regarding material, illumination, and light path. These weights distinguish the pixel pairs with different levels of photometric inconsistency. We also design corresponding models for material estimation based on sequential images and illumination estimation based on point clouds. In addition, we establish the first SLAM-related dataset of non-Lambertian scenes with complete ground truth of illumination and material. Extensive experiments demonstrated that our PBA method outperforms existing approaches in accuracy.

The ability to accomplish manipulation and locomotion tasks in the presence of significant time-varying external loads is a remarkable skill of humans that has yet to be replicated convincingly by humanoid robots. Such an ability will be a key requirement in the environments we envision deploying our robots: dull, dirty, and dangerous. External loads constitute a large model bias, which is typically unaccounted for. In this work, we enable our humanoid robot to engage in loco-manipulation tasks in the presence of significant model bias due to external loads. We propose an online estimation and control framework involving the combination of a physically consistent extended Kalman filter for inertial parameter estimation coupled to a whole-body controller. We showcase our results both in simulation and in hardware, where weights are mounted on Nadia’s wrist links as a proxy for engaging in tasks where large external loads are applied to the robot.

Motion prediction is critical for autonomous off-road driving, however, it presents significantly more challenges than on-road driving because of the complex interaction between the vehicle and the terrain. Traditional physics-based approaches encounter difficulties in accurately modeling dynamic systems and external disturbance. In contrast, data-driven neural networks require extensive datasets and struggle with explicitly capturing the fundamental physical laws, which can easily lead to poor generalization. By merging the advantages of both methods, neuro-symbolic approaches present a promising direction. These methods embed physical laws into neural models, potentially significantly improving generalization capabilities. However, no prior works were evaluated in real-world settings for off-road driving. To bridge this gap, we present PhysORD, a neural-symbolic approach integrating the conservation law, i.e., the Euler-Lagrange equation, into data-driven neural models for motion prediction in off-road driving. Our experiments showed that PhysORD can accurately predict vehicle motion and tolerate external disturbance by modeling uncertainties. It outperforms existing methods both in accuracy and efficiency and demonstrates data-efficient learning and generalization ability in long-term prediction.

This paper proposes a precise drone rephotography system for fixed-domain patrolling scenarios. The proposed system integrates computer-vision-based localization and fine-tuned pixel-level dense flow prediction to achieve consistent and precise rephotography images with viewpoints that closely align with those of target images. The proposed Keypoints Alignment Through Dense Flow Prediction (KADFP) model effectively handles challenges arising from lighting variations and background differences. Moreover, a novel flight procedure is implemented in the proposed system. This procedure involves using an Interleaved Drone Controller to alternate between translation and rotation adjustments to ensure smooth flight dynamics during rephotography. Experiments indicated that the proposed system provided considerably more precise rephotography results (error of 4.72 pixels indoors) than did an existing localization approach (error of 35.56 pixels).

We introduce PhotoBot, a framework for fully automated photo acquisition based on an interplay between high-level human language guidance and a robot photographer. We propose to communicate photography suggestions to the user via reference images that are selected from a curated gallery. We leverage a visual language model (VLM) and an object detector to characterize the reference images via textual descriptions and then use a large language model (LLM) to retrieve relevant reference images based on a user’s language query through text-based reasoning. To correspond the reference image and the observed scene, we exploit pretrained features from a vision transformer capable of capturing semantic similarity across marked appearance variations. Using these features, we compute suggested pose adjustments for an RGB-D camera by solving a perspective-n-point (PnP) problem. We demonstrate our approach using a manipulator equipped with a wrist camera. Our user studies show that photos taken by PhotoBot are often more aesthetically pleasing than those taken by users themselves, as measured by human feedback. We also show that PhotoBot can generalize to other reference sources such as paintings.

This study proposed a peristaltic motion-type inspection robot equipped with a "linear antagonistic mechanism using artificial muscles with an endoskeletal structure" to amplify propulsion and traction. We sought to develop an in-pipe inspection robot for long, narrow, and complex pipes requiring large propulsion, traction, and flexibility. In a previous study, we proposed a linear antagonistic mechanism allowing the inspection robot to generate both high propulsion and traction along with flexibility in narrow pipes. The proposed mechanism consisted of two extension actuators and a gripping actuator sandwiched between these extension actuators. The large extension force by the extension actuators is distributed to both propulsion and traction. However, owing to the piston-shaped configuration of the extension actuators, the generated force decreased in a manner dependent on the cross-sectional area within narrow pipelines. Therefore, the in-pipe inspection robot took time to move in long-distance, small-diameter pipes with multiple bends. This paper describes a "linear antagonistic mechanism using artificial muscles with an endoskeletal structure" that amplifies propulsion and traction by inserting a tension spring (skeleton) inside the contraction actuators (artificial muscles) and utilizing the action force generated by the actuator and transmitted by the tension spring. In this study, the developed robot with an endoskeleton exhibited maximum propulsion of 60.2 N, surpassing its non-endoskeleton counterpart by a factor of 1.61. Furthermore, the robot equipped with the endoskeleton passed through an elbow pipe 1.29 times faster than that without the endoskeleton, reducing the time from 741 to 576 s. The function value that compares the propulsion and traction considering the effects of the applied pressure and pipe diameter required for long-distance inspection was more than 1.13 times that of the previous study. In addition, the non-dimensionalized traction was ... Show More

Suction cups are popular for picking and transporting packages in warehouse applications. To maximize throughput, high transport speeds are desired. Many packages are deformable and may detach from the suction cups due to inertial loading if trajectories use excessive velocities. This paper introduces a novel methodology that analyzes package deformation through its curvature at the package-suction cup contact interface to generate a Factor-of-Safety (FOS) score for each waypoint in a given trajectory. By maintaining the FOS above a predetermined threshold, the trajectory planner is able to generate transport trajectories that are both safe and time-optimized. Experimental results show the method’s efficacy, demonstrating a 21.92% reduction in transport times compared to a conservative trajectory generation. Our FOS predictor identified trajectories that ensured safe package transport with 100% accuracy across all 627 real-world experiments.

In Model Predictive Control (MPC), discrepancies between the actual system and the predictive model can lead to substantial tracking errors and significantly degrade performance and reliability. While such discrepancies can be alleviated with more complex models, this often complicates controller design and implementation. By leveraging the fact that many trajectories of interest are periodic, we show that perfect tracking is possible when incorporating a simple observer that estimates and compensates for periodic disturbances. We present the design of the observer and the accompanying tracking MPC scheme, proving that their combination achieves zero tracking error asymptotically, regardless of the complexity of the unmodelled dynamics. We validate the effectiveness of our method, demonstrating asymptotically perfect tracking on a high-dimensional soft robot with nearly 10,000 states and a fivefold reduction in tracking errors compared to a baseline MPC on small-scale autonomous race car experiments.

Many robotic systems rely on visual sensing to accomplish simultaneously the tasks of state estimation, mapping, and path planning. One one hand, the usage of camera sensors represents a power-efficient and lightweight option for solving this problem. On the other hand, these tasks pose requirements on the quality of the visual input (e.g. number of tracked features for Visual Odometry) that are often in contrast to the optimal viewpoint planning for local mapping and obstacle avoidance. Dealing with this constraint is actively researched in the field of perception-aware planning. The approaches delivered by this field mostly concern Micro air vehicles (MAVs), but could be applied to a larger group of robotic systems. We propose a perception-aware trajectory planner for a class of robotic systems that can orient their cameras independently from their direction of travel. By using motion primitives, our planner does not require differentiable models for motion and perception objectives. We evaluate our method in simulation, showing increased capabilities in localization-aware motions around obstacles, and demonstrate its run-time capability on a real planetary rover. The code is released publicly under github.com/DLR-RM/palp.

This paper introduces a shared control framework designed specifically for agricultural mobile manipulators engaged in harvesting operations. The shared control strategy allows for achieving such operations by dynamically exchanging the control between the robotic system and a human operator depending on the uncertainty in the environment perception. For this purpose, the robot’s behavior is dynamically adapted to switch between two control modes with a different level of autonomy of the robot. The level of autonomy is encoded in two different admittance behaviors which are included in a first-order Hierarchical Quadratic Programming (HQP) control framework, that allows the robot to simultaneously address other control objectives at the same time. Experimental results with a dual-arm mobile robot, developed as part of the EU-funded CANOPIES project, demonstrate the effectiveness of the proposed method in real conditions.

Autonomous Vehicles (AVs) have recently attracted considerable attention due to their potential to significantly reduce road accidents and improve people’s lives. However, they rely solely on the data collected by their mounted sensors to make predictions, which can lead to inaccurate results if a sensor becomes occluded or damaged. This issue can be addressed by employing Vehicle-to-Vehicle communication, which allows a Connected Autonomous Vehicle (CAV) to interact with other CAVs within its field of view and exchange information about their surrounding objects. Existing research on cooperative perception has primarily focused on clear weather scenarios, with limited exploration into adverse weather conditions. This paper demonstrates the necessity of Vehicle-to-Vehicle communication by showcasing its benefits in maintaining high accuracy under adverse weather conditions. A collaborative perception system is introduced and its performance in foggy weather scenarios is assessed to further improve adverse weather perception. The pipeline of the network combines state-of-the-art methods for accurate object detection. Specifically, with PointPillars as the backbone, the Spatial-wise Adaptive Feature Fusion method is used to aggregate information from different vehicles. The model is trained on the large-scale dataset OPV2V and evaluated on modified data to simulate fog. The experiments show that cooperative perception can maintain high detection accuracy even in challenging weather conditions. Finally, a comparative analysis of LiDAR detectors for cooperative perception in bad weather conditions is presented.

Precise pedicle screw placement is crucial in spine surgery, where minor inaccuracies can result in significant complications. Despite introducing robot-assisted navigation systems to aid surgeons, accommodating the spine’s nonrigid movements (due to patient movement or interactions with the surgeon) often necessitates repeated intraoperative imaging, leading to increased radiation exposure. To address this challenge, we propose a novel method that utilizes the functional map (FM) framework to transfer drilling trajectories from preoperative CT scans to partially observed and noisy spine models. Specifically, the FM correspondences enhance the registration quality of pre-operative and perioperative 3D spine model data, even in the presence of non-rigid deformations. Through comprehensive simulations, we assess the method’s effectiveness across various cases of complex deformations using a spine model consisting of five lumbar vertebrae obtained through CT scans. Validation involves evaluating registration errors in translation and rotation and verifying the clinical validity of transferred drilling trajectories. The results demonstrate the method’s efficiency in transferring drilling trajectories onto noisy, partially observed, and deformed spine models.

The efficient navigation of autonomous vehicles across rugged and unstructured terrains remains a significant challenge. Most existing research in this area emphasizes the need for complex mappings or intricate multi-step methodologies. However, these traditional approaches often struggle to adapt to dynamic changes in environmental conditions. In this paper, we introduce PathFormer, an end-to-end framework designed specifically to address these challenges. PathFormer utilizes transformers to decode free-space semantics and configurations directly from camera images, enabling efficient path planning without the reliance on detailed, pre-existing maps. The performance of PathFormer was rigorously evaluated across diverse datasets, where it demonstrated superior capabilities, outperforming other state-of-the-art methods by 3.68% in precisely segmenting free-space regions and showing a 13.65% improvement in correctly predicting traversable paths.

The study of non-line-of-sight (NLOS) imaging is growing due to its many potential applications, including rescue operations and pedestrian detection by self-driving cars. However, implementing NLOS imaging on a moving camera remains an open area of research. Existing NLOS imaging methods rely on time-resolved detectors and laser configurations that require precise optical alignment, making it difficult to deploy them in dynamic environments. This work proposes a data-driven approach to NLOS imaging, PathFinder, that can be used with a standard RGB camera mounted on a small, power-constrained mobile robot, such as an aerial drone. Our experimental pipeline is designed to accurately estimate the 2D trajectory of a person who moves in a Manhattan-world environment while remaining hidden from the camera’s field-of-view. We introduce a novel approach to process a sequence of dynamic successive frames in a line-of-sight (LOS) video using an attention-based neural network that performs inference in real-time. The method also includes a preprocessing selection metric that analyzes images from a moving camera which contain multiple vertical planar surfaces, such as walls and building facades, and extracts planes that return maximum NLOS information. We validate the approach on in-the-wild scenes using a drone for video capture, thus demonstrating low-cost NLOS imaging in dynamic capture environments. The real-world dataset that we collected and used to train the network can be found at https://srchandr.github.io/DynamicNLOS/.

We present a sample-based motion planning algorithm specialised to a class of underactuated systems using path parameterisation. The structure this class presents under a path parameterisation enables the trivial computation of dynamic feasibility along a path. Using this, a specialised state-based steering mechanism within an RRT motion planning algorithm is developed, enabling the generation of both geometric paths and their time parameterisations without introducing excessive computational overhead. We find with two systems that our algorithm computes feasible trajectories with higher rates of success and lower mean computation times compared to existing approaches.

Path re-planning and repairing are key topics for robust planning and navigation in open dynamic environments, finding applications in various domains such as fleet control of Unmanned Ground Vehicles (UGVs) in warehouses. The use of UGVs in open and dynamic environments requires flexible cooperation between human operators and the UGV fleet within a shared environment. In this paper, we propose a local strategy to re-plan the path of robots encountering unexpected and dynamic obstacles. Specifically, starting from a given Multi-Agent path, we model the re-planning problem as a Markov Decision Process (MDP) considering a stochastic obstacle lifespan, and we propose two local approaches based on Monte-Carlo Tree Search to re-plan the path of the robots that encounter obstacles. We compare these approaches with traditional Multi-Agent Path Finding (MAPF) algorithms to obtain new collision-free paths when an obstacle is detected. The evaluation is performed in simulation using benchmarking instances of warehouses and experimentally in a research facility with a scaled-down Industry 4.0 production line.

Most industrial parts are designed from parametric shapes with the properties of diversity and uncertainty. We propose a 6DoF pose estimation network, ParametricNet++, based on pointwise regression and sparse keypoint recovery, which is extended from ParametricNet to include optimizations of keypoint selection and prediction. Keypoint selection optimization selects geometrically unique keypoints and keypoint groups to reduce the difficulty of scene keypoint prediction and template keypoint recovery. Keypoint prediction optimization predicts keypoints from rough to precise, which improves the accuracy of scene keypoint prediction and template keypoint recovery. Compared with other state-of-the-art methods, the average of APs of ParametricNet++ is improved by over 15% on the public Siléane dataset, and the average of mAPs is improved by 12% and 14% on L-dataset and G-dataset from Parametric dataset, respectively. In particular, ParametricNet++ outperforms our original ParametricNet by 5% for both learning and generalization ability evaluation on the Parametric dataset. The experimental results demonstrate that ParametricNet++ lays a solid foundation for robot grasping in industrial scenarios.

This paper describes a novel parametric optimization procedure for three flexure cross hinges (TFCH) integrated into multi-link leg mechanisms with closed-loop kinematics. Despite advantages such as compliance, no need for joint lubrication, light weight and cost-efficiency, such shaftless mechanisms have not been widely used, especially in the field of dynamic locomotion, also because their design is challenging and barely studied. Using a morphological computation approach, we have optimized the TFCH geometry to achieve the desired joint stiffness using frequency analysis, ensuring safe and stable hopping under external perturbations. We combined rigid body dynamics with lumped stiffness model and finite element modeling using the SPACAR toolbox to simulate various designs within our optimization pipeline. To illustrate the efficiency of the resulting designs, we built a prototype and conducted a series of full-scale experiments with ramp jumps whose trajectories were recorded by a motion capture system. The experiments showed that TFCH can be effectively integrated into leg mechanisms, providing benefits such as impact robustness, energy recuperation, and the ability to work in extreme conditions.