IROS 2024 Papers

Acquiring accurate depth information of transparent objects using off-the-shelf RGB-D cameras is a well-known challenge in Computer Vision and Robotics. Depth estimation/completion methods are typically employed and trained on datasets with quality depth labels acquired from either simulation, additional sensors or specialized data collection setups and known 3d models. However, acquiring reliable depth information for datasets at scale is not straightforward, limiting training scalability and generalization. Neural Radiance Fields (NeRFs) are learning-free approaches and have demonstrated wide success in novel view synthesis and shape recovery. However, heuristics and controlled environments (lights, backgrounds, etc) are often required to accurately capture specular surfaces. In this paper, we propose using Visual Foundation Models (VFMs) for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness. Our proposed method Segmentation-AIDed NeRF (SAID-NeRF) shows significant performance on depth completion datasets for transparent objects and robotic grasping.

Conventional approaches in robotics for perceiving the environment and signaling the robot’s state or intention for human-robot interaction involve the use of separate sensing and signaling systems. This can sometimes result in high costs and complex system installations. In this study, we propose an alternative approach, integrating both robot sensing and signaling mechanisms into a single utility module (called S-BUN, Soft Bifunctional Utility module for robot sensing aNd signaling). Soft material (Ecoflex 00-10 silicone) is used to form its bun-like structure with a central cavity filled with a NaCl solution. Inspired by honeycombs, the module’s surface incorporates a hexagonal pattern to enhance structural robustness. The design of S-BUN enables it to function as a sensor for both non-contact proximity and touch sensing, utilizing the NaCl solution. Additionally, it serves as a signaling mechanism to indicate the robot’s state through the active inflation and deflation dynamics of the module, simulating lifelike breathing patterns. Through our experiments, we present S-BUN’s capabilities in proximity and touch sensing, including its ability to discern various touch intensities. Finally, we demonstrate the application of S-BUN in the context of reactive behavioral control for a crawling robot and human-robot interaction scenarios.

The development of robotic grippers and hands for automation aims to emulate human dexterity without sacrificing the efficiency of industrial grippers. This study introduces Rotograb, a tendon-actuated robotic hand featuring a novel rotating thumb. The aim is to combine the dexterity of human hands with the efficiency of industrial grippers. The rotating thumb enlarges the workspace and allows in-hand manipulation. A novel joint design minimizes movement interference and simplifies kinematics, using a cutout for tendon routing. We integrate teleoperation, using a depth camera for real-time tracking and autonomous manipulation powered by reinforcement learning with proximal policy optimization. Experimental evaluations demonstrate that Rotograb’s rotating thumb greatly improves both operational versatility and workspace. It can handle various grasping and manipulation tasks with objects from the YCB dataset, with particularly good results when rotating objects within its grasp. Rotograb represents a notable step towards bridging the capability gap between human hands and industrial grippers. The tendon-routing and thumb-rotating mechanisms allow for a new level of control and dexterity. Integrating teleoperation and autonomous learning underscores Rotograb’s adaptability and sophistication, promising substantial advancements in both robotics research and practical applications.

Robust moisture detection is crucial for building maintenance and cost reduction. Current methods are often limited by the type of roofing material or are cumbersome and expensive. Ground Penetrating Radar (GPR) has shown promise in recent works in moisture detection due to its effectiveness across a broader range of materials, its compactness and lightweight nature, and its ability to image the subsurface. We introduce Roofus, an integrated robotic moisture detection system for flat rooftops, designed to overcome traditional method limitations. It combines a remotely controlled robot with deep learning GPR data processing and automatic map generation. Real-world data is collected and manually annotated for supervised learning. We investigate a novel approach to interpreting GPR data via deep learning using Transformer-based classifiers. LiDAR inertial odometry is employed to integrate multiple individual GPR scans into a holistic moisture map over the rooftop. When evaluated against existing methods such as infrared thermal imaging, electrical capacitance surveys, and nuclear moisture gauges, our system shows promising viability for industry application.

This work focuses on localizing a single target robot with multi-robot formations in 2D space. The cooperative robots employ inter-robot range measurements to assess the target position. In the presence of noisy measurements, the choice of formation geometries significantly impacts the accuracy of the target robot’s pose estimation. While an infinite number of geometries exists to optimize localization accuracy, the current practice is to choose the final formation geometry based on convenience criteria such as simplicity or proximity to the initial position of the robots. The former leads to the selection of regular polygon-shaped formations, while the latter results in behaviour-based formations. Different from existing works, we conduct a complete robustness study of formation geometries in the presence of deviations from the desired formation and range measurement errors. In 2D scenarios, we establish necessary and sufficient conditions for formation geometries to be robust against robot positioning errors. This result substantiates the extensive use of regular polygon formations. However, our analysis reveals the lack of robustness of the commonly used square formation geometry, which stands as an exception. Simulation results illustrate the advantages of these robust geometries in enhancing target localization accuracy.

This paper presents a stochastic/robust nonlinear model predictive control (NMPC) to enhance the robustness of model-based legged locomotion against contact uncertainties. We integrate the contact uncertainties into the covariance propagation of stochastic/robust NMPC framework by lever-aging the guard saltation matrix and an extended Kalman filter-like covariance update. We achieve fast stochastic/robust NMPC computation by utilizing the zero-order algorithm with additional improvements in computational efficiency concerning the feedback gains. We conducted numerical experiments and demonstrate that the proposed method can accurately forecast future state covariance and generate trajectories that satisfies constraints even in the presence of the contact uncertainties. Hardware experiments on the perceptive locomotion of a wheeled-legged robot were also carried out, validating the feasibility of the proposed method in a real-world system with limited on-board computation.

In robotics, while rigid joints are common due to their accuracy and fast response ability, elastic joints are wellknown for their safety when interacting with the environment. To harmonise the advantages of these joint types, robots with mixed rigid-elastic joints can be considered. In this paper, a robust-adaptive two-loop control algorithm is proposed to control this type of hybrid robots when there are uncertainties in system parameters. In the outer loop, a robust control algorithm is proposed to deal with the uncertainties in the parameters of the joint dynamics, together with an adaptive controller for the rigid joints. In the inner loop, another robust control algorithm is proposed to handle the uncertainties in system parameters of the elastic joint’s motor contribution, and a similar adaptive control algorithm is presented to manipulate the elastic joints’ motors. The stability of the system is assured by Lyapunov’s stability theory. Finally, simulations are conducted to verify the proposed control algorithm.

Task and motion planning (TAMP) can enhance intelligent multi-robot coordination. TAMP becomes signifi-cantly more complicated in obstacle-cluttered environments and in the presence of robot dynamic uncertainties. We propose a control framework that solves the motion-planning problem for multi-robot teams with uncertain dynamics, addressing a key component of the TAMP pipeline. The principal part of the proposed algorithm constitutes a decentralized feedback control policy for tracking of reference paths taken by the robots while avoiding collision and adapting in real time to the underlying dynamic uncertainties. The proposed framework further leverages sampling-based motion planners to free the robots from local-minimum configurations. Extensive experimental results in complex, realistic environments illustrate the superior efficiency of the proposed approach, in terms of planning time and number of encountered local minima, with respect to state-of-the-art baseline methods.

We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of- the-art methods by a large margin. The project webpage is available at: https://github.com/VladPyatov/ihls

In this work, we address the challenges of robust precision landing maneuvers for a quadrotor on both stationary and moving ground targets in the presence of disturbances that can cause the quadrotor to deviate from its desired trajectory, leading to maneuver failure. To overcome this, we propose a novel online adaptive trajectory planning approach based on the online temporal scaling adaptation of dynamic movement primitives (DMPs). This adaptation enables the desired trajectory to be dynamically adjusted in response to tracking errors and the goal’s state. Consequently, our proposed approach enhances accuracy, precision, and safety during landing maneuvers. The effectiveness of the approach is evaluated through comprehensive experiments conducted in both physical simulations and real-world environments, covering various disturbance scenarios.

In the pursuit of object retrieval using an aerial manipulator, developing robust visual servoing techniques in the presence of projection and motion model uncertainties is paramount. This paper proposes a novel approach to conducting image-space planning within the controllable-space of the aerial manipulator. Our new strategy resolves the inherent challenge of adhering to a piecewise linear camera trajectory which is infeasible for an aerial manipulator due to the platform’s underactuation and presence of secondary tasks for visual servoing. Through this approach, we introduce center of gravity alignment and camera orientation potential fields without relying on specific degrees of freedom from the arm. Moreover, we introduce a new approach that utilizes an image-resolution scaling technique involving an adaptive virtual camera focal length, leading to a numerically well-conditioned image Jacobian. Our proposed framework maintains robustness to the uncertainty in the intrinsic parameters of the camera. We substantiate the efficacy of our methodology through experiments conducted in a realistic physics-based simulation environment.

Recently, Bird’s Eye View (BEV) detection methodologies that utilize surround-view cameras have seen significant advancements in autonomous driving systems. Traditional methods, however, are constrained by their reliance on specific camera parameters, which poses challenges in generalizing across different vehicle-mounted cameras with varying poses and under adverse conditions. To address these challenges, we propose a robust BEV representation network that integrates Dual-Space Positional Encoding (DSPE) and image perception. This network is designed to enhance resilience to calibration errors and pose fluctuations, resulting in reliable detection performance on the Nuscenes dataset, even with imprecise extrinsic inputs. Our approach demonstrates competitive accuracy when compared to other methods that do not rely on temporal data, highlighting the effectiveness of our DSPE strategy in improving the robustness and accuracy of BEV detection in dynamic and challenging environments.

We study how to generalize the visuomotor policy of a mobile manipulator from the perspective of visual observations. The mobile manipulator is prone to occlusion owing to its own body when only a single viewpoint is employed and a significant domain shift when deployed in diverse situations. However, to the best of the authors’ knowledge, no study has been able to solve occlusion and domain shift simultaneously and propose a robust policy. In this paper, we propose a robust imitation learning method for mobile manipulators that focuses on task-related viewpoints and their spatial regions when observing multiple viewpoints. The multiple viewpoint policy includes attention mechanism, which is learned with an augmented dataset, and brings optimal viewpoints and robust visual embedding against occlusion and domain shift. Comparison of our results for different tasks and environments with those of previous studies revealed that our proposed method improves the success rate by up to 29.3 points. We also conduct ablation studies using our proposed method. Learning task-related viewpoints from the multiple viewpoints dataset increases robustness to occlusion than using a uniquely defined viewpoint. Focusing on task-related regions contributes to up to a 33.3-point improvement in the success rate against domain shift.

The conventional quadrotor is incapable of controlling position and orientation independently. To mitigate this deficiency, we use a tilt-augmented quadrotor for greater mobility in a constrained environment. When the rotors tilt in a tilt-augmented quadrotor, it leads to changes in moment- of-inertia. This changes in the moment-of-inertia and external disturbances will introduce uncertainty terms into the model. In this paper, we design an adaptive super twisting sliding mode observer which guarantees finite time estimation of uncertain terms with unknown maximum bound. With the help of this observer, a backstepping controller using SO(3) is developed to establish exponential convergence to the desired trajectory. The exponential convergence of the backstepping controller and finite time convergence of the observer are shown using the Lyapunov approach. Hardware experiments are performed to compare the performance of both the existing controller and our proposed controller and corresponding videos are at https://www.youtube.com/watch?v=brTd5UYvciM.

We study the design of robust and agile controllers for hybrid underactuated systems. Our approach breaks down the task of creating a stabilizing controller into: 1) learning a mapping that is invariant under optimal control, and 2) driving the actuated coordinates to the output of that mapping. This approach, termed Zero Dynamics Policies, exploits the structure of underactuation by restricting the inputs of the target mapping to the subset of degrees of freedom that cannot be directly actuated, thereby achieving significant dimension reduction. Furthermore, we retain the stability and constraint satisfaction of optimal control while reducing the online computational overhead. We prove that controllers of this type stabilize hybrid underactuated systems and experimentally validate our approach on the 3D hopping platform, ARCHER. Over the course of 3000 hops the proposed framework demonstrates robust agility, maintaining stable hopping while rejecting disturbances on rough terrain.

In this work, we present a simplified quasi-static model of a two-point contact gripper turning a circular valve to predict the reaction torques produced at the base of the valve as a function of the axis misalignment. Specifically, we learned that geometric features such as (i) the misalignment vector being tangent to the reaction torques, (ii) length of the misalignment vector being directly proportional to the magnitude of the reaction torques and (iii) small axial misalignments resulting in well defined ‘double-loops’ in the 2D reaction torques space are indicative of axis misalignment.

This study addresses contact-rich object insertion tasks under unstructured environments using a robot with a soft wrist, enabling safe contact interactions. For the unstructured environments, we assume that there are uncertainties in object grasp and hole pose and that the soft wrist pose cannot be directly measured. Recent methods employ learning approaches and force/torque sensors for contact localization; however, they require data collection in the real world. This study proposes a sim-to-real approach using a privileged training strategy. This method has two steps. 1) The teacher policy is trained to complete the task with sensor inputs and ground truth privileged information such as the peg pose, and then 2) the student encoder is trained with data produced from teacher policy rollouts to estimate the privileged information from sensor history. We performed sim-to-real experiments under grasp and hole pose uncertainties. This resulted in 100%, 95%, and 80% success rates for circular peg insertion with 0°, +5°, and -5° peg misalignments, respectively, and start positions randomly shifted ± 10 mm from a default position. Also, we tested the proposed method with a square peg that was never seen during training. Additional simulation evaluations revealed that using the privileged strategy improved success rates compared to training with only simulated sensor data. Our results demonstrate the advantage of using sim-to-real privileged training for soft robots, which has the potential to alleviate human engineering efforts for robotic assembly.

With the background of research on materials laboratory automation, this study aims to construct an automation system for measuring dielectric property, which is an electrical property of materials. The automation system is composed of the combination of a manipulation by a force-sensing robot and a control system for the measurement instrument. As challenges for the automation system, we worked on stabilizing a polymer film placement during insertion into the measurement instrument, implementing a communication control system between different platforms, and constructing a polymer film transfer environment. In the measurement experiment using the automation system, it was confirmed that the dielectric properties could be measured as well as that of a human.

Improving robot deployment is a central step towards speeding up robot-based automation in manufacturing. A main challenge in robot deployment is how to best place the robot within the workcell. To tackle this challenge, we combine two knowledge sources: robotic knowledge of the system and workcell context awareness of the user, and intersect them with an Augmented Reality interface. RobotGraffiti is a unique tool that empowers the user in robot deployment tasks. One simply takes a 3D scan of the workcell with their mobile device, adds contextual data points that otherwise would be difficult to infer from the system, and receives a robot base position that satisfies the automation task. The proposed approach is an alternative to expensive and time-consuming digital twins, with a fast and easy-to-use tool that focuses on selected workcell features needed to run the placement optimization algorithm. The main contributions of this paper are the novel user interface for robot base placement data collection and a study comparing the traditional offline simulation with our proposed method. We showcase the method with a robot base placement solution and obtain up to 16 times reduction in time.