IROS 2023 Papers

The discovery of ocean worlds such as Enceladus, Titan, and Europa motivates the development of versatile autonomous mobility systems to enable the next era of space exploration where there is large uncertainty in terrain specifications due to a lack of prior surface reconnaissance missions. To explore these environments, we propose Exobiology Extant Life Surveyor (EELS): the first large-scale (4 lm long with 400 Nm peak torque) snake robot. The large scale is achieved by using a screw-based active skin mechanism to decouple motion and shape control. Autonomous mobility for such a system remains an open problem due to its many Degrees of Freedom (DoFs), complex terrain interactions, and intermittent localization failures in GPS-denied perceptually degraded environments due to the presence of fog, dust, featureless terrains, etc. We propose NEO, an autonomy architecture that scales to large DoFs to generate a versatile set of gaits to achieve mobility in unknown extreme environments. We also discuss the resilience capabilities of NEO that achieves closed-loop tracking performance by leveraging exteroception when available but can also operate with proprioception only, leading to resiliency against localization failures via graceful degradation in performance rather than unsafe behaviors. A quantitative hardware evaluation of exteroceptive leader-follower gait is performed indoors on synthetic ice along with qualitative results of field deployment of the proprioceptive leader-follower and sidewinding gaits in extreme environments of icy and sandy terrains with mobility-stressing elements such as trenches, undulations, and steep slopes (up to 35 degrees). We present a set of lessons learned from field deployments with a summary of challenges and open research problems. Video: www.rohanthakker.in/eels-neo-autonomy.html

Visual-inertial initialization can be classified into joint and disjoint approaches. Joint approaches tackle both the visual and the inertial parameters together by aligning observations from feature-bearing points based on IMU integration then use a closed-form solution with visual and acceleration observations to find initial velocity and gravity. In contrast, disjoint approaches independently solve the Structure from Motion (SFM) problem and determine inertial parameters from up-to-scale camera poses obtained from pure monocular SLAM. However, previous disjoint methods have limitations, like assuming negligible acceleration bias impact or accurate rotation estimation by pure monocular SLAM. To address these issues, we propose EDI, a novel approach for fast, accurate, and robust visual-inertial initialization. Our method incorporates an Error-state Kalman Filter (ESKF) to estimate gyroscope bias and correct rotation estimates from monocular SLAM, overcoming dependence on pure monocular SLAM for rotation estimation. To estimate the scale factor without prior information, we offer a closed-form solution for initial velocity, scale, gravity, and acceleration bias estimation. To address gravity and acceleration bias coupling, we introduce weights in the linear least-squares equations, ensuring acceleration bias observability and handling outliers. Extensive evaluation on the EuRoC dataset shows that our method achieves an average scale error of 5.8% in less than 3 seconds, outperforming other state-of-the-art disjoint visual-inertial initialization approaches, even in challenging environments and with artificial noise corruption.

Prism-based LiDARs are more compact and cheaper than the conventional mechanical multi-line spinning LiDARs, which have become increasingly popular in robotics, recently. However, there are several challenges for these new LiDAR sensors, including small field of view, severe motion distortions, and irregular patterns. These difficulties hinder them from being widely used in LiDAR odometry, practically. To tackle these problems, we present an effective continuous-time LiDAR odometry (ECTLO) method for the Risley-prism-based LiDARs with non-repetitive scanning patterns. A single range image covering historical points in LiDAR's small FoV is adopted for efficient map representation. To account for the noisy data from occlusions after map updating, a filter-based point-to-plane Gaussian Mixture Model is used for robust registration. Moreover, a LiDAR-only continuous-time motion model is employed to relieve the inevitable distortions. Extensive experiments have been conducted on various testbeds using the prism-based LiDARs with different scanning patterns, whose promising results demonstrate the efficacy of our proposed approach.

In this paper, we explore the dynamic grasping of moving objects through active pose tracking and reinforcement learning for hand-eye coordination systems. Most existing vision-based robotic grasping methods implicitly assume target objects are stationary or moving predictably. Performing grasping of unpredictably moving objects presents a unique set of challenges. For example, a pre-computed robust grasp can become unreachable or unstable as the target object moves, and motion planning must also be adaptive. In this work, we present a new approach, Eye-on-hAnd Reinforcement Learner (EARL), for enabling coupled Eye-on-Hand (EoH) robotic manipulation systems to perform real-time active pose tracking and dynamic grasping of novel objects without explicit motion prediction. EARL readily addresses many thorny issues in automated hand-eye coordination, including fast-tracking of 6D object pose from vision, learning control policy for a robotic arm to track a moving object while keeping the object in the camera's field of view, and performing dynamic grasping. We demonstrate the effectiveness of our approach in extensive experiments validated on multiple commercial robotic arms in both simulations and complex real-world tasks.

Exactly estimating and tracking the motion of surrounding dynamic objects is one of important tasks for the autonomy of a quadruped manipulator. However, with only an onboard RGB camera, it is still a challenging work for a quadruped manipulator to track the motion of a dynamic object moving with unknown and changing velocities. To address this problem, this manuscript proposes a novel image-based visual servoing (IBVS) approach consisting of three elements: a spherical projection model, a robust super-twisting observer, and a model predictive controller (MPC). The spherical projection model decouples the visual error of the dynamic target into linear and angular ones. Then, with the presence of the visual error, the robustness of the observer is exploited to estimate the unknown and changing velocities of the dynamic target without depth estimation. Finally, the estimated velocity is fed into the model predictive controller (MPC) to generate joint torques for the quadruped manipulator to track the motion of the dynamical target. The proposed approach is validated through hardware experiments and the experimental results illustrate the approach's effectiveness in improving the autonomy of the quadruped manipulator.

A coverage-based collaborative control strategy is developed in this paper for a multi-robot system with heterogeneous effective sensing ranges and safe operation zones to simultaneously estimate the states of and follow multiple targets governed by stochastic dynamics. Multiplicatively weighted Voronoi diagrams are exploited to define each robot's dominant region considering its limited sensing radius. The asymptotically stable system dynamics enabling the heterogeneous multi-robot system to (locally) optimally cover the time-varying probability density distributions that characterize the uncertainties of the targets' positions is derived, and minimally perturbed by control barrier functions designed to ensure that each robot moves within its safe operation zone in a collision-free manner. Specific target dynamics and measurement models are chosen in the experiment, whose results demonstrate the effectiveness of the proposed dynamic multi-target tracking approach.

Planning robot motions in complex environments is a fundamental research challenge and central to the autonomy, efficiency, and ultimately adoption of robots. While often the environment is assumed to be static, real-world settings, such as assembly lines, contain complex shaped, moving obstacles and changing target states. Therein robots must perform safe and efficient motions to achieve their tasks. In repetitive environments and multi-goal settings, reusable roadmaps can substantially reduce the overall query time. Most dynamic roadmap-based planners operate in state-time-space, which is computationally demanding. Interval-based methods store availabilities as node attributes and thereby circumvent the dimensionality increase. However, current approaches do not consider higher-order constraints, which can ultimately lead to collisions during execution. Furthermore, current approaches must replan when the goal changes. To this end, we propose a novel roadmap-based planner for systems with third-order differential constraints operating in dynamic environments with moving goals. We construct a roadmap with availabilities as node attributes. During the query phase, we use a Double-Integrator Minimum Time (DIMT) solver to recursively build feasible trajectories and accurately estimate arrival times. An exit node set in combination with a moving goal heuristic is used to efficiently find the fastest path through the roadmap to the moving goal. We evaluate our method with a simulated UAV operating in dynamic 2D environments and show that it also transfers to a 6-DoF manipulator. We show higher success rates than other state-of-the-art methods both in collision avoidance and reaching a moving goal.

Micro Aerial Vehicles (MAVs) often face a high risk of collision during autonomous flight, particularly in cluttered and unstructured environments. To mitigate the collision impact on sensitive onboard devices, resilient MAVs with mechanical protective cages and reinforced frames are commonly used. However, compliant and impact-resilient MAVs offer a promising alternative by reducing the potential damage caused by impacts. In this study, we present novel findings on the impact-resilient capabilities of MAVs equipped with passive springs in their compliant arms. We analyze the effect of compliance through dynamic modeling and demonstrate that the inclusion of passive springs enhances impact resilience. The impact resilience is extensively tested to stabilize the MAV following wall collisions under high-speed and large-angle conditions. Additionally, we provide comprehensive comparisons with rigid MAVs to better determine the tradeoffs in flight by embedding compliance onto the robot's frame.

Hybrid wheeled-legged quadrupeds have the potential to navigate challenging terrain with agility and speed and over long distances. However, obstacles can impede their progress by requiring the robots to either slow down to step over obstacles or modify their path to circumvent the obstacles. We propose a motion optimization framework for quadruped robots that incorporates non-steerable wheels and dynamic jumps, enabling them to perform hybrid wheeled-legged locomotion while overcoming obstacles without slowing down. Our approach involves a model predictive controller that uses a time-varying rigid body dynamics model of the robot, including legs and wheels, to track dynamic motions such as jumping. We also introduce a method for driving with minimal leg swings to reduce energy consumption by sparing the effort involved in lifting the wheels. Our method was tested successfully on the wheeled Mini Cheetah and the Unitree AlienGo robots. Further videos and results are available at https://www.ais.uni-bonn.de/∼hosseini/iros2023

Pericardiocentesis is an important surgical intervention to treat a medical condition called pericardial effusion, during which excessive fluid accumulates around the heart, potentially leading to life-threatening situation. It involves the insertion of a needle and catheter towards the heart into the pericardial space to drain the excessive fluid under ultrasound (US) guidance. The risky procedure requires surgeons to acquire sufficient training to ensure safe execution of the procedure. However, existing heart simulators lack dynamic features, do not offer realistic images under US imaging, and are not reusable. This work presents a dynamic heart simulator (DHS) with pericardial effusion to mimic the beating motion of the human heart and the realistic US imaging results. The beating heart motion is realized using a hydraulic actuation system connected to a double-layer balloon set. The clear and realistic US imaging results are obtained through a unique formula proposed for the chest tissue and the cardiac muscle. A characterization method was also developed to allow customization of important anatomical parameters in the DHS. The experimental results show that the DHS allowed highly realistic simulation of the beating heart, cardiac muscle, and pericardium under US imaging and has been demonstrated to enable successful US-guided pericardiocentesis.

Continuous enhancement in wearable technologies has led to several innovations in the healthcare, virtual reality, and robotics sectors. One form of wearable technology is wear-able sensors for kinematic measurements of human motion. However, measuring the kinematics of human movement is a challenging problem as wearable sensors need to conform to complex curvatures and deform without limiting the user's natural range of motion. In fine motor activities, such challenges are further exacerbated by the dense packing of several joints, coupled joint motions, and relatively small deformations. This work presents the design, fabrication, and characterization of a thin, breathable sensing glove capable of reconstructing fine motor kinematics. The fabric glove features capacitive sensors made from layers of conductive and dielectric fabrics, culminating in a non-bulky and discrete glove design. This study demonstrates that the glove can reconstruct the joint angles of the wearer with a root mean square error of 7.2 degrees, indicating promising applicability to dynamic pose reconstruction for wearable technology and robot teleoperation.

For over three decades, finger gaiting has remained largely a subject for theoretical inquiries. Successful execution of a sequence of finger gaits does not simply reduce to planning collision-free paths for the involved fingers. A major issue is how to move the gaiting finger without losing the finger contacts with the object, which will most likely undergo a motion as the contact forces need to be adapted during the gait. This paper focuses on a single finger gait executed on a tool by an anthropomorphic hand driven by an arm. To improve stability, the tool's tip is leveraged as a pivot on the supporting plane. The gait consists of three stages: removal, during which the contact force on the gaiting finger gradually decreases to zero; relocation, during which the finger follows a pre-planned path (relative to the moving object) to establish a new contact; and addition, during which the contact force on the relocated finger increases to some desired level. Hybrid position/impedance control employs reference finger forces that satisfy the friction cone constraints and are dynamically consistent with the object's motion, which in turn provides reference poses for the fingertips to maintain their contacts during the gait. Finger gaits have been demonstrated on a kitchen knife and a screwdriver with an Adept SCARA robot and a Shadow Dexterous Hand.

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.

Neural networks have already demonstrated attractive performance for solving motion planning problems, especially in static and predictable environments. However, efficient neural planners that can adapt to unpredictable dynamic environments, a highly demanded scenario in many practical applications, are still under-explored. To fill this research gap and enrich the existing motion planning approaches, in this pa-per, we propose DynGMP, a graph neural network (GNN)-based planner that provides high-performance planning solutions in unpredictable dynamic environments. By fully leveraging the prior exploration experience and minimizing the replanning cost incurred by environmental change, DynGMP achieves high planning performance and efficiency simultaneously. Empirical evaluations across different environments show that DynGMP can achieve close to 100% success rate with fast planning speed and short path cost. Compared with existing non-learning and learning-based counterparts, DynGMP shows very significant planning performance improvement, e.g., at least 2.7×, 2.2×, 2.4\times2.4\times and 2\times2\times faster planning speed with low path distance in four environments, respectively.

Closing the domain gap between training and deployment and incorporating multiple sensor modalities are two challenging yet critical topics for self-driving. Existing work only focuses on single one of the above topics, overlooking the simultaneous domain and modality shift which pervasively exists in real-world scenarios. A model trained with multi-sensor data collected in Europe may need to run in Asia with a subset of input sensors available. In this work, we propose DualCross, a cross-modality cross-domain adaptation framework to facilitate the learning of a more robust monocular bird's-eye-view (BEV) perception model, which transfers the point cloud knowledge from a LiDAR sensor in one domain during the training phase to the camera-only testing scenario in a different domain. This work results in the first open analysis of cross-domain cross-sensor perception and adaptation for monocular 3D tasks in the wild. We benchmark our approach on large-scale datasets under a wide range of domain shifts and show state-of-the-art results against various baselines. Our project webpage is at https://yunzeman.github.io/DualCross.

Satisfying safety constraints in reinforcement learning (RL) is an important issue, especially in real-world applications. Many studies have approached safe RL with the Lagrangian method, which introduces dual variables. However, applying a trained policy with the optimal dual variable to a new environment can be hazardous since the optimal value of the dual variable, which represents a level of safety, depends on the environmental setting. To this end, we propose a new framework, dual variable actor-critic (DVAC), that solves the safe RL problem by simultaneously training a single policy over different safety levels. We introduce a universal policy and universal Q-function, which have a dual variable as an argument. Then, we extend the soft actor-critic so that the universal policy is guaranteed to converge to the Pareto optimal policy sets. We evaluate the proposed method in simulation and real-world environments. The universal policy learned with the proposed method ranges from extremely safe to high performance according to the dual variables, and is nearly Pareto optimal compared to policies learned with the baseline methods. In addition, the agent is able to adapt to environments with unseen state distributions without additional training by identifying a suitable dual variable using the proposed method.

We present a novel optimization algorithm called DroNeRF for the autonomous positioning of monocular camera drones around an object for real-time 3D reconstruction using only a few images. Neural Radiance Fields, or NeRF, is a novel view synthesis technique used to generate new views of an object or scene from a set of input images. Using drones in conjunction with NeRF provides a unique and dynamic way to generate novel views of a scene, especially with limited scene capabilities of restricted movements. Our approach focuses on calculating optimized pose for individual drones while solely depending on the object geometry without using any external localization system. The unique camera positioning during the data capturing phase significantly impacts the quality of the 3D model. To evaluate the quality of our generated novel views, we compute different perceptual metrics like the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). Our work demonstrates the benefit of using an optimal placement of various drones with limited mobility to generate perceptually better results.

Driver distraction detection is an important function of driver monitoring systems and intelligent vehicles. Most previous research only focuses on the system development for daytime operations. In this paper, we propose a network model, V2IA-Net, which is able to use the daytime visible and nighttime infrared images for the driver distraction detection task. With the visible-infrared image translation, driver action recognition and head pose detection, the driver distraction behavior can be analyzed in real-time performance. To provide realistic driving scenes for network training and testing, a visible-infrared image dataset, VID, is created. The proposed V2IA-Net is trained on the unpaired images, and capable of common feature extraction for visible-infrared image conversion. In the experiments, our technique is compared with various driver distraction detection models. The results have demonstrated the effectiveness of the proposed method. Source code and datasets are available at https://github.com/kk2487/V2IA-Net.

In this paper, we present the dynamic model and control of DoubleBee, a novel hybrid aerial-ground vehicle consisting of two propellers mounted on tilting servo motors and two motor-driven wheels. DoubleBee exploits the high energy efficiency of a bicopter configuration in aerial mode, and enjoys the low power consumption of a two-wheel self-balancing robot on the ground. Furthermore, the propeller thrusts act as additional control inputs on the ground, enabling a novel decoupled control scheme where the attitude of the robot is controlled using thrusts and the translational motion is realized using wheels. A prototype of DoubleBee is constructed using commercially available components. The power efficiency and the control performance of the robot are verified through comprehensive experiments. Challenging tasks in indoor and outdoor environments demonstrate the capability of DoubleBee to traverse unstructured environments, fly over and move under barriers, and climb steep and rough terrains.