IROS 2022 Papers

Manipulation of deformable objects is a challenging task for a robot. It would be problematic to use a single sensory input to track the behaviour of such objects: vision can be subjected to occlusions, whereas tactile inputs cannot capture the global information that is useful for the task. In this paper, we study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects, for the first time. We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted using visual-tactile fusion, compared to using a single sensing modality. To this end, we developed a benchmark in simulation for manipulating the deformable linear objects using multimodal sensing inputs. The policy of the agent uses distilled information, e.g., the pose of the object in both visual and tactile perspectives, instead of the raw sensing signals, so that it can be directly transferred to real environments. In this way, we disentangle the perception system and the learned control policy. Our extensive experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases, compared to 77% when only one of the signals is given. Our results can provide valuable insights for the future design of tactile sensors and for deformable objects manipulation. Code and videos can be found at: https://github.com/lpecyna/SoftSlidingGym.

System modeling and parameter identification of micro aerial vehicles (MAV) are crucial for robust autonomy, especially under highly dynamic motions. Visual-inertial-aided online parameter identification has recently seen research attention due to the demanding of adaptation to platform configuration changes with minimal onboard sensor requirements. To this end, we design an online MAV system identification algorithm to tightly fuse visual, inertial and MAV aerodynamic information within a lightweight multi-state constraint Kalman filter (MSCKF) framework. In particular, while one could blindly fuse the MAV dynamic-induced relative motion constraints in EKF, we numerically show that due to the (quadrotor) MAV system modeling inaccuracy, they often become overconfident and negatively impact the state estimates. As such, we leverage the Schmidt-Kalman filter (SKF) for MAV system parameter identification to prevent corruption of state estimates. Through extensive simulations and real-world experiments, we validate the proposed SKF-based scheme and demonstrate its ability to perform robust system identification even in the presence of an inconsistent MAV dynamic model under different motions.

Robotic applications are continuously striving towards higher levels of autonomy. To achieve that goal, a highly robust and accurate state estimation is indispensable. Combining visual and inertial sensor modalities has proven to yield accurate and locally consistent results in short-term applications. Unfortunately, visual-inertial state estimators suffer from the accumulation of drift for long-term trajectories. To eliminate this drift, global measurements can be fused into the state estimation pipeline. The most known and widely available source of global measurements is the Global Positioning System (GPS). In this paper, we propose a novel approach that fully combines stereo Visual-Inertial Simultaneous Localisation and Mapping (SLAM), including visual loop closures, with the fusion of global sensor modalities in a tightly-coupled and optimisation-based framework. Incorporating measurement uncertainties, we provide a robust criterion to solve the global reference frame initialisation problem. Furthermore, we propose a loop-closure-like optimisation scheme to compensate drift accumulated during outages in receiving GPS signals. Experimental validation on datasets and in a real-world experiment demonstrates the robustness of our approach to GPS dropouts as well as its capability to estimate highly accurate and globally consistent trajectories compared to existing state-of-the-art methods.

In this paper, we present a tightly-coupled visual-inertial object-level multi-instance dynamic SLAM system. Even in extremely dynamic scenes, it can robustly optimise for the camera pose, velocity, IMU biases and build a dense 3D reconstruction object-level map of the environment. Our system can robustly track and reconstruct the geometries of arbitrary objects, their semantics and motion by incrementally fusing associated colour, depth, semantic, and foreground object probabilities into each object model thanks to its robust sensor and object tracking. In addition, when an object is lost or moved outside the camera field of view, our system can reliably recover its pose upon re-observation. We demonstrate the robustness and accuracy of our method by quantitatively and qualitatively testing it in real-world data sequences.

An increasing number of nonspecialist robotic users demand easy-to-use machines. In the context of visual servoing, the removal of explicit image processing is becoming a trend, allowing an easy application of this technique. This work presents a deep learning approach for solving the perception problem within the visual servoing scheme. An artificial neural network is trained using the supervision coming from the knowledge of the controller and the visual features motion model. In this way, it is possible to give a geometrical interpretation to the estimated visual features, which can be used in the analytical law of the visual servoing. The approach keeps perception and control decoupled, conferring flexibility and interpretability on the whole framework. Simulated and real experiments with a robotic manipulator validate our approach.

In this study, we present a visual servo control framework for fully automated nasopharyngeal swab robots. The proposed framework incorporates a deep learning-based nostril detection with a cascade approach to reliably identify the nostrils with high accuracy in real time. In addition, a partitioned visual servoing scheme that combines image-based visual servoing with axial control is formulated for accurately positioning the sampling swabs at the nostril with a multi-DOF robot arm. As the visual servoing is designed to minimize an error between the detected nostril and the swab, it can compensate for potential errors in real operation, such as positioning error by inaccurate camera-robot calibration and kinematic error by unavoidable swab deflection. The performance of the visual servo control was tested on a head phantom model for 30 unused swabs, and then compared with a method referring to only the 3D nostril target for control. Consequently, the swabs reached the nostril target with less than an average error of 1.2±0.5 mm and a maximum error of 2.0 mm via the visual servo control, while the operation without visual feedback yielded an average error of 10.6±2.3 mm and a maximum error of 16.2 mm. The partitioned visual servoing allows the swab to rapidly converge to the nostril target within 1.0 s without control instability. Finally, the swab placement at the nostril among the entire procedure of fully automated NP swab was successfully demonstrated on a human subject via the visual servo control.

Soft robotic grippers facilitate contact-rich manipulation, including robust grasping of varied objects. Yet the beneficial compliance of a soft gripper also results in significant deformation that can make precision manipulation challenging. We present visual pressure estimation & control (VPEC), a method that infers pressure applied by a soft gripper using an RGB image from an external camera. We provide results for visual pressure inference when a pneumatic gripper and a tendon-actuated gripper make contact with a flat surface. We also show that VPEC enables precision manipulation via closed-loop control of inferred pressure images. In our evaluation, a mobile manipulator (Stretch RE1 from Hello Robot) uses visual servoing to make contact at a desired pressure; follow a spatial pressure trajectory; and grasp small low-profile objects, including a microSD card, a penny, and a pill. Overall, our results show that visual estimates of applied pressure can enable a soft gripper to perform precision manipulation.

The accuracy and robustness of visual odometry (VO) is significantly affected by the high dynamic range (HDR) environments, because traditional cameras have a limited dynamic range and inevitably miss information in both overexposed and underexposed areas. To overcome the above challenge, we use an spatially varying exposure (SVE) camera, which captures four images with different exposure levels simultaneously. Then, we propose a VO pipeline that leverages the advantages of the SVE camera. Specifically, we extract ORB features from four images in parallel firstly instead of fusing four images, then perform merging and filtering to provide more robust features. We demonstrate that the proposed system outperforms comparable state-of-the-art methods in terms of robustness and accuracy. The real-time performance of the proposed system is also guaranteed due to the elaborate design of the parallel algorithm.

Exploring the relationship among objects and giving the correct operation sequence is vital for robotic manipulation. However, most previous algorithms only model the relationship between pairs of objects independently, ignoring the interaction effect between them, which may generate redundant or missing relations in complex scenes, such as multi-object stacking and partial occlusion. To solve this problem, a Gated Graph Neural Network (GGNN) is designed for visual manipulation relationship detection, which can help robots detect targets in complex scenes and obtain the appropriate grasping order. Firstly, the robot extracts feature from the input image and estimate object categories. Then GGNN is used to effectively capture the dependencies between objects in the whole scene, update the relevant features, and output the grasping sequence. In addition, by embedding positional encoding into pair object features, accurate context information is obtained to reduce the adverse effects of complex scenes. Finally, the constructed algorithm is applied to the physical robot for grasping. Experiment results on the Visual Ma-nipulation Relationship Dataset (VMRD) and the large-scale relational grasp dataset named REGRAD show that our method significantly improves the accuracy of relationship detection in complex scenes, and can be well generalized in the real world.

Lower limb exoskeletons offer support for patients suffering from mobility disorders due to injury, stroke, etc. But these devices are not used in day-to-day life and environments due to their limited human-computer interface to perceive and handle different terrains and tasks. In this paper, we introduce a simple vision-based environment perception pipeline for lower- limb exoskeletons for obstacle crossing tasks. The proposed pipeline consists of three stages, namely, ground plane and obstacle detection, estimating obstacle location and dimensions, and obstacle tracking. To reduce noisy artifacts and reliably detect obstacles, we propose a similarity metric based on color, gradient orientation, and 2D surface normal. Depth map of the detected obstacle region is utilized for estimating the obstacle location and dimensions. Also, we consider two obstacle tracking modes for obstacle crossing, visual tracking using a RGB-D camera and positional tracking using a SLAM camera. The proposed vision-based perception pipeline is integrated with an exoskeleton, where we propose a control scheme that can vary step length adaptively to successfully cross detected obstacles. We conduct offline and online experiments to validate the proposed perception pipeline and provide insights on the same. Our experiments show that the proposed pipeline allows exoskeletons to understand their environment and successfully cross obstacles.

Visual navigation for insect-scale robots is very challenging because in such a small scale, the size, weight, and power (SWaP) constraints do not appear to permit visual navigation techniques such as SLAM (Simultaneous Localization and Mapping) because they are likely to be too power-hungry. We propose to use a biology-inspired approach, which we term the bilinear optic flow approximation, that is more computationally efficient. We build on previous work that has shown that the bilinear approximation can be used for visual servoing. Here, we show that a bilinear approximator can be learned that is able to stabilize the heading of a robot while performing continuous forward motion in a corridor-shaped environment. This is a necessary capability for confined-space navigation that insect-sized robots are likely to perform. In this work, we describe the underlining methodology of the method and built a 2D visual simulation environment and omnidirectional camera model to validate our results.

Recent Earth observation satellites are now equipped with new instrument that allows image feedback in real-time. Problematic such as ground target tracking, moving or not, can now be addressed by precisely controlling the satellite attitude. In this paper, we propose to consider this problem using a visual servoing approach. While focusing on the target, the control scheme has also to take into account the satellite motion induced by its orbit, Earth rotational velocities, potential target own motion, but also rotational velocities and accelerations constraints of the system. We show the efficiency of our system using both simulation (considering real Earth image) and experiments on a robot that replicates actual high resolution satellite constraints.

In this paper, we address the vision-based detection and tracking problems of multiple aerial vehicles using a single camera and Inertial Measurement Unit (IMU) as well as the corresponding perception consensus problem (i.e., uniqueness and identical IDs across all observing agents). We design several vision-based decentralized Bayesian multi-tracking filtering strategies to resolve the association between the incoming unsorted measurements obtained by a visual detector algorithm and the tracked agents. We compare their accuracy in different operating conditions as well as their scalability according to the number of agents in the team. This analysis provides useful insights about the most appropriate design choice for the given task. We further show that the proposed perception and inference pipeline which includes a Deep Neural Network (DNN) as visual target detector is lightweight and capable of concurrently running control and planning with Size, Weight, and Power (SWaP) constrained robots on-board. Experimental results show the effective tracking of multiple drones in various challenging scenarios such as heavy occlusions.

Online path planning for multiple unmanned aerial vehicle (multi-UAV) systems is considered a challenging task. It needs to ensure collision-free path planning in real-time, especially when the multi-UAV systems can become very crowded on certain occasions. In this paper, we presented a vision-based decentralized collision-avoidance policy learning method for multi-UAV systems. The policy takes depth images and inertial measurements as sensory inputs and outputs UAV's steering commands, and it is trained together with the latent representation of depth images using a policy gradient-based reinforcement learning algorithm and autoencoder in the multi-UAV three-dimensional workspaces. Each UAV follows the same trained policy and acts independently to reach the goal without colliding or communicating with other UAVs. We validate our method in various simulated scenarios. The experimental results show that our learned policy can guarantee fully autonomous collision-free navigation for multi-UAV in three-dimensional workspaces, and its navigation performance will not be greatly affected by the increase in the number of UAVs.

Developing robust vision-guided controllers for quadrupedal robots in complex environments with various obstacles, dynamical surroundings and uneven terrains is very challenging. While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the vision-guided RL policy in the real world. Our key insight is that the asynchronous multi-modal observations, caused by different latencies in different components of the real robot, create a large sim2real gap for a RL policy. In this paper, we propose Multi-Modal Delay Randomization (MMDR) to address this issue when training in simulation. Specifically, we randomize the selections for both the proprioceptive states and the visual observations in time during training, aiming to simulate the asynchronous inputs when deploying to the real robot. With this technique, we are able to train a RL policy for end-to-end locomotion control in simulation, which can be directly deployed on the real A1 quadruped robot running in the wild. We evaluate our method in different outdoor environments with complex terrain and obstacles. We show that the robot can smoothly maneuver at a high speed while avoiding the obstacles, achieving significant improvement over the baselines. Our project page with videos is at https://mehooz.github.io/mmdr-wild/.

Human safety has always been the main priority when working near an industrial robot. With the rise of Human-Robot Collaborative environments, physical barriers to avoiding collisions have been disappearing, increasing the risk of accidents and the need for solutions that ensure a safe Human-Robot Collaboration. This paper proposes a safety system that implements Speed and Separation Monitoring (SSM) type of operation. For this, safety zones are defined in the robot's workspace following current standards for industrial collaborative robots. A deep learning-based computer vision system detects, tracks, and estimates the 3D position of operators close to the robot. The robot control system receives the operator's 3D position and generates 3D representations of them in a simulation environment. Depending on the zone where the closest operator was detected, the robot stops or changes its operating speed. Three different operation modes in which the human and robot interact are presented. Results show that the vision-based system can correctly detect and classify in which safety zone an operator is located and that the different proposed operation modes ensure that the robot's reaction and stop time are within the required time limits to guarantee safety.

Legged robots, specifically quadruped robots, have good locomotion performance in complex and rugged terrain and are becoming widely used in field exploration and rescue missions. To achieve full autonomy in such scenarios, robots need not only accurate localization but also an accurate understanding of the surrounding terrain, which will be used for robots path planning and foothold planning. However, due to the kinetic characteristic and limitation of size, quadruped robots have the disadvantages of high-frequency jitter and limited field of sensors, which lead to some challenges in environmental perception. In this paper, we propose a vision-assisted rugged terrain environment reconstruction and localization method for quadruped robots. We use a depth camera to assist in the generation of high-precision localization and terrain reconstruction results, which can help achieve the autonomous mobility of quadruped robots in this environment. We test our method on a quadruped robot platform. Our experimental results show less error and lower drift in different stairs terrain types than the commonly used lidar-based localization method.

This paper introduces mathematical models of touch sensors for mobile robots based on visibility. Serving a purpose similar to the pinhole camera model for computer vision, the introduced models are expected to provide a useful, idealized characterization of task-relevant information that can be inferred from their outputs or observations. Possible tasks include navigation, localization and mapping when a mobile robot is deployed in an unknown environment. These models allow direct comparisons to be made between traditional depth sensors, highlighting cases in which touch sensing may be interchangeable with time of flight or vision sensors, and char-acterizing unique advantages provided by touch sensing. The models include contact detection, compression, load bearing, and deflection. The results could serve as a basic building block for innovative touch sensor designs for mobile robot sensor fusion systems.

Spina Bifida (SB) is a birth defect developed during the early stage of pregnancy in which there is incomplete closing of the spine around the spinal cord. The growing interest in fetoscopic Spina-Bifida repair, which is performed in fetuses who are still in the pregnant uterus, prompts the need for appropriate training. The learning curve for such procedures is steep and requires excellent procedural skills. Computer-based virtual reality (VR) simulation systems offer a safe, cost-effective, and configurable training environment free from ethical and patient safety issues. However, to the best of our knowledge, there are currently no commercial or experimental VR training simulation systems available for fetoscopic SB-repair procedures. In this paper, we propose a novel VR simulator for core manual skills training for SB-repair. An initial simulation realism validation study was carried out by obtaining subjective feedback (face and content validity) from 14 clinicians. The overall simulation realism was on average marked 4.07 on a 5-point Likert scale (1 - ‘very unrealistic’, 5 - ‘very realistic’). Its usefulness as a training tool for SB-repair as well as in learning fundamental laparoscopic skills was marked 4.63 and 4.80, respectively. These results indicate that VR simulation of fetoscopic procedures may contribute to surgical training without putting fetuses and their mothers at risk. It could also facilitate wider adaptation of fetoscopic procedures in place of much more invasive open fetal surgeries.