IROS 2021 Papers

Dynamic objects in the environment, such as people and other agents, lead to challenges for existing simultaneous localization and mapping (SLAM) approaches. To deal with dynamic environments, computer vision researchers usually apply some learning-based object detectors to remove these dynamic objects. However, these object detectors are computationally too expensive for mobile robot on-board processing. In practical applications, these objects output noisy sounds that can be effectively detected by on-board sound source localization. The directional information of the sound source object can be efficiently obtained by direction of sound arrival (DoA) estimation, but the depth estimation is difficult. Therefore, in this paper, we propose a novel audio-visual fusion approach that fuses sound source direction into the RGB-D image and thus removes the effect of dynamic obstacles on the multi-robot SLAM system. Experimental results of multirobot SLAM in different dynamic environments show that the proposed method uses very small computational resources to obtain very stable self-localization results.

Event-based visual perception is becoming increasingly popular owing to interesting sensor characteristics enabling the handling of difficult conditions such as highly dynamic motion or challenging illumination. The mostly complementary nature of event cameras however still means that best results are achieved if the sensor is paired with a regular frame-based sensor. The present work aims at answering a simple question: Assuming that both cameras do not share a common optical center, is it possible to exploit the hybrid stereo setup's baseline to perform accurate stereo depth estimation? We present a learning based solution to this problem leveraging modern spatio-temporal input representations as well as a novel hybrid pyramid attention module. Results on real data demonstrate competitive performance against pure frame-based stereo alternatives as well as the ability to maintain the advantageous properties of event-based sensors.

Most of the state-of-the-art visual-inertial SLAM methods pay less attention to the scene structure of man-made environments. In this paper, based on the assumption of multiple local Manhattan worlds (MWs), we propose a Manhattan frame (MF) re-identification method to build relative rotation constraints between MF matching pairs and tightly couple these constraints into global bundle adjust module. Specifically, a coarse-to-fine vanishing point (VP) estimation method and pose guided MF temporal consistency verification method are firstly proposed to improve the accuracy and robustness of MF estimation. Then unreliable MF matching pairs are filtered out by a spatial temporal consistency check. Finally, the relative rotation constraints of the remaining MF matching pairs are combined into global bundle adjustment energy function for further optimization. We have validated our proposed method on both synthetic and real-world datasets. When comparing with the baseline method [1], the real-time absolute trajectory error (ATE) of our proposed method has decreased by 29.1%, 19.8% on TartanAir hospital and EuRoC datasets respectively. Our method also exceeds existing state-of-the-art algorithms on both synthetic and real-world datasets.

Most of the state-of-the-art visual inertial SLAM methods pay less attention to 2D-2D and 3D-2D matching with more reliable features in a long time span, which easily results in continuous estimation drift. In this paper, we propose an efficient drift-free visual-inertial SLAM method by a pose guided feature matching method to re-identify existing features from a spatial-temporal sensitive sub-global map. The re-identified features serve as augmented visual measurements to anchor the current frame and gradually decrease the accumulated error in the long run. When incorporating the measurements into the optimization module, it benefits to build a drift-free global map in the system. Extensive experiments show that our feature re-identification method is both effective and efficient. Specifically, when combining the feature re-identification with the state-of-the-art SLAM method [1], our method achieves 67.3% and 87.5% absolute trajectory error reduction with only a small additional computational cost on two public SLAM benchmark DBs: EuRoC and TUM-VI respectively.

Video prediction methods generally consume substantial computing resources in training and deployment, among which keypoint-based approaches show promising improvement in efficiency by simplifying dense image prediction to light keypoint prediction. However, keypoint locations are often modeled only as continuous coordinates, so noise from semantically insignificant deviations in videos easily disrupt learning stability, leading to inaccurate keypoint modeling. In this paper, we design a new grid keypoint learning framework, aiming at a robust and explainable intermediate keypoint representation for long-term efficient video prediction. We have two major technical contributions. First, we detect keypoints by jumping among candidate locations in our raised grid space and formulate a condensation loss to encourage meaningful keypoints with strong representative capability. Second, we introduce a 2D binary map to represent the detected grid keypoints and then suggest propagating keypoint locations with stochasticity by selecting entries in the discrete grid space, thus preserving the spatial structure of keypoints in the long-term horizon for better future frame generation. Extensive experiments verify that our method outperforms the state-of-the-art stochastic video prediction methods while saves more than 98% of computing resources. We also demonstrate our method on a robotic-assisted surgery dataset with promising results. Our code is available at https://github.com/xjgaocs/Grid-Keypoint-Learning.

For robotic grasping tasks with diverse target objects, some deep learning-based methods have achieved state-of-the-art results using direct visual input. In contrast, actor-critic deep reinforcement learning (RL) methods typically perform very poorly when applied to grasp diverse objects, especially when learning from raw images and sparse rewards. To render these RL techniques feasible for vision-based grasping tasks, we used state representation learning (SRL), in which we encode essential information for subsequent use in RL. However, typical representation learning procedures are unsuitable for extracting pertinent information for learning grasping skills owing to the high complexity of visual inputs for representation learning, in which a robot attempts to grasp a target object. We found that the proposed preprocessed input image is the key to capturing effectively a compact representation. This enables deep RL to learn robotic grasping skills from highly varied and diverse visual inputs. Further, we demonstrate the effectiveness of the proposed approach with varying levels of preprocessing in a realistic simulated environment. We also describe how the resulting model can be transferred to a real-world robot and also demonstrate a 68% success rate on real-world grasp attempts.

Sampling-based motion planning algorithms such as RRT* are well-known for their ability to quickly find an initial solution and then converge to the optimal solution asymptotically as the number of samples tends to infinity. However, the convergence rate can be slow for high-dimensional planning problems, particularly for dynamical systems where the sampling space is not just the configuration space but the full state space. In this paper, we introduce the idea of using a partial-final-state-free (PFF) optimal controller in kinodynamic RRT* [1] to reduce the dimensionality of the sampling space. Instead of sampling the full state space, the proposed accelerated kinodynamic RRT*, called Kino-RRT*, only samples part of the state space, while the rest of the states are selected by the PFF optimal controller. We also propose a delayed and intermittent update of the optimal arrival time of all the edges in the RRT* tree to decrease the computation complexity. We tested the proposed algorithm using 4-D and 10-D state-space linear systems and showed that Kino-RRT* converges much faster than the kinodynamic RRT* algorithm.

Localization is a crucial prerequisite for automated valet parking, in which a vehicle is required to navigate itself in a GPS-denied parking lot. Traditional visual localization methods usually build a feature map and use it for future localizations. However, the feature map is not robust to changes in illumination, appearance, and viewing perspective. To deal with this issue, we need a more stable map. In this paper, we propose to use the parking lot’s HD vector map directly for localization. The vector representation is ultimately stable but brings challenges in data association as well. To this end, we present a novel data association method to match the surround-view images with the vector map. In addition, we also propose a closed-form relocalization strategy by exploiting distinctive road mark combinations in the vector map. Experiments show that the proposed method is able to achieve centimeter-level localization accuracy in a multi-floor parking lot.

Recent advances in unsupervised learning for object detection, segmentation, and tracking hold significant promise for applications in robotics. A common approach is to frame these tasks as inference in probabilistic latent-variable models. In this paper, however, we show that the current state-of-the-art struggles with visually complex scenes such as typically encountered in robot manipulation tasks. We propose APEX, a new latent-variable model which is able to segment and track objects in more realistic scenes featuring objects that vary widely in size and texture, including the robot arm itself. This is achieved by a principled mask normalisation algorithm and a high-resolution scene encoder. To evaluate our approach, we present results on the real-world Sketchy dataset. This dataset, however, does not contain ground truth masks and object IDs for a quantitative evaluation. We thus introduce the Panda Pushing Dataset (P2D) which shows a Panda arm interacting with objects on a table in simulation and which includes ground-truth segmentation masks and object IDs for tracking. In both cases, APEX comprehensively outperforms the current state-of-the-art in unsupervised object segmentation and tracking. We demonstrate the efficacy of our segmentations for robot skill execution on an object arrangement task, where we also achieve the best or comparable performance among all the baselines.

Dynamic inference that adaptively skips parts of model execution based on the complexity of input data can effectively reduce the computation cost of deep learning models during the inference. However, current architectures for dynamic inference only consider the exits at the block level, whose results may not be suitable for different applications. In this paper, we present the Auto-Dynamic-DeepLab (ADD), a network architecture that enables the fine-grained dynamic inference for semantic image segmentation. To allow the exit points in the cell level, ADD utilizes Neural Architecture Search (NAS), supported by the framework of Auto-DeepLab, to seek the optimal network structure. In addition, ADD replaces the cells in Auto-DeepLab with the densely connected cells to ease the interference among multiple classifiers and employs the earlier decision maker (EDM) to further improve the performance. Experimental results show that ADD can achieve similar accuracy as Auto-DeepLab in terms of mIoU with a 1.6 times speedup. For the fast mode, ADD can achieve 2.15 times speedup with only a 1.3% accuracy drop compared to those of Auto-DeepLab.

There is increasing demand for robots that provide a mode of transportation in environments in which people coexist. However, conventional mobile robots, especially those carrying people, are limited in terms of their environments and tasks. For example, wheeled robots are limited to moving on flat ground. Walking robots are limited to entertainment and so on. The originality of the present paper is the development of a novel movement mechanism for a mobility apparatus that can handle various daily use scenes. We first clarify functional requirements for daily use. We then propose a transformable human-carrying wheel–leg mobility. The leg is based on a serial link and compactly uses a parallel link mechanism such that the motor is placed on top, which improves responsiveness when having a high payload and compact shape. By conducting simulations and experiments with the prototype, it is confirmed that expected operations can be realized. In particular, it is confirmed that the weight of the leg tips is reduced, such that the shaking of the waist in the direction of travel during the ascent of a step is reduced by 68%. The above results reveal that the prototype can be used in daily life.

Human-in-the-loop telerobotic systems (HiLTS) are robotic tools designed to extend and in some circumstances improve the dexterous capabilities of the human operator in virtual and remote environments. Dexterous manipulation, however, depends on how well the telerobot is incorporated into the operator’s sensorimotor control scheme. Empirical evidence suggests that haptic feedback can lead to improved dexterity. Unfortunately, haptic feedback can also introduce dynamics between the leader and follower of the telerobot that affect both stability and device performance. While concerted research effort has focused on masking these device dynamics or bypassing them altogether, it is not well understood how human operators incorporate these dynamics into their control scheme. We believe that to advance dexterous telerobotic manipulation, it is crucial to understand the process by which human operators incorporate teleoperator dynamics and distinguish them from the dynamics of the environment. Key to this knowledge is an understanding of how advanced telerobotic architectures compare to the gold standard, the rigid mechanical teleoperators first introduced in the 1950’s. In this manuscript, we present a teleoperator testbed that has reconfigurable transmissions between the leader and follower to change its dynamic behavior. The intent of this testbed is to investigate the effect of the teleoperator’s dynamics on perception of and task performance in the remote/virtual environment. We describe the hardware and software components of the testbed and then demonstrate how the different teleoperator transmissions can lead to differences, sometimes significant, in the dynamics that would be felt by the operator when exploring the same environment.

With an increasing number of robotic and prosthetic devices, there is a need for intuitive interfaces which enable the user to efficiently interact with them. The conventional interfaces are generally bulky and unsuitable for dynamic and unstructured environments. An alternative to the traditional interfaces is the class of Muscle-Machine Interfaces (MuMIs) that allow the user to have an embodied interaction with the devices they are controlling. In this work, we present a wearable, lightweight, Forcemyography (FMG) based armband for Human-Machine Interaction fabricated entirely out of 3D-printed parts and silicone components. The armband uses six force sensing units, each housing an Force Sensitive Resistor (FSR) sensor. The capabilities of the armband are evaluated while decoding four different gestures (pinch, power, tripod, extension) and rest state and its performance is compared with a state-of-the-art Electromyography (EMG) bioamplifier. The decoding performance of the decoding models trained on the data acquired from the armband is significantly better than the performance of the models trained on raw EMG data. The hardware design and the related processing software, are disseminated in an open-source manner.

This paper presents a hand-worn assistive device to assist a visually impaired person with object manipulation. The device uses a Google Pixel 3 as the computational platform, a Structure Core (SC) sensor for perception, a speech interface, and a haptic interface for human-device interaction. W-ROMA is intended to assist a visually impaired person to locate a target object (nearby or afar) and guide the user to move towards and eventually take a hold of the object. To achieve this objective, three functions, including object detection, wayfinding, and motion guidance, are developed. Object detection locates the target object’s position if it falls within the camera’s field of view. Wayfinding enables the user to approach the object. The haptic/speech interface guides the user to move close to the object and then guides the hand to reach the object. A new visual-inertial odometery (VIO), called RGBD-VIO, is devised to accurately estimate the device’s pose (position and orientation), which is then used to generate the motion command to guide the user and his/her hand to reach the object. Experimental results demonstrate that RGBD-VIO outperforms the state-of-the-art VIO methods in 6-DOF device pose estimation and the device is effective in assistive object manipulation.

Recovering rigid registration between successive camera poses lies at the heart of 3D reconstruction, SLAM and visual odometry. Registration relies on the ability to compute discriminative 2D features in successive camera images for determining feature correspondences, which is very challenging in feature-poor environments, i.e. low-texture and/or low-light environments. In this paper, we aim to address the challenge of recovering rigid registration between successive camera poses in feature-poor environments in a Visual Inertial Odometry (VIO) setting. In addition to inertial sensing, we instrument a small aerial robot with an RGBD camera and propose a framework that unifies the incorporation of 3D geometric entities: points, lines, and planes. The tracked 3D geometric entities provide constraints in an Extended Kalman Filtering framework. We show that by directly exploiting 3D geometric entities, we can achieve improved registration. We demonstrate our approach on different texture-poor environments, with some containing only flat texture-less surfaces providing essentially no 2D features for tracking. In addition, we evaluate how the addition of different 3D geometric entities contributes to improved pose estimation by comparing an estimated pose trajectory to a ground truth pose trajectory obtained from a motion capture system. We consider computationally efficient methods for detecting 3D points, lines and planes, since our goal is to implement our approach on small mobile robots, such as drones.

Deep reinforcement learning has achieved great success in laser-based collision avoidance work because the laser can sense accurate depth information without too much redundant data, which can maintain the robustness of the algorithm when it is migrated from the simulation environment to the real world. However, high-cost laser devices are not only difficult to apply on a large scale but also have poor robustness to irregular objects, e.g., tables, chairs, shelves, etc. In this paper, we propose a vision-based collision avoidance framework to solve the challenging problem. Our method attempts to estimate the depth and incorporate the semantic information from RGB data to obtain a new form of data, pseudo-laser data, which combines the advantages of visual information and laser information. Compared to traditional laser data that only contains the one-dimensional distance information captured at a certain height, our proposed pseudo-laser data encodes the depth information and semantic information within the image, which makes our method more effective for irregular obstacles. Besides, we adaptively add noise to the laser data during the training stage to increase the robustness of our model in the real world, due to the estimated depth information is not accurate. Experimental results show that our framework achieves state-of-the-art performance in several unseen virtual and real-world scenarios.

We present a topological method for finding coarsely diverse pathways. The use of pre-computed paths for online planning in a dynamic context reduces the overhead of re-planning alternate routes. Our algorithm applied the notion of discrete Morse theory to identify critical points incident on the obstacles and used this information to identify and return a diverse set of coarse paths. Three sampling-based planning approaches are converted to topology-aware planners and compared to another that employs the SPARS2 path planning algorithm. We report on the number of coarse pathways found, computation time, and average path length and show that our approach outperformed previously published path diversity algorithms.

This paper presents an underactuated robotic gripper that consists of three fingers. This gripper is driven by seven actuators and capable of grasping a wide range of objects in different working scenarios. A combination of a four-bar mechanism and parallelograms ensures that each finger can provide the basic pinch grasp and power grasp. Detailed fingertip grasping force analysis shows the large payload of this gripper. To fulfill multiple challenging grasping tasks, the fingertip orientation of each finger was designed to be decoupled from the finger flexion motion. Particular emphasis is placed on the environmental contact-based grasp and active transition from the pinch grasp to the power grasp. Detailed analysis shows that the contact-based fingertip grasp could be used to grasp thin objects lying on a flat surface with widths less than 168 mm safely and stably. To prevent over-squeezing of the grasped object in the active transition, the fingertip orientation and finger flexion motion were controlled coordinately. Moreover, a combination of the contact-based grasp and the direct power grasp was also allowed to grasp objects lying on a flat surface and having smooth surfaces robustly. Experimental results demonstrate the effectiveness of the proposed gripper in real-world applications.

In minimally invasive surgery, miniaturisation and in situ adjustable stiffness of robotic manipulators are desired features. Previous research proposed a simple and effective tendon-driven curve-joint manipulator design using a variable neutral-line mechanism, which highly satisfies both criteria. A kinematic model was developed for such a manipulator based on the geometry of the structure. However, such a model assumes that joint angles are all equal between disks without a rigorous derivation, and fails if not all the shapes of the disks are identical. Moreover, the model does not involve an analysis of the tension of each tendon. This paper suggested a static model for predicting the articulation of such a manipulator given the applied tensions on driving tendons. It validates the assumption of equally distributed joint angles and works for manipulators with more general configurations of disks and tendons. It also sets a foundation for further development of tension based control and external force estimation. Simulations on Adams were conducted to prove the correctness of the proposed model. A video demonstrating the simulation results can be found via https://youtu.be/MXhL1LGwLtw