IROS 2021 Papers

Current existing stereo visual odometry algorithms are computationally too expensive for robots with restricted resources. Executing these algorithms on such robots leads to a low frame rate and unacceptable decay in accuracy. We modify S-MSCKF, one of the most computationally efficient stereo Visual Inertial Odometry (VIO) algorithm, to improve its speed and accuracy when tracking low numbers of features. Specifically, we implement the Inverse Lucas-Kanade (ILK) algorithm for feature tracking and stereo matching. An outlier detector based on the average sum square difference of the template and matching warp in the ILK ensures higher robustness, e.g., in the presence of brightness changes. We restrict stereo matching to slide the window only in the x-direction to further decrease the computational costs. Moreover, we limit detection of new features to the regions of interest that have too few features. The modified S-MSCKF uses half of the processing time while obtaining competitive accuracy. This allows the algorithm to run in real-time on the extremely limited Raspberry Pi Zero single-board computer.

Plane features can be used to reduce drift errors in SLAM systems, especially in indoor environments. It is easy and efficient to extract planes from a dense point cloud, which is commonly generated from a RGB-D camera or a 3D lidar. But when using a stereo camera, it is hard to compute dense point clouds accurately or efficiently. In this paper, we propose a novel method to compute plane parameters using intersecting lines, which are extracted from stereo images. Plane features are commonly extracted from the surface of man-made objects or structures, which have regular shapes and straight edge lines. In three dimensions, two intersecting lines determine a unique plane. Therefore, we extract line segments from both left and right images of a stereo camera. By stereo matching, we compute lines’ endpoints and direction vectors, and then a plane from two intersecting lines is calculated. We discard inaccurate plane features in the frame tracking. Adding such plane features in the stereo SLAM system reduces drift errors and refines the performance. Finally, we build a global map consisting of both points and planes, which can reflect real scene structures. We test our proposed system on public datasets and demonstrate its accurate estimation results, compared with state-of-the-art SLAM systems. To benefit the research of plane-based SLAM, we release our codes at https://github.com/fishmarch/Stereo-Plane-SLAM.

Perception using stereo requires external light. In the absence of natural light, active, structured light provides light where it is needed. In this work, we demonstrate how a free moving line striping laser can be used to perceive and model terrains. In this formulation, we do not need to know the position of the laser with respect to the stereo pair which precludes the need for calibrating the laser position. This also saves an actuator as the laser can be passively articulated using a spring. In this paper, we present the algorithms to efficiently extract the laser pixels from the stereo image pair. We also present techniques to use the structure of the light and overcome degenerate cases to build cleaner maps. This perception method benefits micro-rovers specifically those trying to operate in the extreme lighting conditions.

Self-supervised learning for depth estimation possesses several advantages over supervised learning. The benefits of no need for ground-truth depth, online fine-tuning, and better generalization with unlimited data attract researchers to seek self-supervised solutions. In this work, we propose a new self-supervised framework for stereo matching utilizing multiple images captured at aligned camera positions. A cross photometric loss, an uncertainty-aware mutual-supervision loss, and a new smoothness loss are introduced to optimize the network in learning disparity maps end-to-end without ground-truth depth information. To train this framework, we build a new multiscopic dataset consisting of synthetic images rendered by 3D engines and real images captured by real cameras. After being trained with only the synthetic images, our network can perform well in unseen outdoor scenes. Our experiment shows that our model obtains better disparity maps than previous unsupervised methods on the KITTI dataset and is comparable to supervised methods when generalized to unseen data. Our source code and dataset are available at https://sites.google.com/view/multiscopic.

Stereo camera systems play an important role in robotics applications to perceive the 3D world. However, conventional cameras have drawbacks such as low dynamic range, motion blur and latency due to the underlying frame- based mechanism. Event cameras address these limitations as they report the brightness changes of each pixel independently with a fine temporal resolution, but they are unable to acquire absolute intensity information directly. Although integrated hybrid event-frame sensors (e.g., DAVIS) are available, the quality of data is compromised by coupling at the pixel level in the circuit fabrication of such cameras. This paper proposes a stereo hybrid event-frame (SHEF) camera system that offers a sensor modality with separate high-quality pure event and pure frame cameras, overcoming the limitations of each separate sensor and allowing for stereo depth estimation. We provide a SHEF dataset targeted at evaluating disparity estimation algorithms and introduce a stereo disparity estimation algorithm that uses edge information extracted from the event stream correlated with the edge detected in the frame data. Our disparity estimation outperforms the state-of-the-art stereo matching algorithm on the SHEF dataset.

Modern model-free reinforcement learning methods have recently demonstrated impressive results on a number of problems. However, complex domains like dexterous manipulation remain a challenge due to the high sample complexity. To address this, current approaches employ expert demonstrations in the form of state-action pairs, which are difficult to obtain for real-world settings such as learning from videos. In this paper, we move toward a more realistic setting and explore state-only imitation learning. To tackle this setting, we train an inverse dynamics model and use it to predict actions for state-only demonstrations. The inverse dynamics model and the policy are trained jointly. Our method performs on par with state-action approaches and considerably outperforms RL alone. By not relying on expert actions, we are able to learn from demonstrations with different dynamics, morphologies, and objects. Videos available on the {\text{project page}}{\text{project page}}.

This paper presents a technique for state estimation of a multi-link system having no joint encoders, which can only be partially observed by a camera. To fully observe the system without changing the current configuration, a gyroscope and an accelerometer are attached to each link as dead-reckoning sensors. Observations of the dead-reckoning sensors are associated with the states of the multi-link system such that the states are fully observable. The camera, which observes part of the system globally, is used as a global corrector in the framework of an extended Kalman filter to filter the dead-reckoning errors accumulated over time. Parametric studies in simulation have investigated and identified the efficacy of the proposed technique in estimating the state of the multilink system. Experimental validation using a two-link arm has demonstrated the applicability of the proposed technique to real-world multi-link systems.

In this paper, we present a solution for simultaneous handling of large components with industrial robots performing synchronized motions with an AGV in flexible flow assembly. For this purpose, we implement an Extended Kalman Filter with a global localization system to track an AGV and multiple manipulators. We propose a model-predictive controller for force compliance and trajectory tracking in multi-robot cooperative, decentralized, and fast manipulation tasks. In order to show the effectiveness of our system, we assemble a truck windshield using two industrial robots and an AGV in motion. In our experiments, we reliably achieve assembly tolerances of 1.5mm at AGV velocities up to 400\frac{{{\text{mm}}}}{{\text{s}}}400\frac{{{\text{mm}}}}{{\text{s}}}. The presented system makes flexible assembly systems with AGVs and freely reconfigurable manipulators possible. It enables the automation of high variant, low volume, large size assembly tasks such as aircraft, truck or steel beam assembly, which are mostly manual processes at present.

We present a novel haptic teleoperation approach that considers not only the safety but also the stability of a teleoperation system. Specifically, we build upon previous work on haptic shared control, which generates a reference haptic feedback that helps the human operator to safely navigate the robot but without taking away their control authority. Crucially, in this approach the force rendered to the user is not directly reflected in the motion of the robot (which is still directly controlled by the user); however, previous work in the area neglected to consider the possible instabilities in feedback loop generated by a user that over-responds to the haptic force. In this paper we introduce a differential constraint on the rendered force that makes the system finite-gain {{\mathcal{L}}_2}{{\mathcal{L}}_2} stable; the constraint results in a Quadratically Constrained Quadratic Program (QCQP), for which we provide a closed-form solution. Our constraint is related to, but less restrictive than, the typical passivity constraint used in previous literature. We conducted an experimental simulation in which a human operator flies a UAV near an obstacle to evaluate the proposed method.

In this paper, an autonomous aerial manipulation task of pulling a plug out of an electric socket is conducted, where maintaining the stability and robustness is challenging due to sudden disappearance of a large interaction force. The abrupt change in the dynamical model before and after the separation of the plug can cause destabilization or mission failure. To accomplish aerial plug-pulling, we employ the concept of hybrid automata to divide the task into three operative modes, i.e, wire-pulling, stabilizing, and free-flight. Also, a strategy for trajectory generation and a design of disturbance-observer-based controllers for each operative mode are presented. Furthermore, the theory of hybrid automata is used to prove the stability and robustness during the mode transition. We validate the proposed trajectory generation and control method by an actual wire-pulling experiment with a multirotor-based aerial manipulator.

Spiking Neural Networks (SNN) are the so-called third generation of neural networks which attempt to more closely match the functioning of the biological brain. They inherently encode temporal data, allowing for training with less energy usage and can be extremely energy efficient when coded on neuromorphic hardware. In addition, they are well suited for tasks involving event-based sensors, which match the event-based nature of the SNN. However, SNNs have not been as effectively applied to real-world, large-scale tasks as standard Artificial Neural Networks (ANNs) due to the algorithmic and training complexity. To exacerbate the situation further, the input representation is unconventional and requires careful analysis and deep understanding. In this paper, we propose SpikeMS, the first deep encoder-decoder SNN architecture for the real-world large-scale problem of motion segmentation using the event-based DVS camera as input. To accomplish this, we introduce a novel spatio-temporal loss formulation that includes both spike counts and classification labels in conjunction with the use of new techniques for SNN backpropagation. In addition, we show that SpikeMS is capable of incremental predictions, or predictions from smaller amounts of test data than it is trained on. This is invaluable for providing outputs even with partial input data for low-latency applications and those requiring fast predictions. We evaluated SpikeMS on challenging synthetic and real-world sequences from EV-IMO, EED and MOD datasets and achieving results on a par with a comparable ANN method, but using potentially 50 times less power.

This paper presents a spectral correlation-based method (SpectGRASP) for robotic grasping of arbitrarily shaped, unknown objects. Given a point cloud of an object, SpectGRASP extracts contact points on the object’s surface matching the hand configuration. It neither requires offline training nor a-priori object models. We propose a novel Binary Extended Gaussian Image (BEGI), which represents the point cloud surface normals of both object and robot fingers as signals on a 2-sphere. Spherical harmonics are then used to estimate the correlation between fingers and object BEGIs. The resulting spectral correlation density function provides a similarity measure of gripper and object surface normals. This is highly efficient in that it is simultaneously evaluated at all possible finger rotations in SO(3). A set of contact points are then extracted for each finger using rotations with high correlation values. We then use our previous work, Local Contact Moment (LoCoMo) similarity metric, to sequentially rank the generated grasps such that the one with maximum likelihood is executed. We evaluate the performance of SpectGRASP by conducting experiments with a 7-axis robot fitted with a parallel-jaw gripper, in a physics simulation environment. Obtained results indicate that the method not only can grasp individual objects, but also can successfully clear randomly organized groups of objects. The SpectGRASP method also outperforms the closest state-of-the-art method in terms of grasp generation time and grasp-efficiency.

The imagination of the surrounding environment based on the experience and semantic cognition has great potential to extend the limited observations to leverage the ability for mapping, collision avoidance and path planning. This paper provides a training-based algorithm for mobile robots to perform spatial imagination based on semantic cognition and evaluates the proposed method for the mapping task. We utilize a photo-realistic simulation environment, Habitat, for training and evaluation. The trained model is composed of Resent-18 as encoder and U-net as the backbone. We demonstrate that the algorithm can perform imagination for unseen parts of the object universally, by recalling the images and experience and compare our approach with traditional semantic mapping methods. It is found that our approach will improve the efficiency and accuracy of semantic mapping.

This paper presents a novel method to generate spatial constraints for motion planning in dynamic environments. Motion planning methods for autonomous driving and mobile robots typically need to rely on the spatial constraints imposed by a map-based global planner to generate a collision-free trajectory. These methods may fail without an offline map or where the map is invalid due to dynamic changes in the environment such as road obstruction, construction, and traffic congestion. To address this problem, triangulation-based methods can be used to obtain a spatial constraint. However, the existing methods fall short when dealing with dynamic environments and may lead the motion planner to an unrecoverable state. In this paper, we propose a new method to generate a sequence of channels across different triangulation mesh topologies to serve as the spatial constraints. This can be applied to motion planning of autonomous vehicles or robots in cluttered, unstructured environments. The proposed method is evaluated and compared with other triangulation-based methods in synthetic and complex scenarios collected from a real-world autonomous driving dataset. We have shown that the proposed method results in a more stable, long-term plan with a higher task completion rate, faster arrival time, a higher rate of successful plans, and fewer collisions compared to existing methods.

Reinforcement learning has been widely applied in exploration, navigation, manipulation, and other fields. Most of the relevant techniques generate kinematic commands (e.g., move, stop, turn) for agents based on the current state information. However, recent dense action representations based research, such as spatial action maps, pointing way-points to the agent in the same domain as its observation of the state shows great promise in mobile manipulation tasks. Inspired by that, we make the first step towards using a spatial action maps based method to effectively explore novel environmental spaces. To reduce the chance of redundant exploration, the visit frequency map (VFM) and its corresponding reward function are introduced to direct the agent to actively search previously unexplored areas. In the experimental section, our work was compared to the same method without VFM and the method based on traditional steering commands with the same input data in various environments. The results show conclusively that our method is more efficient than other methods. The project page is: https://github.com/zxwang96/sam-exploration

Path planning for multiple robots (MRPP) represents a task of finding non-colliding paths for robots via which they can navigate from their initial positions to specified goal positions. The problem is often modeled using undirected graphs where robots move between vertices across edges while no two robots can simultaneously occupy the same vertex nor can traverse an edge in opposite directions. Contemporary optimal solving algorithms include dedicated search-based methods, that solve the problem directly, and compilation-based algorithms that reduce MRPP to a different formalism for which an efficient solver exists, such as constraint programming (CP), mixed integer linear programming (MILP), or Boolean satisfiability (SAT). In this paper, we enhance existing SAT-based algorithm for MRPP via sparsification of the set of candidate paths for each robot from which the target Boolean encoding is derived. Suggested sparsification of the set of paths led to a smaller target Boolean formulae that can be constructed and solved faster while optimality guarantees of the approach have been kept.

This paper focuses on the problem of multi-robot source-seeking, where a group of mobile sensors localizes and moves close to a single source using only local measurements. Drawing inspiration from the optimal sensor placement research, we develop an algorithm that estimates the source location while approaches the source following gradient descent steps on a loss function defined on the Fisher information. We show that exploiting Fisher information gives a higher chance of obtaining an accurate source location estimate and naturally leads the sensors to the source. Our numerical experiments demonstrate the advantages of our algorithm, including faster convergence to the source than other algorithms, flexibility in the choice of the loss function, and robustness to measurement modeling errors. Moreover, the performance improves as the number of sensors increases, showing the advantage of using multi-robots in our source-seeking algorithm. We also implement physical experiments to test the algorithm on small ground vehicles with light sensors, demonstrating success in seeking a moving light source.

SLAM in deformable environments is a very challenging research topic. Some research works have been presented by different research groups in the past few years. However, there are still some challenging research questions remaining unanswered. This paper discusses some of these research questions focusing on the case when point features are used to describe the deformable environments. The SLAM problems are formulated as extensions of point feature based SLAM in static environments, including both optimisation based offline SLAM and filter based online SLAM. To illustrate the problems and questions more clearly, some concepts and results using simple 2D examples are presented. The MATLAB source codes of the results are made publicly available (https://github.com/cyb1212/DeformableSLAM2D.git) to help the readers understand the problems more clearly.

To date, soft robots have been increasingly designed and analyzed, especially, Soft Continuum Manipulators (SCMs). Due to dexterous deformability, their Inverse Kinematics (IK) is still difficult to solve. Cyclic Coordinate Descent (CCD) algorithm is one of the classical optimization algorithms to solve IK of rigid manipulators with prismatic or rotational joints. However, it cannot be directly extrapolated to SCMs with configuration space parameters such as center arc, bending angle, and torsion angle. Here, we modified the CCD algorithm from a new view and proposed several tricks to set constraints. Numerical and experimental results show that the soft-CCD algorithm can quickly and accurately generate solutions for IK of SCMs. This study provides the necessary kinematic foundation for fault tolerance, obstacle avoidance, trajectory planning, and other further explorations of SCMs.