IROS 2021 Papers

A novel siamese autoencoder visual odometry system named SAEVO is proposed in this paper. SAEVO can jointly estimate the 6-DoF pose and the depth using deep neural networks trained with monocular clips only. The main idea of the proposed method is an unsupervised deep learning scheme that combines siamese networks with auto-encoder for multi-scale matching to estimate ego-motion. Also, two unsupervised losses are designed to align extracted features from the siamese autoencoder networks. A system overview is shown in Fig. 1. The experiments on KITTI and CityScapes datasets demonstrate the SAEVO achieves good performance in terms of pose and depth accuracy, and competitive performance to state-of-the-art methods.

Precision agriculture is a fast-growing field that aims at introducing affordable and effective automation into agricultural processes. Nowadays, algorithmic solutions for navigation in vineyards require expensive sensors and high computational workloads that preclude large-scale applicability of autonomous robotic platforms in real business case scenarios. From this perspective, our novel proposed control leverages the latest advancement in machine perception and edge AI techniques to achieve highly affordable and reliable navigation inside vineyard rows with low computational and power consumption. Indeed, using a custom-trained segmentation network and a low-range RGB-D camera, we are able to take advantage of the semantic information of the environment to produce smooth trajectories and stable control in different vineyards scenarios. Moreover, the segmentation maps generated by the control algorithm itself could be directly exploited as filters for a vegetative assessment of the crop status. Extensive experimentations and evaluations against real-world data and simulated environments demonstrated the effectiveness and intrinsic robustness of our methodology.

Due to the high sensitivity and complexity of robotic surgery tasks, acquiring appropriate skill levels by trainee surgeons through an effective training process is very important and affects the patient’s safety and the quality of surgical outcomes. With the advanced deep learning technology and the recent availability of surgical procedures data, intelligent methods can be deployed to assess and transfer the skills of an experienced surgeon (mentor) to a novice surgeon (trainee). In this paper, we introduce a novel deep-learning-based skill transfer scheme consisting of a deep convolutional model, SkillNet, and a skill transfer algorithm for robotic surgery training. The proposed SkillNet extracts skill-related features of the mentor from different layers of the network. Then, trainee’s maneuver is enhanced by the proposed skill transfer algorithm while minimizing deviations from the trainee’s original intended trajectory. For validation, the JIGSAWS dataset and also our own experimental data were used to prove the generalizability of SkillNet in capturing skill-related features. The capability of the skill transfer algorithm in enhancing trainee trajectories in terms of predictability, hand tremor reduction, and noise cancellation were investigated separately. The obtained results indicate that this approach can be used as a high-performance filter that makes minor corrections to the input trajectory and improves the skill level of the trainee’s trajectory in practice.

Online human leg tracking and gait analysis are crucial functionalities for mobility assistant robots, like intelligent walkers. Usually, such walkers are equipped with various sensors for the extraction of human-related features for adaptive human-robot interaction and assistance. We treat the gait detection problem jointly, presenting a novel method for detecting and recognizing gait features from 2D range data produced by a laser sensor mounted on a robotic walker. We propose an effective Convolutional Neural Network (CNN) as a powerful feature extractor for detecting the user’s leg centers in range data represented as occupancy grid maps. We couple the CNN with a Long Short Term Memory (LSTM) network for learning the legs’ motion temporal dynamics while walking, improving the prior detection, and providing better leg occlusion handling. Moreover, we perform gait analysis by recognizing gait phases over both legs by feeding the leg tracking output to a subsequent LSTM. Our proposed lightweight framework has been trained and tested on real patients-data. The presented experimental results show our method’s efficiency in providing accurate detections compared to state-of-the-art and application to an online system due to its high frequency, making it a competitive method for gait detection on robotic mobility assistants.

Depth completion recovers a dense depth map from sensor measurements. Current methods are mostly tailored for very sparse depth measurements from LiDARs in outdoor settings, while for indoor scenes Time-of-Flight (ToF) or structured light sensors are mostly used. These sensors provide semi-dense maps, with dense measurements in some regions and almost empty in others. We propose a new model that takes into account the statistical difference between such regions. Our main contribution is a new decoder modulation branch added to the encoder-decoder architecture. The encoder extracts features from the concatenated RGB image and raw depth. Given the mask of missing values as input, the proposed modulation branch controls the decoding of a dense depth map from these features differently for different regions. This is implemented by modifying the spatial distribution of output signals inside the decoder via Spatially-Adaptive Denormalization (SPADE) blocks. Our second contribution is a novel on-the- y sensor simulation strategy that allows us to train on a semi-dense sensor data when the ground truth depth map is not available. Our model achieves the state of the art results on indoor Matterport3D dataset [1]. Being designed for semi-dense input depth, our model is still competitive with LiDAR-oriented approaches on the KITTI dataset [2]. Our sensor simulation strategy significantly improves prediction quality with no dense ground truth available, as validated on the NYUv2 dataset [3].

Forecasting the future behavior of all traffic agents in the vicinity is a key task to achieve safe and reliable autonomous driving systems. It is a challenging problem as agents adjust their behavior depending on their intentions, the others’ actions, and the road layout. In this paper, we propose Decoder Fusion RNN (DF-RNN), a recurrent, attention-based approach for motion forecasting. Our network is composed of a recurrent behavior encoder, an inter-agent multi-headed attention module, and a context-aware decoder. We design a map encoder that embeds polyline segments, combines them to create a graph structure, and merges their relevant parts with the agents’ embeddings. We fuse the encoded map information with further inter-agent interactions only inside the decoder and propose to use explicit training as a method to effectively utilize the information available. We demonstrate the efficacy of our method by testing it on the Argoverse motion forecasting dataset and show its state-of-the-art performance on the public benchmark.

We propose a decentralized, learning-based solution to the challenging problem of unlabeled multi-agent navigation among obstacles, where robots need to simultaneously tackle the problems of goal assignment, local collision avoidance, and navigation. Our method has each robot infer their desired action by communicating with each other as well as a set of position-fixed routers. The inference is carried out on a graph neural network (GNN) with both robot and router nodes. We train our GNN using imitation learning on a small group of robots, where we modify the centralized version of the concurrent goal assignment and planning algorithm (CAPT) as our expert. By sharing weights among all robots and routers, our model can scale to unseen environments with any number of possibly kinodynamic agents during test time. We have achieved a success rate of 91.2% and 85.6% for point and car-like robots, respectively. Source code will be publicly available upon the publication of the work.

Aerial manipulators have great potential in accomplishing a variety of aerial tasks. One class of aerial manipulators, multi-UAV parallel robots, consists of multiple UAVs connected to a payload or an end-effector by passive kinematic chains. The primary limitation of such aerial manipulators is the dependence on motion capture (MOCAP) systems that provide precise and high-rate exteroceptive pose measurements of all bodies in a common inertial frame, but which are impractical in the majority of real applications.This paper proposes a novel methodology of controlling multi-UAV parallel robots, using a Flying Parallel Robot (FPR) as a case study, that could be deployed without a system of external localisation. Intrinsic measurements acquired onboard the UAVs are used to recover a set of robot states that avoid using coordinates derived from a global frame and allow control of the robot by teleoperation. Two decentralized control methods are proposed, based on inter-UAV communicating or non-communicating scenarios. Experiments with intrinsic measurements emulated by MOCAP are carried out to show the performance of the proposed method.

We study the problem of decentralized classification conducted over a network of mobile sensors. We model the multiagent classification task as a hypothesis testing problem where each sensor has to almost surely find the true hypothesis from a finite set of candidate hypotheses. Each sensor makes noisy local observations and can also share information on their observations with other mobile sensors in communication range. In order to address the state-space explosion in the multiagent system, we propose a decentralized synthesis procedure that guarantees that each sensor will almost surely converge to the true hypothesis even in the presence of faulty or malicious agents. Additionally, we employ a contract-based synthesis approach that produces trajectories designed to empirically increase information-sharing between mobile sensors in order to converge faster to the true hypothesis. We implement and test the approach on experiments with both physical and simulated hardware to showcase the approach’s scalability and viability in real-world systems. Finally, we run a Gazebo/ROS simulated experiment with 12 agents to demonstrate the scalability of our approach in large environments with many agents.

This paper introduces a distributed multi-robot collision avoidance algorithm based on the concept of Buffered Voronoi Cells (BVC). We propose a novel algorithm for avoiding deadlocks consisting of three stages: deadlock prediction, deadlock recovery, and deadlock recovery success prediction. Simple heuristics (such as the right-hand rule) are often used to avoid deadlocks. Such heuristics might reduce deadlock in simple configurations and sparsely populated environments, but they begin to fail in complex configurations and more densely populated environments. We evaluate the performance of our algorithm using an open-source web-based multi-robot simulation. The results show that while the proposed algorithm does not eliminate the occurrence of deadlocks, it drastically reduces their occurrence, and leads to a considerable improvement in performance, especially in high-density environments. We also validate the real-world performance of the proposed algorithm in live experiments.

Robust off-road perception for autonomous navigation is hard to achieve. Versatile environments, different hardware, and numerous disturbances limit the perceptional portability in changing applications and cross-platform. This contribution proposes sensor-fusion considering the data quality of uncertain sensors to increase the classification and mapping components’ perceptual robustness. The resulting benefits on perception are demonstrated using the autonomous off-road robot U5023.

Recent years have witnessed the fast evolution and promising performance of the convolutional neural network (CNN)-based trackers, which aim at imitating biological visual systems. However, current CNN-based trackers can hardly generalize well to low-light scenes that are commonly lacked in the existing training set. In indistinguishable night scenarios frequently encountered in unmanned aerial vehicle (UAV) tracking-based applications, the robustness of the state-of-the-art (SOTA) trackers drops significantly. To facilitate aerial tracking in the dark through a general fashion, this work proposes a low-light image enhancer namely DarkLighter, which dedicates to alleviate the impact of poor illumination and noise iteratively. A lightweight map estimation network, i.e., ME-Net, is trained to efficiently estimate illumination maps and noise maps jointly. Experiments are conducted with several SOTA trackers on numerous UAV dark tracking scenes. Exhaustive evaluations demonstrate the reliability and universality of DarkLighter, with high efficiency. Moreover, DarkLighter has further been implemented on a typical UAV system. Real-world tests at night scenes have verified its practicability and dependability.

Localizing a vehicle on a prebuilt HD vector map is a prerequisite for many autonomous driving applications. Existing visual localization approaches usually require a separate local feature layer to function. The separate localization layer suffers from the robustness issue inherited from the local features. Also, it could be difficult to create a feature layer that aligns perfectly with an existing vector map. In this paper, we propose a monocular visual localization method that exploits the vector map directly as the localization layer. The method detects semantic traffic elements from the images and matches them with the vectors in the map. To deal with the harmful problem of false matches, we propose to align the vector map to the distance transforms of the semantic detections, which enables a non-explicit and differentiable data association process. The system is able to achieve centimeter and sub-meter accuracies in lateral and longitudinal directions, respectively.

Path planning for a robot is one of the major problems in the area of robotics. When a robot is given a task in the form of a Linear Temporal Logic (LTL) specification such that the task needs to be carried out repetitively, we want the robot to follow the shortest cyclic path so that the number of times the robot completes the mission within a given duration gets maximized. In this paper, we address the LTL path planning problem in a dynamic environment where the newly arrived dynamic obstacles may invalidate some of the available paths at any arbitrary point in time. We present DT*, an SMT-based receding horizon planning strategy that solves an optimization problem repetitively based on the current status of the workspace to lead the robot to follow the best available path in the current situation. We implement our algorithm using the Z3 SMT solver and evaluate it extensively on an LTL specification capturing a pick-and-drop application in a warehouse environment and an office environment2. We compare our SMT-based algorithm with two carefully crafted greedy algorithms. Our experimental results show that the proposed algorithm can deal with the dynamism in the workspace in LTL path planning effectively.

We present a method for efficiently exploring highly convoluted environments. The method incorporates two planning stages - an exploration stage for extending the boundary of the map, and a relocation stage for explicitly transiting the robot to different sub-areas in the environment. The exploration stage develops a local Rapidly-exploring Random Tree (RRT) in the free space of the environment, and the relocation stage maintains a global graph through the mapped environment, both are dynamically expanded over replanning steps. The method is compared to existing state-of-the-art methods in various challenging simulation and real environments. Experiment comparisons show that our method is twice as efficient in exploring spaces using less processing than the existing methods. Further, we release a benchmark environment to evaluate exploration algorithms as well as facilitate development of autonomous navigation systems. The benchmark environment and our method are open-sourced.

In this paper, we propose a map-based end-to-end DRL approach for three-dimensional (3D) obstacle avoidance in a partially observed environment, which is applied to achieve autonomous navigation for an indoor mobile robot using a depth camera with a narrow field of view. We first train a neural network with LSTM units in a 3D simulator of mobile robots to approximate the Q-value function in double DRQN. We also use a curriculum learning strategy to accelerate and stabilize the training process. Then we deploy the trained model to a real robot to perform 3D obstacle avoidance in its navigation. We evaluate the proposed approach both in the simulated environment and on a robot in the real world. The experimental results show that the approach is efficient and easy to be deployed, and it performs well for 3D obstacle avoidance with a narrow observation angle, which outperforms other existing DRL-based models by 15.5% on success rate.

Learning an accurate model of the environment is essential for model-based control tasks. Existing methods in robotic visuomotor control usually learn from data with heavily labelled actions, object entities or locations, which can be demanding in many cases. To cope with this limitation, we propose a method, dubbed DMotion, that trains a forward model from video data only, via disentangling the motion of controllable agent to model the transition dynamics. An object extractor and an interaction learner are trained in an end-to-end manner without supervision. The agent’s motions are explicitly represented using spatial transformation matrices containing physical meanings. In the experiments, DMotion achieves superior performance on learning an accurate forward model in a Grid World environment, as well as a more realistic robot control environment in simulation. With the accurate learned forward models, we further demonstrate their usage in model predictive control as an effective approach for robotic manipulations. Code, video and more materials are available at: https://hyperplane-lab.github.io/dmotion.

This paper presents DLL, a fast direct map-based localization technique using 3D LIDAR for its application to aerial robots. DLL implements a point cloud to map registration based on non-linear optimization of the distance of the points and the map, thus not requiring features, neither point correspondences. Given an initial pose, the method is able to track the pose of the robot by refining the predicted pose from odometry. Through benchmarks using real datasets and simulations, we show how the method performs much better than Monte-Carlo localization methods and achieves comparable precision to other optimization-based approaches but running one order of magnitude faster. The method is also robust under odometric errors. The approach has been implemented under the Robot Operating System (ROS), and it is publicly available.

In imitation learning from observation (IfO), a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. In this work, we hypothesize that we can incorporate ideas from model-based reinforcement learning with adversarial methods for IfO in order to increase the data efficiency of these methods without sacrificing performance. Specifically, we consider time-varying linear Gaussian policies, and propose a method that integrates the linear-quadratic regulator with path integral policy improvement into an existing adversarial IfO framework. The result is a more data-efficient IfO algorithm with better performance, which we show empirically in four simulation domains: using far fewer interactions with the environment, the proposed method exhibits similar or better performance than the existing technique.