IROS 2020 Papers

Visually poor scenarios are one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present MOZARD, a multi-modal localization system for urban outdoor environments using vision and LiDAR. By fusing key point based visual multi-session information with semantic data, an improved localization recall can be achieved across vastly different appearance conditions. In particular we focus on the use of curbstone information because of their broad distribution and reliability within urban environments. We present thorough experimental evaluations on several driving kilometers in challenging urban outdoor environments, analyze the recall and accuracy of our localization system and demonstrate in a case study possible failure cases of each subsystem. We demonstrate that MOZARD is able to bridge scenarios where our previous key point based visual approach, VIZARD, fails, hence yielding an increased recall performance, while a similar localization accuracy of 0.2m is achieved.

Extrinsic perturbation always exists in multiple sensors. In this paper, we focus on the extrinsic uncertainty in multi-LiDAR systems for 3D object detection. We first analyze the influence of extrinsic perturbation on geometric tasks with two basic examples. To minimize the detrimental effect of extrinsic perturbation, we propagate an uncertainty prior on each point of input point clouds, and use this information to boost an approach for 3D geometric tasks. Then we extend our findings to propose a multi-LiDAR 3D object detector called MLOD. MLOD is a two-stage network where the multi-LiDAR information is fused through various schemes in stage one, and the extrinsic perturbation is handled in stage two. We conduct extensive experiments on a real-world dataset, and demonstrate both the accuracy and robustness improvement of MLOD. The code, data and supplementary materials are available at: https://ram-lab.com/file/site/mlod.

In this paper, we propose a new concept of robot which is hybrid, including aerial and crawling subsystems and an arm, and also modular with interchangeable crawling subsystems for different pipe configurations, since it has been designed to cover most industrial oil & gas end-users' requirements. The robot has the same ability than aerial robots to reach otherwise inaccessible locations, but makes the inspection more efficient, increasing operation time since crawling requires less energy than flying, and achieving better accuracy in the inspection. It also integrates safety-related characteristics for operating in the potentially explosive atmosphere of a refinery, being able to immediately interrupt the inspection if a hazardous situation is detected and carry the sensible parts such as batteries and electronic devices away as soon as possible. The paper presents the design of this platform in detail and shows the feasibility of the whole system performing indoor experiments.

Multi-agent navigation in dynamic environments is of great industrial value when deploying a large scale fleet of robot to real-world applications. This paper proposes a decentralized partially observable multi-agent path planning with evolutionary reinforcement learning (MAPPER) method to learn an effective local planning policy in mixed dynamic environments. Reinforcement learning-based methods usually suffer performance degradation on long-horizon tasks with goal-conditioned sparse rewards, so we decompose the long-range navigation task into many easier sub-tasks under the guidance of a global planner, which increases agents' performance in large environments. Moreover, most existing multi-agent planning approaches assume either perfect information of the surrounding environment or homogeneity of nearby dynamic agents, which may not hold in practice. Our approach models dynamic obstacles' behavior with an image-based representation and trains a policy in mixed dynamic environments without homogeneity assumption. To ensure multi-agent training stability and performance, we propose an evolutionary training approach that can be easily scaled to large and complex environments. Experiments show that MAPPER is able to achieve higher success rates and more stable performance when exposed to a large number of non-cooperative dynamic obstacles compared with traditional reaction-based planner LRA* and the state-of-the-art learning-based method.

This paper addresses reactive generation of step time and location of biped robots for balance recovery against a severe push. Key idea is to reformulate the balance recovery problem into a tracking problem for "hybrid" inverted pendulum model of the biped, where taking a new step implicitly yields a discrete jump of the tracking error. This interpretation offers a Lyapunov-based approach to reactive step generation, which is possibly more intuitive and easier to analyze than large-scaled or nonlinear optimization-based approaches. With the continuous error dynamics for the divergent component of motion (DCM), our strategy for step generation is to decrease the "post-step" Lyapunov level for DCM error at each walking cycle, until it eventually becomes smaller than a threshold so that no more footstep needs to be adjusted. We show that implementation of this idea while obeying physical constraints can be done by employing a hybrid tracking controller (together with a reference model) as our reactive step generator, consisting of a simple DCM-based continuous controller and a small-sized quadratic programming-based discrete controller. The validity of the proposed scheme is verified by simulation results.

Rapid progress in embedded computing hardware increasingly enables on-board image processing on small robots. This development opens the path to replacing costly sensors with sophisticated computer vision techniques. A case in point is the prediction of scene depth information from a monocular camera for autonomous navigation. Motivated by the aim to develop a robot swarm suitable for sensing, monitoring, and search applications in forests, we have collected a set of RGB images and corresponding depth maps. Over 100000 RGB/depth image pairs were recorded with a custom rig from the perspective of a small ground rover moving through a forest. Taken under different weather and lighting conditions, the images include scenes with grass, bushes, standing and fallen trees, tree branches, leaves, and dirt. In addition GPS, IMU, and wheel encoder data were recorded. From the calibrated, synchronized, aligned and timestamped frames about 9700 image-depth map pairs were selected for sharpness and variety. We provide this dataset to the community to fill a need identified in our own research and hope it will accelerate progress in robots navigating the challenging forest environment. This paper describes our custom hardware and methodology to collect the data, subsequent processing and quality of the data, and how to access it.

The novelty of this study consists in a multi-modality approach to scene classification, where image and audio complement each other in a process of deep late fusion. The approach is demonstrated on a difficult classification problem, consisting of two synchronised and balanced datasets of 16,000 data objects, encompassing 4.4 hours of video of 8 environments with varying degrees of similarity. We first extract video frames and accompanying audio at one second intervals. The image and the audio datasets are first classified independently, using a fine-tuned VGG16 and an evolutionary optimised deep neural network, with accuracies of 89.27% and 93.72%, respectively. This is followed by late fusion of the two neural networks to enable a higher order function, leading to accuracy of 96.81% in this multi-modality classifier with synchronised video frames and audio clips. The tertiary neural network implemented for late fusion outperforms classical state-of-the-art classifiers by around 3% when the two primary networks are considered as feature generators. We show that situations where a single-modality may be confused by anomalous data points are now corrected through an emerging higher order integration. Prominent examples include a water feature in a city misclassified as a river by the audio classifier alone and a densely crowded street misclassified as a forest by the image classifier alone. Both are examples which are correctly classified by our multi-modality approach.

In many applications of mobile robot, the environment is constantly changing. How to use historical information to analysis environmental changes and generate a map corresponding with current environment is important to achieve high-precision localization. Inspired by predictive mechanism of brain, this paper presents a long-term localization approach named ArmMPU (ARMA-based Map Prediction and Update) based on time series modeling and prediction. Autoregressive moving average model (ARMA), a kind of time series modeling method, is employed for environmental map modeling and prediction, then predicted map and filtered observation are fused to fix the prediction error. The simulation and experiment results show that the proposed method improves long-term localization performance in dynamic environments.

This paper presents an approach for multi-robot long-term planning under uncertainty over the duration of actions. The proposed methodology takes advantage of generalized stochastic Petri nets with rewards (GSPNR) to model multi-robot problems. A GSPNR allows for unified modeling of action selection, uncertainty on the duration of action execution, and for goal specification through the use of transition rewards and rewards per time unit. Our approach relies on the interpretation of the GSPNR model as an equivalent embedded Markov reward automaton (MRA). We then build on a state-of-the-art method to compute the long-run average reward over MRAs, extending it to enable the extraction of the optimal policy. We provide an empirical evaluation of the proposed approach on a simulated multi-robot monitoring problem, evaluating its performance and scalability. The results show that the synthesized policy outperforms a policy obtained from an infinite horizon discounted reward formulation as well as a carefully hand-crafted policy.

To reduce the risk of radiation leakages similar to the incident at the Fukushima Daiichi Nuclear Power Station, robots have been employed to remove fuel debris from reactors. To perform this process safely, it is important to monitor the interior of a reactor. A camera and neutron sensors are attached to the end of a robotic arm to monitor the interior of the reactor. The basic design requirement for the monitoring system is that the arm must be highly extendable and rigid. To achieve this, a novel compact long-reach manipulator with a joint structure built using a low-melting-point alloy (LMPA) is proposed. The LMPA enables switching between the free and locked states of the rotational joints of the manipulator. Herein, we first explain the design of the proposed joint structure and verify whether it has adequate mechanical strength. The required maximum torque to be sustained by the structure was calculated using the cantilever model, and the actual breaking torque was measured by the tensile test. Experimental results confirmed that the joint could withstand approximately 1.86 times the required torque. Finally, the effectiveness of induction heating, which is used to switch between the free and locked states of the joints, was evaluated experimentally. The LMPA arm was installed in the coil of the induction heating module, and the time required to melt LMPA was measured. The experimental results confirmed that the induction heating can change the state of the LMPA joint, and the time required for the melting is approximately 30.3 s. Therefore, the findings of this study show that the proposed system is capable of averting nuclear disasters through the prevention of radiation leakages at nuclear plants.

To access rough terrain and enhance the mobility in sandy terrain, a configurable paddle-wheel robot was pro-posed. This report addresses the paddle terradynamics, and the experimental verification of the locomotion performance of the robot over dry sandy terrain. To study the interactive forces between the paddle and the media, a terradynamic model is built and verified through experiments. To explore the locomotion performance, an indoor platform that allows the paddle-wheel module to move freely in both horizontal and vertical directions is created. Forward locomotion speed, height variant, and specific resistance are evaluated with different con-figurations. The protruding paddles have successfully reduced the slippage so as to increase the locomotion efficiency in sandy terrain. The performance of the whole robot has also been verified in outdoor sandy terrain.

We propose a method to facilitate robot navigation relative to sketched maps of human environments. Our main contribution centers around using thin plate splines for registering the robot's LIDAR observation with the hand-drawn maps. Thin plate splines are particularly effective for this task because they are able to handle many of the nonrigid deformations commonly seen in sketches of maps, which render traditional rigid transformations inappropriate. Our proposed approach uses a convolutional neural network to efficiently predict the control points which define the spline transform, from which we then compute the pose of the robot on the hand drawn map for navigation purposes. Our systematic evaluations in simulation using a synthetic dataset and real, hand-drawn sketches show that the proposed spline-based registration approach outperforms baseline methods.

Tactile sensors are used in robot manipulation to reduce uncertainty regarding hand-object pose estimation. However, existing sensor technologies tend to be bulky and provide signals that are difficult to interpret into actionable changes. Here, we achieve wireless tactile sensing with soft and conformable magnetic stickers that can be easily placed on objects within the robot's workspace. We embed a small magnetometer within the robot's fingertip that can localize to a magnetic sticker with sub-mm accuracy and enable the robot to pick up objects in the same place, in the same way, every time. In addition, we utilize the soft magnets' ability to exhibit magnetic field changes upon contact forces. We demonstrate the localization and force-feedback features with a 7-DOF Franka arm on deformable tool use and a key insertion task for applications in home, medical, and food robotics. By increasing the reliability of interaction with common tools, this approach to object localization and force sensing can improve robot manipulation performance for delicate, high-precision tasks.

Modern localization techniques allow ground vehicle robots to determine their position with centimeter-level accuracy under nominal conditions, enabling them to utilize fixed maps to navigate their environments. However, when localization measurements become unavailable, the position accuracy will drop and uncertainty will increase. While research and development on localization estimation seeks to reduce the severity of these outages, the question of what actions a robot should take under high localization uncertainty is still unresolved, and can vary on a platform-by-platform and mission-by-mission basis. In this paper, we exploit localization uncertainty measures to adapt system control parameters in real time. Offline, we optimize non-linear activation functions whose control parameters and relevant weights are trained and learned using Evolutionary Algorithm (EA). Subsequently, in real time, we apply the optimized adaptation functions to the controller look-ahead distance and intermediate linear and angular velocity commands, which we identify as the most sensitive to localization error. Evolutionary runs are conducted in which a simulated target vehicle is tasked with following a randomly generated path while minimizing cross-track error, with time varying localization uncertainty added. These runs produce situation-dependent weights for parameters to the adaptation functions, which are transferred to the physical platform, a 1:5-scale autonomous vehicle. In simulation, our system was able to reduce cross-track error, which in certain cases exceeds 250 centimeters on non-adapted systems, to below 15 centimeters on average using EA-derived weights and parameters applied to our proposed adaptation system. Evaluation on the physical platform demonstrates that without the adaptation module in place, the platform is unable to successfully follow the path; with the adaptation module, the platform automatically adjusts its velocity and look-ahead distance to compensa... Show More

This paper presents a method to validate localization safety for a preplanned trajectory in a given environment. Localization safety is defined as integrity risk and quantified as the probability of an undetected localization failure. Integrity risk differs from previously used metrics in robotics in that it accounts for unmodeled faults and evaluates safety under the worst possible combination of faults. The methodology can be applied prior to mission execution and thus can be employed to evaluate the safety of potential trajectories. The work has been formulated for localization via smoothing, which differs from previously reported integrity monitoring methods that rely on Kalman filtering. Simulation and experimental results are analyzed to show that localization safety is effectively quantified.

We present a Lloyd-based navigation solution for robots that are required to move in a dynamic environment, where static obstacles (e.g, furnitures, parked cars) and unpredicted moving obstacles (e.g., humans, other robots) have to be detected and avoided on the fly. The algorithm can be computed in real-time and falls in the category of the reactive methods. Moreover, we propose an extension to the multi-agent case that deals with cohesion and cooperation between agents. The goodness of the method is proved through extensive simulations and, for the single agent navigation in human-shared environment, also with experiments on a unicycle-like robot.

Modular robots are defined as autonomous kinematic machines with variable morphology. They are composed of several thousands or even millions of modules which are able to coordinate in order to behave intelligently. Clustering the modules in modular robots has many benefits, including scalability, energy-efficiency, reducing communication delay and improving the self-configuration processes that focuses on finding a sequence of reconfiguration actions to convert robots from an initial configuration to a goal one. The main idea is to divide the nodes in an initial shape into some clusters based on the final goal shape in order to reduce the time complexity and enhance the self-reconfiguration tasks. In this paper, we propose a robust clustering approach based on a distributed density-cut graph algorithm to divide the networks into a pre-defined number of clusters based on the final goal shape. The result is an algorithm with linear complexity that scales to large modular robot systems. We implement and demonstrate our algorithm on a real Blinky Blocks system and evaluate it in simulation on networks of up to 30,000 modules.

The ability of legged systems to traverse highly- constrained environments depends by and large on the performance of their motion and balance controllers. This paper presents a controller that excels in a scenario that most state- of-the-art balance controllers have not yet addressed: line walking, or walking on nearly null support regions. Our approach uses a low-dimensional virtual model (2-DoF) to generate balancing actions through a previously derived four- term balance controller and transforms them to the robot through a derived kinematic mapping. The capabilities of this controller are tested in simulation, where we show the 90kg quadruped robot HyQ crossing a bridge of only 6 cm width (compared to its 4 cm diameter spherical foot), by balancing on two feet at any time while moving along a line. Additional simulations are carried to test the performance of the controller and the effect of external disturbances. Lastly, we present our preliminary experimental results showing HyQ balancing on two legs while being disturbed.

Communication is one of the most popular and efficient means of multi-robot coordination. Due to potential real-world constraints, such as limited bandwidth and contested scenarios, a communication strategy requiring to send, for example, all n bits of an environment representation might not be feasible in situations where the robots' data exchanges are frequent and large. To this end, we propose and implement lightweight, bandwidth-efficient, robot-to-robot communication protocols inspired by communication complexity results for data synchronization without exchanging the originally required n bits. We have tested our proposed approach both in simulation and with real robots. Simulation results show that the proposed method is computationally fast and enables the robots to synchronize the data (near) accurately while exchanging significantly smaller amounts of information (in the order of log n bits). Real-world experiments with two mobile robots show the practical feasibility of our proposed approach.