IROS 2021 Papers

Automation of excavation tasks requires real-time trajectory planning satisfying various constraints. To guarantee both constraint feasibility and real-time trajectory re-plannability, we present an integrated framework for real-time optimization-based trajectory planning of a hydraulic excavator. The proposed framework is composed of two main modules: a global planner and a real-time local planner. The global planner computes the entire global trajectory considering excavation volume and energy minimization while the local counterpart tracks the global trajectory in a receding horizon manner, satisfying dynamic feasibility, physical constraints, and disturbance-awareness. We validate the proposed planning algorithm in a simulation environment where two types of operations are conducted in the presence of emulated disturbance from hydraulic friction and soil-bucket interaction: shallow and deep excavation. The optimized global trajectories are obtained in an order of a second, which is tracked by the local planner at faster than 30 Hz. To the best of our knowledge, this work presents the first real-time motion planning framework that satisfies constraints of a hydraulic excavator, such as force/torque, power, cylinder displacement, and flow rate limits.

Estimating a scene’s depth to achieve collision avoidance against moving pedestrians is a crucial and fundamental problem in the robotic field. This paper proposes a novel, low complexity network architecture for fast and accurate human depth estimation and segmentation in indoor environments, aiming to applications for resource-constrained platforms (including battery-powered aerial, micro-aerial, and ground vehicles) with a monocular camera being the primary perception module. Following the encoder-decoder structure, the proposed framework consists of two branches, one for depth prediction and another for semantic segmentation. Moreover, network structure optimization is employed to improve its forward inference speed. Exhaustive experiments on three self-generated datasets prove our pipeline’s capability to execute in real-time, achieving higher frame rates than contemporary state-of-the-art frameworks (114.6 frames per second on an NVIDIA Jetson Nano GPU with TensorRT) while maintaining comparable accuracy.

Hamilton-Jacobi (HJ) reachability analysis is a powerful technique used to verify the safety of autonomous systems. HJ reachability is ideal for analysing nonlinear systems with disturbances and flexible set representations. A drawback to this approach is that it suffers from the curse of dimensionality, which prevents real-time deployment on safety-critical systems. In this paper, we show that a customized hardware design on an Field Programmable Gate Array (FPGA) could accelerate 4D grid-based HJ reachability analysis up to 14 times compared to an optimized implementation and 103 times compared to state-of-the-art MATLAB toolboxes on a 16-thread CPU. Because of this, we are able to achieve guaranteed real- time collision avoidance in dynamic environments that abruptly change with a 4D car model by re-solving the HJ partial differential equation (PDE) at a frequency of 4Hz on an FPGA. Our design can overcome the complex data access pattern while taking advantage of the parallel nature of the computations for solving the HJ PDE. The low latency of our computation is consistent, which is crucial for safety-critical systems. The methodology presented here is without loss of generality: it can potentially be applied to different systems dynamics, and more- over, leveraged for higher dimensional systems. We validate our approach in real world collision avoidance experiments with a robot car in a changing environment. We also provide the code of our hardware design and an AWS AFI image.

Reactive and safe agent modellings are important for nowadays traffic simulator designs and safe planning applications. In this work, we proposed a reactive agent model which can ensure safety without comprising the original purposes, by learning only high-level decisions from expert data and a low level decentralized controller guided by the jointly learned decentralized barrier certificates. Empirical results show that our learned road user simulation models can achieve a significant improvement in safety comparing to state-of-the-art imitation learning and pure control-based methods, while being similar to human agents by having smaller error to the expert data. Moreover, our learned reactive agents are shown to generalize better to unseen traffic conditions, and react better to other road users and therefore can help understand challenging planning problems pragmatically.

A new strategy is proposed for scheduling Visual Odometry (VO) measurements for wheeled ground vehicles. Rather than having a fixed interval or distance between image acquisitions, we propose to trigger VO based on covariances from an Adaptive Extended Kalman Filter. The adopted model uses process noise to drive wheel slip estimation, which, when correctly identified, can be used with Wheel Odometry to provide frequent position estimates. When more dynamic terrain is detected, more VO measurements are scheduled to maintain localization accuracy. On the other hand, when the terrain is stable, VO usage is limited. The system is validated in a simple one-dimensional case using data captured during field trials using a representative rover. The results are promising as trajectories that were subjected to large errors are corrected.

Zero-shot execution of unseen robotic tasks is important to allowing robots to perform a wide variety of tasks in human environments, but collecting the amounts of data necessary to train end-to-end policies in the real-world is often infeasible. We describe an approach for sim-to-real training that can accomplish unseen robotic tasks using models learned in simulation to ground components of a simple task planner. We learn a library of parameterized skills, along with a set of predicates-based preconditions and termination conditions, entirely in simulation. We explore a block-stacking task because it has a clear structure, where multiple skills must be chained together, but our methods are applicable to a wide range of other problems and domains, and can transfer from simulation to the real-world with no fine tuning. The system is able to recognize failures and accomplish long-horizon tasks from perceptual input, which is critical for real-world execution. We evaluate our proposed approach in both simulation and in the real-world, showing an increase in success rate from 91.6% to 98% in simulation and from 10% to 80% success rate in the real-world as compared with naive baselines. For experiment videos including both real-world and simulation, see: https://www.youtube.com/playlist?list=PL-oD0xHUngeLfQmpngYkGFZarstfPOXqX

This paper presents a reactive control framework for robotic bipedal running over random discrete terrain using a shift-invariant funnel library that can be composed into motion plans. The main contribution of the paper is the formalization of the funnel library and the introduction of a receding-horizon reactive planner that utilizes this funnel library. The proposed controller generates a robust motion plan on the fly, without the need for predefined footholds or center of mass trajectory, via sequential-composition of pre-computed funnels. The proposed method utilizes three layers of robustness: high-level reactive planning, low-level robust control, and funnel-based, pre-computed, performance guarantees. For clarity, we provide a detailed example, using the 5 degrees-of-freedom sagittal biped model. We demonstrate in simulation the ability of the framework to overcome uncertainty in sensing and model parameters.

Scene text detection plays an important role on vision-based robot navigation to many potential landmarks such as nameplates, information signs, floor button in the elevators. Recently, scene text detection with segmentation-based methods has been receiving more and more attention. The segmentation results can be used to efficiently predict scene text of various shapes, such as irregular text in most scene text images. However, two kinds of texts remain unsolved: 1) tiny and 2) blurry instances. Moreover, the annotations for tiny/blurry texts are usually ignored during training, while tiny/blurry texts can still offer visual auxiliaries for robots to understand the world. Therefore, in this paper, we propose a new approach to effectively detect both clear and blurry texts. Specifically, we propose a re-attention module without increasing the learnable parameters, which first predicts the region of texts as the candidate region and leverages the same network to detect the candidate region again for reducing the required memory. Moreover, to avoid the errors from the first detection propagating to the re-attended area, we propose a new fusion module that learns to integrate the results of the re-attended regions and the first prediction. Experimental results manifest that the proposed method outperforms state-of-the-art methods on four challenging datasets.

The efficient evaluation the dynamic stability of legged robots on non-coplanar terrains is important when developing motion planning and control policies. The inference time of this measure has a strong influence on how fast a robot can react to unexpected events, plan its future footsteps or its body trajectory. Existing approaches suitable for real-time decision making are either limited to flat ground or to quasi-static locomotion. Furthermore, joint-space feasibility constraints are usually not considered in receding-horizon planning as their high dimensionality prohibits this. In this paper we propose the usage of a stability criterion for dynamic locomotion on rough terrain based on the Feasible Region (FR) and the Instantaneous Capture Point (ICP) and we leverage a Neural Network (NN) to quickly estimate it. We show that our network achieves satisfactory accuracy with respect to its analytical counterpart with a speed up of three orders-of-magnitude. It also enables the evaluation of the stability margin's gradient. We demonstrate this learned stability margin in two diverse applications - Reinforcement Learning (RL) and nonlinear Trajectory Optimization (TO) for legged robots. We demonstrate on a full-sized quadruped robot that the network enables the computation of physically-realizable Center of Mass (CoM) trajectories and foothold locations satisfying friction constraints and joint-torque limits in a receding-horizon fashion and on non-coplanar terrains.

This paper addresses the visibility-based pursuit-evasion problem where a team of pursuer robots operating in a two-dimensional polygonal space seek to establish visibility of an arbitrarily fast evader. This is a computationally challenging task for which the best known complete algorithm takes time doubly exponential in the number of robots. However, recent advances that utilize sampling-based methods have shown progress in generating feasible solutions. An aspect of this problem that has yet to be explored concerns how to ensure that the robots can recover from catastrophic failures which leave one or more robots unexpectedly incapable of continuing to contribute to the pursuit of the evader. To address this issue, we propose an algorithm that can rapidly recover from catastrophic failures. When such failures occur, a replanning occurs, leveraging both the information retained from the previous iteration and the partial progress of the search completed before the failure to generate a new motion strategy for the reduced team of pursuers. We describe an implementation of this algorithm and provide quantitative results that show that the proposed method is able to recover from robot failures more rapidly than a baseline approach that plans from scratch.

In this paper we explore the use of block coordinate descent (BCD) to optimize the centroidal momentum dynamics for dynamically consistent multi-contact behaviors. The centroidal dynamics have recently received a large amount of attention in order to create physically realizable motions for robots with hands and feet while being computationally more tractable than full rigid body dynamics models. Our contribution lies in exploiting the structure of the dynamics in order to simplify the original non-convex problem into two convex subproblems. We iterate between these two subproblems for a set number of iterations or until a consensus is reached. We explore the properties of the proposed optimization method for the centroidal dynamics and verify in simulation that motions generated by our approach can be tracked by the quadruped Solo12. In addition, we compare our method to a recently proposed convexification using a sequence of convex relaxations as well as a more standard interior point method used in the off-the-shelf solver IPOPT to show that our approach finds similar, if not better, trajectories (in terms of cost), and is more than four times faster than both approaches. Finally, compared to previous approaches, we note its practicality due to the convex nature of each subproblem which allows our method to be used with any off-the-shelf quadratic programming solver.

This work is dedicated to simultaneous continuous-time trajectory estimation and mapping based on Gaussian Processes (GP). State-of-the-art GP-based models for Simultaneous Localization and Mapping (SLAM) are computationally efficient but can only be used with a restricted class of kernel functions. This paper provides the algorithm based on GP with Random Fourier Features (RFF) approximation for SLAM without any constraints. The advantages of RFF for continuous-time SLAM are that we can consider a broader class of kernels and, at the same time, maintain computational complexity at reasonably low level by operating in the Fourier space of features. The accuracy-speed trade-off can be controlled by the number of features. Our experimental results on synthetic and real-world benchmarks demonstrate the cases in which our approach provides better results compared to the current state-of-the-art.

In this paper, we propose to fuse radar measurements with Visual Inertial Odometry (RVIO) or Thermal Inertial Odometry (RTIO). FMCW radar sensor data enables to estimate the 3D ego velocity independent of the visual conditions. Fusion with VIO or TIO heavily improves the robustness in challenging conditions such as darkness, direct sunlight or fog.Specifically, we propose RRxIO: An extension to the state of the art filter based VIO framework Robust Visual Inertial Odometry (ROVIO) to fuse radar sensor data. Due to the drift free 3D radar ego velocity estimates, scale errors are reduced and only a small number of features needs to be tracked to achieve good results. This yields faster runtimes while outperforming state of the art VIO frameworks regarding accuracy. RVIO is able to bridge phases of degraded or even no visual features resulting in a low cost system even for poor visual conditions. RTIO is robust even in environments with small temperature gradients and bridges phases of no thermal images caused by Non-Uniformity Corrections (NUCs).We evaluated our system with various experiments in different environments and visual conditions. Comparison to other state of the art dual domain approaches including radar-inertial, visual-inertial and thermal-inertial proves superior performance of our approach regarding accuracy and processing time.

Awareness of its surroundings is a crucial capability for a robot meant to be working alongside other robots or human operators. When considering safety norms and modalities, in particular the Speed and Separation Monitoring (SSM), proper proximity information can make the difference in the overall efficiency of a use case, for example avoiding unnecessary penalizations in the cycle-time. This paper presents a method to exploit the proximity perception capabilities of radar sensors to construct a continuous speed control algorithm for a UR10 robot. With respect to standard implementations of the SSM in industrial and collaborative environments, the proposed speed control is enhanced by the addition of direct human’s velocity measurement, full direction of travel and target classification. The results are evalauted according to the SSM metrics for safety and productivity, showing an overall increase in efficiency while still maintaining safety level requirements.

Feature extraction plays an important role in visual localization. Unreliable features on dynamic objects or repetitive regions will interfere with feature matching and challenge indoor localization greatly. To address the problem, we propose a novel network, RaP-Net, to simultaneously predict region-wise invariability and point-wise reliability, and then extract features by considering both of them. We also introduce a new dataset, named OpenLORIS-Location, to train the proposed network. The dataset contains 1553 images from 93 indoor locations. Various appearance changes between images of the same location are included and can help the model to learn the invariability in typical indoor scenes. Experimental results show that the proposed RaP-Net trained with OpenLORIS-Location dataset achieves excellent performance in the feature matching task and significantly outperforms state-of-the-arts feature algorithms in indoor localization. The RaPNet code and dataset are available at https://github.com/ivipsourcecode/RaP-Net.

Robust real-time detection and motion forecasting of traffic participants is necessary for autonomous vehicles to safely navigate urban environments. In this paper, we present RV-FuseNet, a novel end-to-end approach for joint detection and trajectory estimation directly from time-series LiDAR data. Instead of the widely used bird’s eye view (BEV) representation, we utilize the native range view (RV) representation of LiDAR data. The RV preserves the full resolution of the sensor by avoiding the voxelization used in the BEV. Furthermore, RV can be processed efficiently due to its compactness. Previous approaches project time-series data to a common viewpoint for temporal fusion, and often this viewpoint is different from where it was captured. This is sufficient for BEV methods, but for RV methods, this can lead to loss of information and data distortion which has an adverse impact on performance. To address this challenge we propose a simple yet effective novel architecture, Incremental Fusion, that minimizes the information loss by sequentially projecting each RV sweep into the viewpoint of the next sweep in time. We show that our approach significantly improves motion forecasting performance over the existing state-of-the-art. Furthermore, we demonstrate that our sequential fusion approach is superior to alternative RV based fusion methods on multiple datasets.

Recent data-driven approaches to visual servoing have shown improved performances over classical methods due to precise feature matching and depth estimation. Some recent servoing approaches use a model predictive control (MPC) framework which generalise well to novel environments and are capable of incorporating dynamic constraints, but are computationally intractable in real-time, making it difficult to deploy in real-world scenarios. On the contrary, single-step methods optimise greedily and achieve high servoing rates, but lack the benefits of the MPC multi-step ahead formulation. In this paper, we make the best of both worlds and propose a lightweight visual servoing MPC framework which generates optimal control near real-time at a frequency of 10.52 Hz. This work utilises the differential cross-entropy sampling method for quick and effective control generation along with a lightweight neural network, significantly improving the servoing frequency. We also propose a flow depth normalisation layer which ameliorates the issue of inferior predictions of two view depth from the flow network. We conduct extensive experimentation on the Habitat simulator and show a notable decrease in servoing time in comparison with other approaches that optimise over a time horizon. We achieve the right balance between time and performance for visual servoing in six degrees of freedom (6DoF), while retaining the advantageous MPC formulation. Our code and dataset are publicly available†.

Hyper-redundant manipulators with slender body and high dexterity are widely applied for operations in confined spaces. Among the motion planning methods for these operations, the follow-the-leader motion controller is generally developed to avoid the obstacles, while the path trajectories are usually given. In this paper, we present an autonomous motion planner with a specialized rapidly exploring random tree (Sp-RRT) approach for follow-the-leader motion of hyper-redundant manipulators. Starting from the target pose in the workspace, the exploring tree can expand to multiple entrances while guaranteeing the final pose of the manipulator’s end-effector. Meanwhile, the dexterity of hyper-redundant manipulators (even with different segments) can be utilized sufficiently with customized expanding parameters. Simulation results compared with existing methods are conducted to demonstrate the aforementioned characteristics and effectiveness. For further validation, we experimentally verify the development with our custom-built hyper-redundant manipulator to realize the generated path with follow-the-leader motion.

Modern visual-inertial navigation systems (VINS) are faced with a critical challenge in real-world deployment: they need to operate reliably and robustly in highly dynamic environments. Current best solutions merely filter dynamic objects as outliers based on the semantics of the object category. Such an approach does not scale as it requires semantic classifiers to encompass all possibly-moving object classes; this is hard to define, let alone deploy. On the other hand, many realworld environments exhibit strong structural regularities in the form of planes such as walls and ground surfaces, which are also crucially static. We present RP-VIO, a monocular visual-inertial odometry system that leverages the simple geometry of these planes for improved robustness and accuracy in challenging dynamic environments. Since existing datasets have a limited number of dynamic elements, we also present a highly-dynamic, photorealistic synthetic dataset for a more effective evaluation of the capabilities of modern VINS systems. We evaluate our approach on this dataset, and three diverse sequences from standard datasets including two real-world dynamic sequences and show a significant improvement in robustness and accuracy over a state-of-the-art monocular visual-inertial odometry system. We also show in simulation an improvement over a simple dynamic-features masking approach. Our code and dataset are publicly available†.