ICRA 2019 Papers

We propose a reconfigurable network for efficient inference dedicated to autonomous platforms equipped with multiple perception sensors. The size of the network for steering autonomous platforms grows proportionally to the number of installed sensors eventually preventing the usage of multiple sensors in real-time applications due to an inefficient inference. Our approach hinges on the observation that multiple sensors provide a large stream of data, where only a fraction of the data is relevant for the performed task at any given moment in time. The architecture of the reconfigurable network that we propose contains separate feature extractors, called experts, for each sensor. The decisive block of our model is the gating network, which online decides which sensor provides the data that is most relevant for driving. It then reconfigures the network by activating only the relevant expert corresponding to that sensor and deactivating the remaining ones. As a consequence, the model never extracts features from data that are irrelevant for driving. The gating network takes the data from all inputs and thus to avoid explosion of computation time and memory space it has to be realized as a small and shallow network. We verify our model on the unmanned ground vehicle (UGV) comprising of the 1/6 scale remote control truck equipped with three cameras. We demonstrate that the reconfigurable network correctly chooses experts in real-time allowing the reduction of computations cost for the whole model without deteriorating its performance.

This work addresses the problem of robot navigation under timed temporal specifications in workspaces cluttered with obstacles. We propose a hybrid control strategy that guarantees the accomplishment of a high-level specification expressed as a timed temporal logic formula, while preserving safety (i.e., obstacle avoidance) of the system. In particular, we utilize a motion controller that achieves safe navigation inside the workspace in predetermined time, thus allowing us to abstract the motion of the agent as a finite timed transition system among certain regions of interest. Next, we employ standard formal verification and convex optimization techniques to derive high-level timed plans that satisfy the agent's specifications. A simulation study illustrates and clarifies the proposed scheme.

The control of field robots in varying and uncertain terrain conditions presents a challenge for autonomous navigation. Online estimation of the wheel-terrain slip characteristics is essential for generating the accurate control predictions necessary for tracking trajectories in off-road environments. Receding horizon estimation (RHE) provides a powerful framework for constrained estimation, and when combined with receding horizon control (RHC), yields an adaptive optimisation-based control method. Presently, such methods assume slip to be constant over the estimation horizon, while our proposed structured blocking approach relaxes this assumption, resulting in improved state and parameter estimation. We demonstrate and compare the performance of this method in simulation, and propose an overlapping-block strategy to ameliorate some of the limitations encountered in applying noise-blocking in a receding horizon estimation and control (RHEC) context.

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the “Stoch”. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.

Needle insertion procedures can greatly benefit from robotic systems to improve their accuracy and success rate. However, a fully automated system is usually not desirable and the clinicians need to be included in the control loop. In this paper we present a teleoperation framework for beveledtip flexible needle steering that enables the user to directly and intuitively control the trajectory of the needle tip via a haptic interface. The 6 degrees of freedom of the needle base are used to perform several automatic safety and targeting tasks in addition to the one controlled by the user. Real-time visual feedback is provided by a 3D ultrasound probe and used to track the 3D location of the needle and of a spherical target. Several haptic force feedback are compared as well as two different levels of mix between automated and user-controlled tasks. A validation of the framework is conducted in gelatin phantom and a mean targeting accuracy of 2.5 mm is achieved. The results show that providing an adequate haptic guidance to the user can reduce the risks of damage to the tissues while still letting the surgeon in control of the tip trajectory.

In this paper, we propose a novel dense surfel mapping system that scales well in different environments with only CPU computation. Using a sparse SLAM system to estimate camera poses, the proposed mapping system can fuse intensity images and depth images into a globally consistent model. The system is carefully designed so that it can build from room-scale environments to urban-scale environments using depth images from RGB-D cameras, stereo cameras or even a monocular camera. First, superpixels extracted from both intensity and depth images are used to model surfels in the system. superpixel-based surfels make our method both runtime efficient and memory efficient. Second, surfels are further organized according to the pose graph of the SLAM system to achieve O(1) fusion time regardless of the scale of reconstructed models. Third, a fast map deformation using the optimized pose graph enables the map to achieve global consistency in real-time. The proposed surfel mapping system is compared with other state-of-the-art methods on synthetic datasets. The performances of urban-scale and room-scale reconstruction are demonstrated using the KITTI dataset [1] and autonomous aggressive flights, respectively. The code is available for the benefit of the community.

This paper describes a novel approach in human-robot interaction driven by ergonomics. With a clear focus on optimising ergonomics, the approach proposed here continuously observes a human user’s posture and by invoking appropriate cooperative robot movements, the user’s posture is, whenever required, brought back to an ergonomic optimum. Effectively, the new protocol optimises the human-robot relative position and orientation as a function of human ergonomics. An RGB-D camera is used to calculate and monitor human joint angles in real-time and to determine the current ergonomics state. A total of 6 main causes of low ergonomic states are identified, leading to 6 universal robot responses to allow the human to return to an optimal ergonomics state. The algorithmic framework identifies these 6 causes and controls the cooperating robot to always adapt the environment (e.g. change the pose of the workpiece) in a way that is ergonomically most comfortable for the interacting user. Hence, human-robot interaction is continuously re-evaluated optimizing ergonomics states. The approach is validated through an experimental study, based on established ergonomic methods and their adaptation for real-time application. The study confirms improved ergonomics using the new approach.

This paper presents planning and control algorithms for a multi-rotor with a suspended load. The suspended load cannot be controlled easily by the multi-rotor due to severe dynamic coupling between them. Difficulties are exacerbated by under-actuated, highly nonlinear nature of multi-rotor dynamics. Although many studies have been proposed to plan trajectories and control this system, there exist only a few reports on real-time trajectory generation. With this in mind, we propose a planning method which is capable of generating collision-free trajectories real-time and applicable to a high-dimensional nonlinear system. Using a differential flatness property, the system can be linearized entirely with elaborately chosen flat outputs. Convexification of non-convex constraints is carried out, and concave obstacle-avoidance constraints are converted to convex ones. After that, a convex optimization problem is solved to generate an optimal trajectory, but semi-feasible trajectory which considers only some parts of the initial state. We apply model predictive control with a sequential linear quadratic solver to compute a feasible collision-free trajectory and to control the system. Performance of the algorithm is validated by flight experiment.

We address the challenge of how a robot can adapt its actions to successfully manipulate objects it has not previously encountered. We introduce Real-time Multisensory Affordance-based Control (RMAC), which enables a robot to adapt existing affordance models using multisensory inputs. We show that using the combination of haptic, audio, and visual information with RMAC allows the robot to learn afforance models and adaptively manipulate two very different objects (drawer, lamp), in multiple novel configurations. Offline evaluations and real-time online evaluations show that RMAC allows the robot to accurately open different drawer configurations and turn-on novel lamps with an average accuracy of 75%.

This paper presents a new Model Predictive Control (MPC) framework for controlling various dynamic movements of a quadrupedal robot. System dynamics are represented by linearizing single rigid body dynamics in three-dimensional (3D) space. Our formulation linearizes rotation matrices without resorting to parameterizations like Euler angles and quaternions, avoiding issues of singularity and unwinding phenomenon, respectively. With a carefully chosen configuration error function, the MPC control law is transcribed into a Quadratic Program (QP) which can be solved efficiently in realtime. Our formulation can stabilize a wide range of periodic quadrupedal gaits and acrobatic maneuvers. We show various simulation as well as experimental results to validate our control strategy. Experiments prove the application of this framework with a custom QP solver could reach execution rates of 160 Hz on embedded platforms.

This work presents a model based traversability analysis method which employs a detailed vehicle model to perform real-time path planning in complex environments. The vehicle model represents the vehicle's wheels and chassis, allowing it to accurately predict the vehicles 3D pose, detailed contact information for each wheel and the occurrence of a chassis collision given a 2D pose on an elevation map. These predictions are weighted, depending on the safety requirements of the vehicle, to provide a scoring function for an A*-like search strategy. The proposed method is designed to run at frame rates of 30Hz on data from a RGB-D sensor to provide reactive planning of safe paths. For evaluation, two wheeled mobile robots in different simulated and real world environment setups were tested to show the reliability and performance of the proposed method.

Understanding the behaviors and intentions of humans are one of the main challenges autonomous ground vehicles still faced with. More specifically, when it comes to complex environments such as urban traffic scenes, inferring the intentions and actions of vulnerable road users such as pedestrians become even harder. In this paper, we address the problem of intent action prediction of pedestrians in urban traffic environments using only image sequences from a monocular RGB camera. We propose a real-time framework that can accurately detect, track and predict the intended actions of pedestrians based on a tracking-by-detection technique in conjunction with a novel spatio-temporal DenseNet model. We trained and evaluated our framework based on real data collected from urban traffic environments. Our framework has shown resilient and competitive results in comparison to other baseline approaches. Overall, we achieved an average precision score of 84.76% with real-time performance at 20 FPS.

Vehicle detection from short-range aerial image faces challenges including vehicle blocking, irrelevant object interference, motion blurring, color variation etc., leading to the difficulty to achieve high detection accuracy and real-time detection speed. In this paper, benefiting from the recent development in MobileNet family network engineering, we propose a compressed MobileNet which is not only internally resistant to the above listed challenges but also gains the best detection accuracy/speed tradeoff when comparing with the original MobileNet. In a nutshell, we reduce the bottleneck architecture number during the feature map downsampling stage but add more bottlenecks during the feature map plateau stage, neither extra FLOPs nor parameters are thus involved but reduced inference time and better accuracy are expected. We conduct experiment on our collected 5-k short-range aerial images, containing six vehicle categories: truck, car, bus, bicycle, motorcycle, crowded bicycles and crowded motorcycles. Our proposed compressed MobileNet achieves 110 FPS (GPU), 31 FPS (CPU) and 15 FPS (mobile phone), 1.2 times faster and 2% more accurate (mAP) than the original MobileNet.

Autonomous navigation through unknown environments is a challenging task that entails real-time localization, perception, planning, and control. UAVs with this capability have begun to emerge in the literature with advances in lightweight sensing and computing. Although the planning methodologies vary from platform to platform, many algorithms adopt a hierarchical planning architecture where a slow, low-fidelity global planner guides a fast, high-fidelity local planner. However, in unknown environments, this approach can lead to erratic or unstable behavior due to the interaction between the global planner, whose solution is changing constantly, and the local planner; a consequence of not capturing higher-order dynamics in the global plan. This work proposes a planning framework in which multi-fidelity models are used to reduce the discrepancy between the local and global planner. Our approach uses high-, medium-, and low-fidelity models to compose a path that captures higher-order dynamics while remaining computationally tractable. In addition, we address the interaction between a fast planner and a slower mapper by considering the sensor data not yet fused into the map during the collision check. This novel mapping and planning framework for agile flights is validated in simulation and hardware experiments, showing replanning times of 5-40 ms in cluttered environments.

Simultaneous Localization And Mapping (SLAM) is a fundamental problem in mobile robotics. While sparse point-based SLAM methods provide accurate camera localization, the generated maps lack semantic information. On the other hand, state of the art object detection methods provide rich information about entities present in the scene from a single image. This work incorporates a real-time deep-learned object detector to the monocular SLAM framework for representing generic objects as quadrics that permit detections to be seamlessly integrated while allowing the real-time performance. Finer reconstruction of an object, learned by a CNN network, is also incorporated and provides a shape prior for the quadric leading further refinement. To capture the structure of the scene, additional planar landmarks are detected by a CNN-based plane detector and modelled as independent landmarks in the map. Extensive experiments support our proposed inclusion of semantic objects and planar structures directly in the bundle-adjustment of SLAM - Semantic SLAM- that enriches the reconstructed map semantically, while significantly improving the camera localization.

This paper addresses the problem of generating quadcopter minimum snap trajectories for real time applications. Previous efforts addressed this problem by either employing a gradient descent method, or by greatly sacrificing optimality for faster solutions that are amenable for onboard implementation. In this work, outputs of the gradient descent method are used offline to train a supervised neural network. We show that the use of neural networks results typically in two orders of magnitude reduction in computational time. Our proposed approach can be used for warm-starting onboard implementable iterative methods with an “educated ” initial guess. This work is motivated by the application for human-machine interface in which a human provides desired trajectory through a smart-tablet interface, which has to be translated into a dynamically feasible trajectory for a quadcopter. The proposed solution is tested in thousands of different examples, demonstrating its effectiveness as a booster for minimum snap trajectory generation for quadcopters.

Deployment of deep learning models in robotics as sensory information extractors can be a daunting task to handle, even using generic GPU cards. Here, we address three of its most prominent hurdles, namely, i) the adaptation of a single model to perform multiple tasks at once (in this work, we consider depth estimation and semantic segmentation crucial for acquiring geometric and semantic understanding of the scene), while ii) doing it in real-time, and iii) using asymmetric datasets with uneven numbers of annotations per each modality. To overcome the first two issues, we adapt a recently proposed real-time semantic segmentation network, making changes to further reduce the number of floating point operations. To approach the third issue, we embrace a simple solution based on hard knowledge distillation under the assumption of having access to a powerful `teacher' network. We showcase how our system can be easily extended to handle more tasks, and more datasets, all at once, performing depth estimation and segmentation both indoors and outdoors with a single model. Quantitatively, we achieve results equivalent to (or better than) current state-of-the-art approaches with one forward pass costing just 13ms and 6.5 GFLOPs on 640×480 inputs. This efficiency allows us to directly incorporate the raw predictions of our network into the SemanticFusion framework [1] for dense 3D semantic reconstruction of the scene.

We present a real-time dense geometric mapping algorithm for large-scale environments. Unlike existing methods which use pinhole cameras, our implementation is based on fisheye cameras whose large field of view benefits various computer vision applications for self-driving vehicles such as visual-inertial odometry, visual localization, and object detection. Our algorithm runs on in-vehicle PCs at approximately 15 Hz, enabling vision-only 3D scene perception for self-driving vehicles. For each synchronized set of images captured by multiple cameras, we first compute a depth map for a reference camera using plane-sweeping stereo. To maintain both accuracy and efficiency, while accounting for the fact that fisheye images have a lower angular resolution, we recover the depths using multiple image resolutions. We adopt the fast object detection framework, YOLOv3, to remove potentially dynamic objects. At the end of the pipeline, we fuse the fisheye depth images into the truncated signed distance function (TSDF) volume to obtain a 3D map. We evaluate our method on large-scale urban datasets, and results show that our method works well in complex dynamic environments.

We present an approach to depth estimation that fuses information from a stereo pair with sparse range measurements derived from a LIDAR sensor or a range camera. The goal of this work is to exploit the complementary strengths of the two sensor modalities, the accurate but sparse range measurements and the ambiguous but dense stereo information. These two sources are effectively and efficiently fused by combining ideas from anisotropic diffusion and semi-global matching.We evaluate our approach on the KITTI 2015 and Middlebury 2014 datasets, using randomly sampled ground truth range measurements as our sparse depth input. We achieve significant performance improvements with a small fraction of range measurements on both datasets. We also provide qualitative results from our platform using the PMDTec Monstar sensor. Our entire pipeline runs on an NVIDIA TX-2 platform at 5Hz on 1280×1024 stereo images with 128 disparity levels.