IROS 2015 Papers

In this work, we present an MAV system that is able to relocalize itself, create consistent maps and plan paths in full 3D in previously unknown environments. This is solely based on vision and IMU measurements with all components running onboard and in real-time. We use visual-inertial odometry to keep the MAV airborne safely locally, as well as for exploration of the environment based on high-level input by an operator. A globally consistent map is constructed in the background, which is then used to correct for drift of the visual odometry algorithm. This map serves as an input to our proposed global planner, which finds dynamic 3D paths to any previously visited place in the map, without the use of teach and repeat algorithms. In contrast to previous work, all components are executed onboard and in real-time without any prior knowledge of the environment.

Localization is essential for robots to operate autonomously, especially for extended periods of time, when estimator drift tends to destroy alignment to any global map. Though there has been extensive work in vision-based localization in recent years, including several systems that show real-time performance, none have been demonstrated running entirely on-board in closed loop on robotic platforms. We propose a fast, real-time localization system that keeps the existing local visual-inertial odometry frame consistent for controllers and collision avoidance, while correcting drift and alignment to a global coordinate frame. We demonstrate our localization system entirely on-board an aerial and ground robot, showing a collaboration experiment where both robots are able to localize against the same map accurately enough to allow the multicopter to land on top of the ground robot. We also perform extensive evaluations for the proposed closed-loop system on ground-truth datasets from MAV flight in an industrial setting.

This paper presents the details and experimental results from an implementation of real-time trajectory generation and parameter estimation of a dynamic model using the Baxter Research Robot from Rethink Robotics. Trajectory generation is based on the maximization of Fisher information in real-time and closed-loop using a form of Sequential Action Control. On-line estimation is performed with a least-squares estimator employing a nonlinear state observer model computed with trep, a dynamics simulation package. Baxter is tasked with estimating the length of a string connected to a load suspended from the gripper with a load cell providing the single source of feedback to the estimator. Several trials are presented with varying initial estimates showing convergence to the actual length within a 6 second time-frame.

This paper presents a sampling-based planning method considering motion uncertainty to generate more human-like driving paths for automated vehicles. Given information in the form of a small set of rules and driving heuristics the planning system optimizes trajectories in a seven dimensional state space. In a post-processing step a set of candidates is evaluated considering the uncertainty of the vehicles motion executing the given trajectory using a Linear-Quadratic Gaussian (LQG). This addresses the problem of indecisive planning behavior in case the optimal solution is unlikely to be followed precisely. The results of our experiments show that the mobile graphics processing unit (GPU) technology can be used as an enabler for real-time applications of computationally expensive planning approaches.

This paper presents a method to track in real-time a 3D textureless object which undergoes large deformations such as elastic ones, and rigid motions, using the point cloud data provided by an RGB-D sensor. This solution is expected to be useful for enhanced manipulation of humanoid robotic systems. Our framework relies on a prior visual segmentation of the object in the image. The segmented point cloud is registered first in a rigid manner and then by non-rigidly fitting the mesh, based on the Finite Element Method to model elasticity, and on geometrical point-to-point correspondences to compute external forces exerted on the mesh. The real-time performance of the system is demonstrated on synthetic and real data involving challenging deformations and motions.

We proposed ReFrESH in our previous publication. It is a self-adaptive infrastructure aimed at managing the performance of multi-robot systems through dynamically diagnosing and maintaining unexpected issues of modules. To integrate ReFrESH and robotic application-level software more conveniently, it is necessary to develop a module design framework to support implementation of self-adaptive real-time software. To this end, based on the port-based object abstraction and port-automation theory, we propose the Extended Port-Based Object (E-PBO). E-PBO has two main advantages: (1) it builds the basis of a programming model to provide specific, yet flexible, guidelines to robotics application engineers for creating and integrating software modules; (2) it forms the basis of a self-adaption model to provide specific methods for evaluating the running task configuration and estimating the new but non-running task configuration (if required) without interfering with the running configuration. E-PBO has been incorporated into the Port-Based Object Real-Time Operating System (PBO/RT) and applied to a visual servoing robotic application, which is demonstrated here.

With today's advanced 3D scanner technology, huge amounts of point cloud data can be generated in short amounts of time. Data compression is thus necessary for storage and especially for transmission, e.g., via wireless networks. While previous approaches delivered good compression ratios and interesting theoretical insights, they are either computationally expensive or do not support incrementally acquired data and locally decompressing the data, two requirements we found necessary in many applications. We present a compression approach that is efficient in storage requirements as well as in computational cost, as it can compress and decompress point cloud data in real-time. Furthermore, it is capable of compressing incrementally acquired data, local decompression and of decompressing a subsampled representation of the original data. Our method is based on local 2D parameterizations of surface point cloud data, for which we describe an efficient approach. We suggest the usage of standard image compression techniques for the compression of local details. While exhibiting state-of-the-art compression ratios, our approach remains easy to implement. In our evaluation, we compare our approach to previous ones and discuss the choice of parameters. Due to our algorithm's efficiency, we consider it as a reference concerning speed and compression rates.

In this paper we present a step-planner embedded in a framework which enables a humanoid robot to navigate among obstacles, exploiting its overall capacities. The system allows the robot to react to changes of user input or changes in a dynamic environment in real-time while walking at reasonable speeds. The proposed method relies neither on external sensors nor on color coding or textured surfaces. The key idea is to use a fast collision model based on swept-sphere-volumes (SSVs) for real-time generation of collision-free footsteps and whole-body trajectories. Using a SSV-based 3D approximation in all control modules enables the robot to avoid collisions with itself and the environment. Obstacles are detected with an on-board RGB-D sensor while the robot navigates through an environment which is not known in advance. A step-planner reacts to high-level user commands like desired velocity and direction or changes in the environment within less than a step. Instead of investigating only the footholds an articulated 3D approximation of the lower leg and the foot is considered to find feasible and optimal footstep locations. Additionally, it provides an initial solution for the swing-foot movement. Finally, collision-free swing-foot trajectories are created in real-time in the feedback control layer using all the foot's degrees of freedom. We validated this approach in experiments with our robot Lola.

Depalletizing is a challenging task for manipulation robots. Key to successful application are not only robustness of the approach, but also achievable cycle times in order to keep up with the rest of the process. In this paper, we propose a system for depalletizing and a complete pipeline for detecting and localizing objects as well as verifying that the found object does not deviate from the known object model, e.g., if it is not the object to pick. In order to achieve high robustness (e.g., with respect to different lighting conditions) and generality with respect to the objects to pick, our approach is based on multi-resolution surfel models. All components (both software and hardware) allow operation at high frame rates and, thus, allow for low cycle times. In experiments, we demonstrate depalletizing of automotive and other prefabricated parts with both high reliability (w.r.t. success rates) and efficiency (w.r.t. low cycle times).

Venipuncture is the most common invasive medical procedure performed in the United States and the number one cause of hospital injury. Failure rates are particularly high in pediatric and elderly patients, whose veins tend to deform, move, or roll as the needle is introduced. To improve venipuncture accuracy in challenging patient populations, we have developed a portable device that autonomously servos a needle into a suitable vein under image guidance. The device operates in real time, combining near-infrared and ultrasound imaging, computer vision software, and a 9 degrees-of-freedom robot that servos the needle. In this paper, we present the kinematic and mechanical design of the latest generation robot. We then investigate in silico and in vitro the mechanics of vessel rolling and deformation in response to needle insertions performed by the robot. Finally, we demonstrate how the robot can make real-time adjustments under ultrasound image guidance to compensate for subtle vessel motions during venipuncture.

Drift of the rotation estimate is a well known problem in visual odometry systems as it is the main source of positioning inaccuracy. We propose three novel algorithms to estimate the full 3D rotation to the surrounding Manhattan World (MW) in as short as 20 ms using surface-normals derived from the depth channel of a RGB-D camera. Importantly, this rotation estimate acts as a structure compass which can be used to estimate the bias of an odometry system, such as an inertial measurement unit (IMU), and thus remove its angular drift. We evaluate the run-time as well as the accuracy of the proposed algorithms on groundtruth data. They achieve zerodrift rotation estimation with RMSEs below 3.4° by themselves and below 2.8° when integrated with an IMU in a standard extended Kalman filter (EKF). Additional qualitative results show the accuracy in a large scale indoor environment as well as the ability to handle fast motion. Selected segmentations of scenes from the NYU depth dataset demonstrate the robustness of the inference algorithms to clutter and hint at the usefulness of the segmentation for further processing.

Robots that cooperate and interact with humans require the capacity to detect and track people, analyze their behavior and understand human social relations and rules. A key piece of information for such tasks are human attributes like gender, age, hair or clothing. In this paper, we address the problem of recognizing such attributes in RGB-D data from varying full-body views. To this end, we extend a recent tessellation boosting approach which learns the best selection, location and scale of a set of simple RGB-D features. The approach outperforms the original approach and a HOG baseline for five human attributes including gender, has long hair, has long trousers, has long sleeves and has jacket. Experiments on a multi-perspective RGB-D dataset with full-body views of over a hundred different persons show that the method is able to robustly recognize multiple attributes across different view directions and distances to the sensor with accuracies up to 90%. Our methods runs in real-time, achieving a classification rate of around 300 Hz for a single attribute.

Nanoscale vibration, a critical nanomechanical property of cell membranes/walls, is a crucial aspect of cell physiology. However, limitations of current nanoscale vibration detecting methods remain the major obstacle for scientific study and cell vibration experiments. Due to the absence of effective method of feeble nanoscale vibration detecting, most sorts of quantitative and dynamic cell vibrations cannot be observed. Therefore, a real-time tracking detection method is vital for the study of cell physiology. In this paper, a real-time tracking detection of nanoscale vibrations based on sweep frequency (SF) - amplitude modulation (AM) method using cantilever sweep frequency as a carrier frequency was proposed. Furthermore, the process of tip-sample vibration coupling is analyzed by using the idea of amplitude modulation model. The nanoscle vibration detecting experiments were carried out on a piezoceramic disc, which can mimic cell vibrations. The experiment results show that the SF-AM AFM real-time vibration tracking and detecting approach can accurately detect and track feeble sample vibration within few nanometers amplitude.

In certain cases analytical derivation of physics-based models of robots is difficult or even impossible. A potential workaround is the approximation of robot models from sensor data-streams employing machine learning approaches. In this paper, the inverse dynamics models are learned by employing a novel real-time deep learning algorithm. The algorithm exploits the methods of self-organized learning, reservoir computing and Bayesian inference. It is evaluated and compared to other state of the art algorithms in terms of generalization ability, convergence and adaptability using five datasets gathered from four robots. Results show that the proposed algorithm can adapt to real-time changes of the inverse dynamics model significantly better than the other state of the art algorithms.

In order for robots to work alongside humans in a range of domains, they will need to operate with a variety of social dynamics that each context will require. This paper builds on previous work with a parameterized turn-taking model, CADENCE, in which different parameter settings resulted in different social dynamics. In contrast to the static parameter settings of previous work, we now investigate the problem of changing these turn-taking parameter sets dynamically within a single interaction session. This ability is necessary for successful peer-to-peer collaborations, in which balance of control between leading and following must be maintained. We present our dynamic switching approach and an experiment with 15 participants. Our results confirm that it is possible to achieve the same changes in social dynamics within a single interaction session that were previously seen only between independent sessions of different parameter settings. Moreover, we show that such a change in social dynamics is contingent upon changing parameters at socially appropriate turn boundaries.

This work proposes a real-time segmentation method for 3D point clouds obtained via Simultaneous Localization And Mapping (SLAM). The proposed method incrementally merges segments obtained from each input depth image in a unified global model using a SLAM framework. Differently from all other approaches, our method is able to yield segmentation of scenes reconstructed from multiple views in real-time, with a complexity that does not depend on the size of the global model. At the same time, it is also general, as it can be deployed with any frame-wise segmentation approach as well as any SLAM algorithm. We validate our proposal by a comparison with the state of the art in terms of computational efficiency and accuracy on a benchmark dataset, as well as by showing how our method can enable real-time segmentation from reconstructions of diverse real indoor environments.

This paper presents a novel point-cloud descriptor for robust and real-time tracking of multiple objects without any object knowledge. Following with the framework of incremental model-free multiple object tracking from our previous work [5][7][6], 6 DoF pose of each object is firstly estimated with input point-cloud data which is then segmented according to the estimated objects, and incremental model of each object is updated from the segmented point-clouds. Here, we propose Joint Color-Spatial Descriptor (JCSD) to enhance the robustness of the pose hypothesis evaluation to the point-cloud scene in the particle filtering framework. The method outperforms widely used point-to-point comparison methods, especially in the partially occluded scene, which is frequently happened in the dynamic object manipulation cases. By means of the robust descriptor, we achieved unsupervised multiple object segmentation accuracy higher than 99%. The model-free multiple object tracking was implemented by using a particle filtering with JCSD as a likelihood function. The robust likelihood function is implemented with GPU, thus facilitating real-time tracking of multiple objects.

Kinematic models of concentric tube robots have matured from considering only tube bending to considering tube twisting as well as external loading. While these models have been demonstrated to approximate actual behavior, modeling error can be significant for medical applications that often call for positioning accuracy of 1-2mm. As an alternative to moving to more complex models, this paper proposes using sensing to adaptively update model parameters during robot operation. Advantages of this method are that the model is constantly tuning itself to provide high accuracy in the region of the workspace where it is currently operating. It also adapts automatically to changes in robot shape and compliance associated with the insertion and removal of tools through its lumen. As an initial exploration of this approach, a recursive on-line estimator is proposed and evaluated experimentally.

Autonomous navigation of micro aerial vehicles (MAVs) in a-priori unknown environments is one of the most challenging problems in robotics. First, a MAV has to incrementally build a 3D geometric map from raw sensor data. Then, based on the mapping information, the path planner has to search for a cost-optimal trajectory to the goal in real-time. It is common practice to discretize the search space into a state lattice; by doing so, we reduce the path planning problem with differential constraints to a graph search problem that is easier to solve. However, a regular 3D state lattice requires a large amount of memory while graph search in a regular 3D state lattice incorporating numerous states is computationally intensive. In this paper, we introduce a novel path planning algorithm which extends the concept of a regular state lattice to an octree-based state lattice, and searches for an optimal trajectory in the octree-partitioned search space. Our octree-based state lattice representation discretizes large swathes of free space into few symbolic octants, and thus, encodes a significantly fewer number of states. As a result, memory consumption is kept to a minimum, and at the same time, graph search is made more efficient. Simulation experiments demonstrate the efficiency of path planning with an octree-based state lattice, and further field trials prove the viability of this path planning algorithm.