IROS 2019 Papers

In this research we focus on developing a reinforcement learning system for a challenging task: autonomous control of a real-sized boat, with difficulties arising from large uncertainties in the challenging ocean environment and the extremely high cost of exploring and sampling with a real boat. To this end, we explore a novel Gaussian processes (GP) based reinforcement learning approach that combines sample-efficient model-based reinforcement learning and model predictive control (MPC). Our approach, sample-efficient probabilistic model predictive control (SPMPC), iteratively learns a Gaussian process dynamics model and uses it to efficiently update control signals within the MPC closed control loop. A system using SPMPC is built to efficiently learn an autopilot task. After investigating its performance in a simulation modeled upon real boat driving data, the proposed system successfully learns to drive a real-sized boat equipped with a single engine and sensors measuring GPS, speed, direction, and wind in an autopilot task without human demonstration.

In the context of mobile gaze tracking techniques, a 3D gaze point can be calculated as the middle point between two 3D visual axes. To infer gaze directions and eyeball positions, a nonlinear optimization problem is typically formulated to minimize the angular disparities between the training gaze directions and prediction ones. Nonetheless, the experimental results reported by some previous works show that this kind of approaches are very likely to yield large prediction errors hence considered less useful for human-machine interactions. In this study, we aim to address this widespread issue in three aspects. At first, instead of using a global regression model, a simple local polynomial model is proposed to back-project a pupil center onto its corresponding visual axis. Based on the Leave-One-Out cross-validation criterion, the partition structure is automatically learned in the process of resolving a homography-like relationship. Secondly, a good starting point for nonlinear-optimization is obtained by the image eyeball center, which can be estimated by systematic parallax errors. Meanwhile, it is necessary to add the suitable constraints for 3D eye positions. Otherwise, the optimization may end up with trivial solutions, i.e., faraway eye positions. Thirdly, we explore a strategy for designing the spatial distribution of calibration points in a principled manner. The experiment results demonstrate that an encouraging gaze estimation accuracy can be achieved by our proposed framework for both the normal vision and eyewear users.

Catheterization instruments are increasingly being improved to accurately diagnose and treat cardiovascular conditions. However, current catheter systems provide limited information about the shape of the catheter and tissue-instrument interaction forces during an intervention. Furthermore, relying on inconsistent feedback of such interaction forces during an intervention may result in tissue injury. This paper presents the first steps to estimate the interaction forces between a catheter and a mock-up arterial environment. We base the proposed method on a Pseudo-Rigid Body approximation of the catheter and integrate three-dimensional shape information provided by Fiber Bragg Grating sensors inside the catheter. The reconstructed forces along the catheter body can be fed back to the surgeon in visual and/or haptic form. In this work, the estimated forces are displayed in real-time in a graphical user interface with the reconstructed catheter shape. Experimental validation demonstrates a root mean square error of 0.03 N and a mean reconstruction error of 0.02 N.

This paper presents an algorithm to do motion planning for a new class of self-reconfigurable modular robot: the variable topology truss (VTT). Modular robots consist of many modules that can be configured into various structures, and motion planning problem for modular robots with many degrees of freedom and many motion constraints is a significant challenge. In this paper, we propose a novel motion planning algorithm for modular robots to handle this problem with huge state space inspired by DNA replication process - the topology of DNA can be changed by cutting and resealing strands as tanglements form. In a variable topology truss, a single node with enough edge modules can split into a pair of nodes and two separate nodes can be merged to become an individual one. This self-reconfiguration ability results in more potential applications for this type of robots in unstructured environment, such as space and underseas but also leads to more challenges for reconfiguration planning. A novel way to model the robot in a nonuniform grid space is presented and a simple local planner is also developed to check the validation of possible actions. This approach significantly simplifies the problem and some experiment results show that the complicated problem can be solved in a reasonable time.

Robots are required to support people's work. In order to alleviate the burden on people, it is desirable that robot can automatically generate and execute complicated motions according to simple directions from people. However, there are multiple grasping methods for one object. In order to select a motion suitable for the direction, it is important to estimate candidates of grasping method for the object. In this research, we purpose to recall candidates of grasping position and hand shape from an object image. In learning, a network that outputs a plurality of grasping method candidates for one object image to each channel of a multi-channel image is used. At this time, a plurality of grasping methods are not learned at same time, learned one by one. The similar grasping method for the similar object shape is automatically clustered to each output channel in the learning process, and a grasping method having a characteristic difference is presented as a candidate. We show the usefulness of this method using experimental examples.

Within this work, we explore intention inference for user actions in the context of a handheld robot setup. Handheld robots share the shape and properties of handheld tools while being able to process task information and aid manipulation. Here, we propose an intention prediction model to enhance cooperative task solving. The model derives intention from the combined information about the user's gaze pattern and task knowledge. Within experimental studies, the model is validated through a comparison of user frustration for the case where the robot follows the predicted location of the user's intended action versus doing the opposite (rebellion). The proposed model yields real-time capabilities and reliable accuracy up to 1.5 s prior to predicted actions being executed.

In collaborative robotics applications, human behaviour is a major source of uncertainty. Predicting the evolution of the current human activity might be beneficial to the effectiveness of task planning, as it enables a higher level of coordination of robot and human activities. This paper addresses the problem of monitoring the advancement of human tasks in real-time giving an estimate of their expected duration. The proposed method relies on dynamic time warping to align the current activity with a reference template. No training phase is required, as the prototypical execution is learnt online from previous instances of the same activity. The applicability and performance of the method within an industrial context have been verified on a realistic assembly task.

Recently, a new underwater imaging formation model presented that the coefficients related to the direct and backscatter transmission signals are dependent on the type of water, camera specifications, water depth, and imaging range. This paper proposes an underwater color correction method that integrates this new model on an underwater robot, using information from a pressure depth sensor for water depth and a visual odometry system for estimating scene distance. Experiments were performed with and without a color chart over coral reefs and a shipwreck in the Caribbean. We demonstrate the performance of our proposed method by comparing it with other statistic-, physic-, and learning-based color correction methods. Applications for our proposed method include improved 3D reconstruction and more robust underwater robot navigation.

The contribution of this paper is the application of compound state-triggered constraints (STCs) to real-time quadrotor path planning. Originally developed for rocket landing applications, STCs are made up of a trigger condition and a constraint condition that are arranged such that satisfaction of the former implies satisfaction of the latter. Compound STCs go a step further by allowing multiple trigger and constraint conditions to be combined via Boolean “and” or “or” operations. The logical implications embodied by STCs can be formulated using continuous variables, and thus enable the incorporation of discrete decision making into a continuous optimization framework. In this paper, compound STCs are used to solve quad-rotor path planning problems that would typically require the use of computationally expensive mixed-integer programming techniques. Two scenarios are considered: (1) a quad-rotor flying through a hoop, and (2) a pair of quadrotors carrying a beam-like payload through an obstacle course. Successive convexification is used to solve the resulting non-convex optimization problem. Monte-Carlo simulation results show that our approach can reliably generate trajectories at rates upwards of 3 and 1.5 Hz for the first and second scenarios, respectively.

We propose a fast and accurate 6D object pose estimation from a RGB-D image. Our proposed method is template matching based and consists of three main technical components, PCOF-MOD (multimodal PCOF), balanced pose tree (BPT) and optimum memory rearrangement for a coarse-to-fine search. Our model templates on densely sampled viewpoints and PCOF-MOD which explicitly handles a certain range of 3D object pose improve the robustness against background clutters. BPT which is an efficient tree-based data structures for a large number of templates and template matching on rearranged feature maps where nearby features are linearly aligned accelerate the pose estimation. The experimental evaluation on tabletop and bin-picking dataset showed that our method achieved higher accuracy and faster speed in comparison with state-of-the-art techniques including recent CNN based approaches. Moreover, our model templates can be trained solely from 3D CAD in a few minutes and the pose estimation run in near real-time (23 fps) on CPU. These features are suitable for any real applications.

Between humans, body motion is an intuitive and ubiquitous means to coordinate interactions, but so far collaborative robots have not embraced this mode of communication. One reason for this is that conventional behavior generation systems use finite state machines which make it exceptionally difficult to ingest and produce the inherently continuous and concurrent information flow that body motion provides. We propose a new, reactive motion generator based on a dynamical system instead. It mimics a conventional state machine except that transitions are not instantaneous but time-extended, reversible, and have phases. Moreover, consecutive states and transitions may overlap during execution. Most notably, more than one transition can be active at the same time if they share predecessor state. Together, these unique features enable instantaneous and gradual responses to novel information and continuous decision processes without changing the state graph itself. We demonstrate the system's capabilities in an object handover task coordinated by body motion.

Mapping and localization are essential capabilities of robotic systems. Although the majority of mapping systems focus on static environments, the deployment in real-world situations requires them to handle dynamic objects. In this paper, we propose an approach for an RGB-D sensor that is able to consistently map scenes containing multiple dynamic elements. For localization and mapping, we employ an efficient direct tracking on the truncated signed distance function (TSDF) and leverage color information encoded in the TSDF to estimate the pose of the sensor. The TSDF is efficiently represented using voxel hashing, with most computations parallelized on a GPU. For detecting dynamics, we exploit the residuals obtained after an initial registration, together with the explicit modeling of free space in the model. We evaluate our approach on existing datasets, and provide a new dataset showing highly dynamic scenes. These experiments show that our approach often surpass other state-of-the-art dense SLAM methods. We make available our dataset with the ground truth for both the trajectory of the RGB-D sensor obtained by a motion capture system and the model of the static environment using a high-precision terrestrial laser scanner. Finally, we release our approach as open source code.

Monocular visual-inertial tracking without good initialization easily fails due to its non-linear nature. Rapid and accurate metric initialization is crucial. In this paper, we propose a novel monocular visual-inertial initialization method which can initialize the IMU states, camera poses, and scale in a rapid and robust way. To avoid mixing gravity and accelerometer bias, we propose to use the detected vertical edges to estimate a better gravity. This improves the observability to the underlying problem even without sufficient movement, so we can solve all the states crucial for a good initialization. We evaluate our approach on EuRoC dataset and compare with existing state-of-the-art methods. The experimental results demonstrate the effectiveness of the proposed method.

One of the big attractions of low-dimensional models for gait design has been the ability to compute solutions rapidly, whereas one of their drawbacks has been the difficulty in mapping the solutions back to the target robot. This paper presents a set of tools for rapidly determining solutions for “humanoids” without removing or lumping degrees of freedom. The main tools are: (1) C-FROST, an open-source C++ interface for FROST, a direct collocation optimization tool; and (2) multi-threading. The results will be illustrated on a 20-DoF floating-base model for a Cassie-series bipedal robot through numerical optimization and physical experiments.

A growing trend in computer vision is the use of synthetic images for the evaluation of computer vision algorithms such as 3D pose estimation. This is partly due to the availability of high-quality render engines, which provide highly realistic synthetic images. However, the realism of the rendered images, and thus the reliability of the evaluations, strongly depends on how accurately the scenes are modeled and it requires considerable time and knowledge to do the modeling manually. Automating the modeling process is therefore crucial for making the rendering of photo-realistic synthetic images accessible to the wider robotics community. We present a method for automatically modeling object and light properties for rigid, opaque plastic objects commonly found in industry. Our method relies on recordings of the environment captured with a consumer 360^{\mathrm{o}}360^{\mathrm{o}} camera to model the light, and on analysis-by-synthesis to estimate the optical properties of the objects. We show that the synthetic images rendered based on our automatic modeling method can be used to predict the overall performance of a monocular 3D pose estimation algorithm.

Mechanical logic is a class of dynamic electromechanical mechanisms which leverages carefully designed mechanical structures to generate programmed control actions from a constant electrical power supply; thus, it can be employed as a control method for fully printable autonomous robots. Composed of a bistable buckled beam driven by conductive super-coiled polymer (CSCP) actuators, this type of electromechanical system features non-trivial relationships between its design parameters and resulting behavioral characteristics. In this paper we present an efficient method to rapidly design mechanical logic structures from desired behavioral specifications. We describe this dynamic system with a simplified, quasi-static model, whose validity is verified by time constant comparison. An analytical formula of the mechanical logic's behavioral characteristics, i.e. its oscillation period, is then derived as a simplified expression of the design parameters. Based on this expression, we formulate the design of mechanical logic from behavioral specifications into an optimization problem that maximizes the robustness to manufacturing tolerances, as demonstrated by an example case study.

We present a continuous-time collision detection algorithm for quickly detecting whether certain polynomial trajectories in time intersect with convex obstacles. The algorithm is used in conjunction with an existing multicopter trajectory generation method to achieve rapid, obstacle-aware motion planning in environments with both static convex obstacles and dynamic convex obstacles whose boundaries do not rotate. In general, this problem is difficult because the presence of convex obstacles makes the feasible space of trajectories nonconvex. The performance of the algorithm is benchmarked using Monte Carlo simulations, and experimental results are presented that demonstrate the use of the method to plan collision-free multicopter trajectories in milliseconds in environments with both static and dynamic obstacles.

Perception in autonomous vehicles is often carried out through a suite of different sensing modalities. Given the massive amount of openly available labeled RGB data and the advent of high-quality deep learning algorithms for image-based recognition, high-level semantic perception tasks are pre-dominantly solved using high-resolution cameras. As a result of that, other sensor modalities potentially useful for this task are often ignored. In this paper, we push the state of the art in LiDAR-only semantic segmentation forward in order to provide another independent source of semantic information to the vehicle. Our approach can accurately perform full semantic segmentation of LiDAR point clouds at sensor frame rate. We exploit range images as an intermediate representation in combination with a Convolutional Neural Network (CNN) exploiting the rotating LiDAR sensor model. To obtain accurate results, we propose a novel post-processing algorithm that deals with problems arising from this intermediate representation such as discretization errors and blurry CNN outputs. We implemented and thoroughly evaluated our approach including several comparisons to the state of the art. Our experiments show that our approach outperforms state-of-the-art approaches, while still running online on a single embedded GPU. The code can be accessed at https://github.com/PRBonn/lidar-bonnetal.

This paper studies the problem of distributed computation of higher order Voronoi partition over a bounded region by a group of robots with both range-limited visibility sensors and communication devices. We model the sensing and communication capabilities by discs with limited radius. Motivated by the concept of dominating region in higher-order Voronoi partition, we propose a detecting ray based algorithm, which computes the boundary points of the dominating region of a robot in an omnidirectional manner, with local position information of its neighbors within the communication range. Simulations are provided to demonstrate the performance of our proposed algorithm by using a thirteen-robot group.