IROS 2018 Papers

Sensory information from an Earth-fixed reference is necessary to guarantee a high localization accuracy of an unmanned underwater vehicle (UUV). However, the implementation of these sensors in an underwater environment is challenging because of signal uncertainties and strong signal attenuation. In this paper, we propose an underwater localization scheme with a sensor fusion of inertial navigation system (INS) and received signal strength of electromagnetic (EM) waves sensors. In the proposed sensor-fusion-based localization scheme, the UUV predicts its location by using INS based on dead-reckoning and corrects the predicted position by Kalman filter using EM waves sensor information when the UUV receives the signals of EM waves sensors in underwater wireless sensor networks. The proposed scheme enables localization with high accuracy and high sampling rate during a long-term task. The results of an experiment performed in a basin environment shows the feasibility of the proposed scheme. The scheme achieved reliable localization accuracy by comparing the pre-measured ground-truth position and long-term navigation. These results show the feasibility of exploiting EM waves attenuation as Earth-fixed reference sensors.

Semantic navigation is the navigation paradigm in which environmental semantic concepts and their relationships are taken into account to plan the route of a mobile robot. This paradigm facilitates the interaction with humans and the understanding of human environments in terms of navigation goals and tasks. At the high level, a semantic navigation system requires two main components: a semantic representation of the environment, and a reasoning system. This paper is focused on develop a model of the environment using semantic concepts. This paper presents two solutions for the semantic navigation paradigm. Both systems implement an ontological model. Whilst the first one uses a relational database, the second one is based on KnowRob. Both systems have been integrated in a semantic navigator. We compare both systems at the qualitative and quantitative levels, and present an implementation on a mobile robot as a proof of concept.

We develop a real-time state estimation system to recover the pose and contact formation of an object relative to its environment. In this paper, we focus on the application of inserting an object picked by a suction cup into a tight space, a key technology for robotic packaging. We propose a framework that fuses tactile and visual sensing. Visual sensing is versatile and non-intrusive, but suffers from occlusions and limited accuracy, especially for tasks involving contact. Tactile sensing is local, but provides accuracy and robustness to occlusions. The proposed algorithm to fuse them is based on iSAM, an on-line estimation technique, which we use to incorporate kinematic measurements from the robot, contact geometry of the object and the container, and visual tracking. In this paper, we generalize previous results in planar settings [1] to a 3D task with more complex contact interactions. A key challenge is that we do not observe contact locations between the suction-held object and the container directly. We propose a data-driven method to infer the contact formation, which is then used in real-time by the state estimator. We demonstrate and evaluate the algorithm in a setup instrumented to provide groundtruth.

We propose to combine recent Convolutional Neural Networks (CNN) models with depth imaging to obtain a reliable and fast multi-person pose estimation algorithm applicable to Human Robot Interaction (HRI) scenarios. Our hypothesis is that depth images contain less structures and are easier to process than RGB images while keeping the required information for human detection and pose inference, thus allowing the use of simpler networks for the task. Our contributions are threefold. (i) we propose a fast and efficient network based on residual blocks (called RPM) for body landmark localization from depth images; (ii) we created a public dataset DIH comprising more than 170k synthetic images of human bodies with various shapes and viewpoints as well as real (annotated) data for evaluation; (iii) we show that our model trained on synthetic data from scratch can perform well on real data, obtaining similar results to larger models initialized with pre-trained networks. It thus provides a good trade-off between performance and computation. Experiments on real data demonstrate the validity of our approach.

Motor rehabilitation is in increasingly high demand to deal with minor functional motor impairments resulting from stroke, cerebellar ataxia, or Parkinson's disease. Juggling physiotherapy has shown to induce brain plasticity and to improve coordination and balance in this context. The physiotherapy, however, relies on large number of repetitions to be effective which prompts to deploy robots to release the burden on therapists both in terms of time as well as physical strain. This paper provides a framework to enable juggling games for patients in interacting with robots through Virtual Reality (VR). A set of throwing motions is recorded from the therapist and is retargeted to the humanoid robot COMAN's wrist. The respective whole-body motion is then solved in a stack of Quadratic Programs (QP) in a real-time architecture that integrates OROCOS and Gazebo. The resulting motion is finally streamed to VR for animation of the robot and the thrown ball, which the user can catch in VR using a controller device. We regard the VR setting as an essential step towards physiotherapeutic robotic juggling, because it ensures safety of the patients and effective testing of the methods and already has potential for actual therapeutic intervention. The control framework, however, is already validated in this paper for switching to full real-time operation on the physical robot.

Real-time 3D reconstruction using low-cost commodity sensors like Kinect or Xtion has been successfully applied in a wide range of fields like augmented reality, robotic teleoperation, and medical diagnosis. Due to the assumption of static scene, popular 3D reconstruction technologies such as KinectFusion and KinFu, find truthful reconstruction with fast motion camera or segmenting a moving object to be a challenge. In this paper, we propose a weighted iterative closest point (ICP) algorithm that uses both depth and RGB information to enhance the stability of camera tracking. Additionally, a GPU-based region growing method that combines depth, normal and intensity level as similarity criteria, is also applied to segment foreground moving objects accurately. For real-time processing and GPU memory efficiency, we also design a combination of point-based and volumetric representation to reconstruct moving objects and static scene, respectively. Both qualitative and quantitative results show that our proposed method improves real-time 3D reconstruction on the performance of camera tracking and segmentation of moving objects with reduced computational complexity.

Classifying human cognitive states from behavioral and physiological signals is a challenging problem with important applications in robotics. The problem is challenging due to the data variability among individual users, and sensor artefacts. In this work, we propose an end-to-end framework for real-time cognitive workload classification with mixture Hyper Long Short Term Memory Networks (m-HyperLSTM), a novel variant of HyperNetworks. Evaluating the proposed approach on an eye-gaze pattern dataset collected from simulated driving scenarios of different cognitive demands, we show that the proposed framework outperforms previous baseline methods and achieves 83.9% precision and 87.8% recall during test. We also demonstrate the merit of our proposed architecture by showing improved performance over other LSTM-based methods.

In this paper, we describe the method and implementation of a real-time tumor tracking system for a pencil beam scanning (PBS) proton therapy system. PBS is an advanced cancer treatment system that can benefit from precise localization of the tumors through motion. We utilize techniques such as cross-correlation matching, correlation filters and small object saliency, creating an array of methods that can detect and track fiducial markers implanted in the cancer tumors. The final aim is to control the proton beam using real-time image guidance. Our technique works robustly on various types of markers such as ceramic/metallic fiducials, visicoil markers and surgical clips. Left and right views of an X-ray fluoroscopy system were utilized to also triangulate the marker positions in full 3D as they are tracked through normal breathing movement and organ motion. We have tested our detection system on data from several patients with different tumor locations both offline and in real-time and wish to implement it within a full treatment system soon. To the best of the authors knowledge, this is the first real time tracking system for PBS therapy that is applicable for various types of fiducials and tumor locations.

This paper proposes a real-time method for estimating the shape of an elastic rod using a robot manipulator equipped with a sense of force. The proposed method does not use optical sensing devices, such as cameras or lasers, but relies only upon the sense of force in the manipulator. In the proposed method, the deformation of an elastic rod is calculated from the obtained force/torque information using the discretized Kirchhoff elastic rod model, and the three-dimensional shape of the rod is then estimated. Furthermore, by integrating the force information with the orientation of the manipulator end-effector, the proposed method can evaluate the effect of gravity on the shape estimation accurately. Experiments were carried out to verify the proposed method, where a thin elastic strip was employed as a typical elastic rod, attached to the end-effector of the robot manipulator, bent and twisted into various shapes. The results show that the proposed method that compensates for gravity is better than a method without gravity compensation, and it estimated the 142 mm length strip shape with a position error no greater than 7.76 mm and an average calculation time of 4.24 ms.

Real-time Segmentation is of crucial importance to robotics related applications such as autonomous driving, driving assisted systems, and traffic monitoring from unmanned aerial vehicles imagery. We propose a novel two-stream convolutional network for motion segmentation, which exploits flow and geometric cues to balance the accuracy and computational efficiency trade-offs. The geometric cues take advantage of the domain knowledge of the application. In case of mostly planar scenes from high altitude unmanned aerial vehicles (UAVs), homography compensated flow is used. While in the case of urban scenes in autonomous driving, with GPS/IMU sensory data available, sparse projected depth estimates and odometry information are used. The network provides 4.7× speedup over the state of the art networks in motion segmentation from 153ms to 36ms, at the expense of a reduction in the segmentation accuracy in terms of pixel boundaries. This enables the network to perform real-time on a Jetson T×2. In order to recuperate some of the accuracy loss, geometric priors is used while still achieving a much improved computational efficiency with respect to the state-of-the-art. The usage of geometric priors improved the segmentation in UAV imagery by 5.2 % using the metric of IoU over the baseline network. While on KITTI-MoSeg the sparse depth estimates improved the segmentation by 12.5 % over the baseline. Our proposed motion segmentation solution is verified on the popular KITTI and VIVID datasets, with additional labels we have produced. The code for our work is publicly available at1.

In this paper, we employ convex optimization to perform real-time 3-dimensional path planning on-board a quad-rotor and demonstrate its real-time capabilities. Building on our previous work, we make the following modifications: (1)we assume the obstacles are mobile, and (2)we introduce a simple framework to continuously recompute and update the trajectory. The contribution of this paper is to demonstrate the feasibility of real-time on-board convex-optimization-based path planning. For multi-rotors with fixed-pitch propellers, this path planning problem has two sources of non-convexity. First, since fixed-pitch actuators produce uni-directional thrust, the commanded total thrust must be maintained above a non-zero minimum in order to retain sufficient independent attitude control authority. The second source of non-convexity is due to the keep-out zones that envelop each obstacle. To circumvent the non-convexities introduced by these control and state constraints, we employ lossless and successive con-vexification, respectively. Consequently, we cast the original problem as a sequence of Second-Order Cone Programming problems, which can be solved quickly and reliably on-board. We conclude by presenting indoor flight demonstration and timing results of a scenario with three mobile obstacles. In this scenario, our algorithm assumes that the obstacles move with constant acceleration, and is re-executed regularly to account for uncertainties in the motion of the obstacles. The results show that new trajectories can be computed at rates in excess of 10 Hz, quickly enough to adapt to the uncertainty introduced in our flight demonstration.

In this work, we introduce pose interpreter networks for 6-DoF object pose estimation. In contrast to other CNN-based approaches to pose estimation that require expensively annotated object pose data, our pose interpreter network is trained entirely on synthetic pose data. We use object masks as an intermediate representation to bridge real and synthetic. We show that when combined with a segmentation model trained on RGB images, our synthetically trained pose interpreter network is able to generalize to real data. Our end-to-end system for object pose estimation runs in real-time (20 Hz) on live RGB data, without using depth information or ICP refinement.

Trajectory planning and replanning in complex environments often reuses very little information from the previous solutions. This is particularly evident when the motion is repeated multiple times with only a limited amount of variation between each run. To address this issue, we propose the DRM-connect algorithm, a combination of dynamic reachability maps (DRM) with lazy collision checking and a fallback strategy based on the RRT-connect algorithm which is used to repair the roadmap through further exploration. This fallback allows us to use much sparser roadmaps. Furthermore, we investigate using an approximate Reeb graph to capture the topology-persistent features of the past solutions of the problem utilising this sparsity. We evaluate DRM-connect with a Reeb graph on reaching tasks, and we compare it to state-of-the-art methods. We show that the proposed method outperforms both RRT-connect and BKPIECE algorithms in the number of collision checks required and we show that our method has the potential to scale to systems with higher number degrees of freedom.

Typical autonomous robotics systems incorporate multiple cameras, LIDAR sensors and sophisticated computing resources. In this paper we present a software framework for utilizing any array of multiple cameras with sufficient field-of-view (FOV) overlap as a light field imaging system. We show that the typical linear arrays that exist on autonomous cars are sufficient to capture stable time resolved light fields even when moving at highway speeds. We elaborate on the potential pitfalls associated with such a technique namely loss of calibration between cameras due to high frequency vibrations and sudden shocks associated with driving over potholes and highlight a method that can compensate for such effects. We demonstrate that the light fields collected by simple linear arrays can be processed in real time for a wide variety of useful applications including occlusion removal, for signal enhancement in featureless images captured in very low light, for reflection removal and for improved visibility in extreme conditions associated with snow and heavy rain.

Grasp planning for multi-fingered hands is computationally expensive due to the joint-contact coupling, surface nonlinearities and high dimensionality, thus is generally not affordable for real-time implementations. Traditional planning methods by optimization, sampling or learning work well in planning for parallel grippers but remain challenging for multi-fingered hands. This paper proposes a strategy called finger splitting, to plan precision grasps for multi-fingered hands starting from optimal parallel grasps. The finger splitting is optimized by a dual-stage iterative optimization including a contact point optimization (CPO) and a palm pose optimization (PPO), to gradually split fingers and adjust both the contact points and the palm pose. The dual-stage optimization is able to consider both the object grasp quality and hand manipulability, address the nonlinearities and coupling, and achieve efficient convergence within one second. Simulation results demonstrate the effectiveness of the proposed approach. The simulation video is available at [1].

Without the 3-D geometry of the target and robust to camera calibration error, image-based visual servoing schemes have gained a lot of attention. However, the depth of the selected feature, which is involved in the interaction matrix relating the time variation of the feature to the velocity twist of the camera, must be estimated correctly to guarantee the stability of the controller. To this end, this paper proposes a new nonlinear reduced-order observer structure to recover the feature depth in real time. Compared with the existing works, the proposed observer has a global asymptotic convergence property and fast convergence rate, and the convergence rate can be easily adjusted only using a single gain parameter. In addition, the proposed observer has a less restrictive observability condition and stronger robustness to noisy measurements. Extensive comparative numerical simulations are carried out to validate the effectiveness of the proposed depth observer.

In this paper, we propose a novel real-time method for tracking planar edge templates. This method tracks an edge template by estimating its homography transformations with respect to the sampled edge pixels detected from the incoming frames. Particularly, we define a cost function based on a new feature map of the to-be-tracked edge template and optimize it by a Lucas-Kanade-like algorithm. The feature map is defined as the fourth root of the distance transform. Our method operates on just edges so that it is good at tracking those low textured targets, such as hollow targets (mug rim), thin targets (cable, ring) and non-Lambertian objects (disc). We validate and compare our method with four other methods on five newly collected real-world video sequences. The results achieves the lowest overall average error (1.58 pixels) and also outperforms others in terms of success rate. The per frame processing time of about 30 ms proves that our method is acceptable in realtime applications. The code and dataset are publicly available at: http://webdocs.cs.ualberta.ca/~xuebin/.

The development of robots that can dance has received considerable attention. However, they are often either limited to a pre-defined set of movements and music or demonstrate little variance when reacting to external stimuli, such as microphone or camera input. In this paper, we contribute with a novel approach allowing a legged robot to listen to live music while dancing in synchronization with the music in a diverse fashion. This is achieved by extracting the beat from an onboard microphone in real-time, and subsequently creating a dance choreography by picking from a user-generated dance motion library at every new beat. Dance motions include various stepping and base motions. The process of picking from the library is defined by a probabilistic model, namely a Markov chain, that depends on the previously picked dance motion and the current music tempo. Finally, delays are determined online by time-shifting a measured signal and a reference signal, and minimizing the least squares error with the time-shift as parameter. Delays are then compensated for by using a combined feedforward and feedback delay controller which shifts the robot whole-body controller reference input in time. Results from experiments on a quadrupedal robot demonstrate the fast convergence and synchrony to the perceived music.

Clustering is crucial for many computer vision applications such as robust tracking, object detection and segmentation. This work presents a real-time clustering technique that takes advantage of the unique properties of event-based vision sensors. Since event-based sensors trigger events only when the intensity changes, the data is sparse, with low redundancy. Thus, our approach redefines the well-known mean-shift clustering method using asynchronous events instead of conventional frames. The potential of our approach is demonstrated in a multi-target tracking application using Kalman filters to smooth the trajectories. We evaluated our method on an existing dataset with patterns of different shapes and speeds, and a new dataset that we collected. The sensor was attached to the Baxter robot in an eye-in-hand setup monitoring real-world objects in an action manipulation task. Clustering accuracy achieved an F-measure of 0.95, reducing the computational cost by 88% compared to the frame-based method. The average error for tracking was 2.5 pixels and the clustering achieved a consistent number of clusters along time.