IROS 2019 Papers

In this study, a three-layered bidirectional Long Short-term Memory (Bi-LSTM) with residual attention, named as RONet, is proposed to achieve localization using range measurements. Accordingly, we acquired our own datasets and tested RONet using realistic conditions. It is shown that the RONet can estimate the position of the mobile robot in real time using the Nvidia Jetson AGX Xavier based only on range measurements. We also analyzed the sequence length of LSTM as a type of hyperparameters. We found that optimal sequence length is eight for more than eight anchors and twelve for fewer anchors compared to sequences with different lengths, given that construction of the network with the optimal sequence length estimates the position precisely and accounts for uncertainties. As verified experimentally, RONet yields more precise performance and results in increased robustness against outliers compared to a conventional range-only approach based on a particle filtering and the other conventional deep-learning-based approaches. We set three cases, reduced the number of anchors, and verified that the RONet was a robust solution. We also confirmed that it is the best solution that yields the smallest Root-Mean-Square-Error (RMSE) values, equal to 4.466 cm, 3.210 cm, and 3.090 cm, in the cases where three, five, and eight anchors were deployed, respectively.

Grasp detection considering the affiliations between grasps and their owner in object overlapping scenes is a necessary and challenging task for the practical use of the robotic grasping approach. In this paper, a robotic grasp detection algorithm named ROI-GD is proposed to provide a feasible solution to this problem based on Region of Interest (ROI), which is the region proposal for objects. ROI-GD uses features from ROIs to detect grasps instead of the whole scene. It has two stages: the first stage is to provide ROIs in the input image and the second-stage is the grasp detector based on ROI features. We also contribute a multi-object grasp dataset, (a) which is much larger than Cornell Grasp Dataset, by labeling Visual Manipulation Relationship Dataset. Experimental results demonstrate that ROI-GD performs much better in object overlapping scenes and at the meantime, remains comparable with state-of-the-art grasp detection algorithms on Cornell Grasp Dataset and Jacquard Dataset. Robotic experiments demonstrate that ROI-GD can help robots grasp the target in single-object and multi-object scenes with the overall success rates of 92.5% and 83.8% respectively.

In this paper, we present the RISE-SLAM algorithm for performing visual-inertial simultaneous localization and mapping (SLAM), while improving estimation consistency. Specifically, in order to achieve real-time operation, existing approaches often assume previously-estimated states to be perfectly known, which leads to inconsistent estimates. Instead, based on the idea of the Schmidt-Kalman filter, which has processing cost linear in the size of the state vector but quadratic memory requirements, we derive a new consistent approximate method in the information domain, which has linear memory requirements and adjustable (constant to linear) processing cost. In particular, this method, the resource-aware inverse Schmidt estimator (RISE), allows trading estimation accuracy for computational efficiency. Furthermore, and in order to better address the requirements of a SLAM system during an exploration vs. a relocalization phase, we employ different configurations of RISE (in terms of the number and order of states updated) to maximize accuracy while preserving efficiency. Lastly, we evaluate the proposed RISE-SLAM algorithm on publicly-available datasets and demonstrate its superiority, both in terms of accuracy and efficiency, as compared to alternative visual-inertial SLAM systems.

This paper proposes a real-time approach for long-term inertial navigation based only on an Inertial Measurement Unit (IMU) for self-localizing wheeled robots. The approach builds upon two components: 1) a robust detector that uses recurrent deep neural networks to dynamically detect a variety of situations of interest, such as zero velocity or no lateral slip; and 2) a state-of-the-art Kalman filter which incorporates this knowledge as pseudo-measurements for localization. Evaluations on a publicly available car dataset demonstrates that the proposed scheme may achieve a final distance error of 20 m for a 21 km long trajectory of a vehicle driving for over an hour, equipped with an IMU of moderate precision (the gyro drift rate is 10 deg/h). To our knowledge, this is the first paper which combines sophisticated deep learning techniques with state-of the-art filtering methods for pure inertial navigation on wheeled vehicles and as such opens up for novel data-driven inertial navigation techniques. Moreover, albeit taylored for IMU-only based localization, our method may be used as a component for self-localization of wheeled robots equipped with a more complete sensor suite.

In this paper, we present a novel method to predict 3D TSDF voxels from a single image for dense 3D reconstruction. 3D reconstruction with RGB images has two inherent problems: scale ambiguity and sparse reconstruction. With the advent of deep learning, depth prediction from a single RGB image has addressed these problems. However, as the predicted depth is typically noisy, de-noising methods such as TSDF fusion should be adapted for the accurate scene reconstruction. To integrate the two-step processing of depth prediction and TSDF generation, we design an RGB-to-TSDF network to directly predict 3D TSDF voxels from a single RGB image. The TSDF using our network can be generated more efficiently in terms of time and accuracy than the TSDF converted from depth prediction. We also use the predicted TSDF for a more accurate and robust camera pose estimation to complete scene reconstruction. The global TSDF is updated from TSDF prediction and pose estimation, and thus dense isosurface can be extracted. In the experiments, we evaluate our TSDF prediction and camera pose estimation results against the conventional method.

This paper explores the use of robots to autonomously assemble parts with variations in colors and textures. Specifically, we focus on peg-in-hole assembly with some initial position uncertainty and holes located on surfaces of different colors and textures. Two in-hand cameras and a force-torque sensor are used to account for the position uncertainty. A program sequence comprising learning-based visual servoing, spiral search, and impedance control is implemented to perform the peg-in-hole task with feedback from the above sensors. Contributions are mainly made in the learning-based visual servoing component of the sequence, where a deep neural network is trained with various sets of synthetic data generated using the concept of domain randomization to predict where a hole is. In the experiments and analysis section, the network is analyzed and compared, and a real-world robotic system to insert pegs to holes using the proposed method is implemented. The results show that the implemented peg-in-hole assembly system can perform successful peg-in-hole insertions on surfaces with various colors and textures. It can generally speed up the entire peg-in-hole process, especially when the initial position uncertainty is large.

Autonomous robots can rely on attention mechanisms to explore complex scenes and select salient stimuli relevant for behaviour. Stimulus selection should be fast to efficiently allocate available (and limited) computational resources to process in detail a subset of the otherwise overwhelmingly large sensory input. The amount of processing required is a product of the amount of data sampled by a robot's sensors; while a standard RGB camera produces a fixed amount of data for every pixel of the sensor, an event-camera produces data only for where there is a contrast change in the field of view, and does so with a lower latency. In this paper, we describe the implementation of a state-of-the-art bottom-up attention model, based on structuring the visual scene in terms of proto-objects. As an event-camera encodes different visual information compared to frame-based cameras, the original algorithm must be adapted and modified. We find that the event-camera's inherent detection of edges removes the need for some early stages of processing in the model. We describe the modifications, compare the event-driven algorithm to the original, and validate the potential for use on the iCub humanoid robot.

The cleaning of residential ventilation ducts is necessary to prevent damage to health. However, since residential ventilation ducts are usually thin and curved, cleaning them is difficult. It is also difficult to obtain thrust within the duct using a wheel or a snake type robot because it is hard to obtain an appropriate reaction force in a deformed pipe. Therefore, the pipe must be gripped stably even if it is flexible so that thrust can be generated to run the cleaning robot through the pipe. In this paper, we modeled a cleaning robot that uses peristaltic motion running through a flexible duct with an inner diameter of 50 mm. Then, the validity of the model was verified by experiment. Finally, a cleaning experiment was conducted and the cleaning rate was 98.7 %.

As smart cameras are becoming ubiquitous in mobile robot systems, there is an increasing concern in camera devices invading people's privacy by recording unwanted images. We want to fundamentally protect privacy by blurring unwanted blocks in images, such as faces, yet ensure that the robots can understand the video for their perception. In this paper, we propose a novel mobile robot framework with a deep learning-based privacy-preserving camera system. The proposed camera system detects privacy-sensitive blocks, i.e., human face, from extreme low resolution (LR) images, and then dynamically enhances the resolution of only privacy-insensitive blocks, e.g., backgrounds. Keeping all the face blocks to be extreme LR of 15x15 pixels, we can guarantee that human faces are never at high resolution (HR) in any of processing or memory, thus yielding strong privacy protection even from cracking or backdoors. Our camera system produces an image on a real-time basis, the human faces of which are in extreme LR while the backgrounds are in HR. We experimentally confirm that our proposed face detection camera system outperforms the state-of-the-art small face detection algorithm, while the robot performs ORB-SLAM2 well even with videos of extreme LR faces. Therefore, with the proposed system, we do not too much sacrifice robot perception performance to protect privacy.

Conventional teleoperative interfaces, used for remote manipulation in the nuclear industry, utilise multiple stationary CCTV cameras for situational awareness. This is cognitively challenging for the human operator, who must mentally combine information from multiple 2D views, before attempting 3D reasoning about the remote environment. To enhance performance on telemanipulation tasks, this paper explores the merits of employing an orbital camera view. The orbital camera's gaze direction is automatically fixated towards the robot's moving end-effector. Meanwhile, the camera can be controlled by the operator to move on a spherical surface around the end-effector, in addition to zooming capability. The robot's Cartesian motion coordinates are also continuously transformed to be controllable with respect to the axes of the orbital camera view. Human test-subject experiments were conducted in a simulation environment using novice robot operators, and a variety of metrics were collected. Although the results do not show an objective difference between the camera modes, participant interviews suggest the orbital camera is the preferred method for visual feedback with its dynamic views helping to overcome positioning difficulties associated with stationary cameras. The experiment also revealed significant confounding factors that contributed to the results being inconclusive. These are discussed and recommendations are made for future empirical experiments to evaluate such systems.

In this paper, we compare different map management techniques for long-term visual navigation in changing environments. In this scenario, the navigation system needs to continuously update and refine its feature map in order to adapt to the environment appearance change. To achieve reliable long-term navigation, the map management techniques have to (i) select features useful for the current navigation task, (ii) remove features that are obsolete, (iii) and add new features from the current camera view to the map. We propose several map management strategies and evaluate their performance with regard to the robot localisation accuracy in long-term teach-and-repeat navigation. Our experiments, performed over three months, indicate that strategies which model cyclic changes of the environment appearance and predict which features are going to be visible at a particular time and location, outperform strategies which do not explicitly model the temporal evolution of the changes.

This paper presents an inverse kinematics (IK) method which can control future velocities and accelerations for multi-body systems. The proposed IK method is formulated as a quadratic programing (QP) that optimizes future joint trajectories. The features of the proposed IK are: (1) the evaluation of accelerations at future time instances, (2) the trajectory representation that can implicitly integrate the time integral formula into QP, (3) the computation of future Jacobian matrices based on the comprehensive theory of differential kinematics proposed in our previous work. Those features enable a stable and fast IK computation while evaluating the future accelerations. We also conducted thorough numerical studies to show the efficiency of the proposed method.

The raise of collaborative robotics has allowed to create new spaces where robots and humans work in proximity. Consequently, to predict human movements and his/her final intention becomes crucial to anticipate robot next move, preserving safety and increasing efficiency. In this paper we propose a human-arm prediction algorithm that allows to infer if the human operator is moving towards the robot to intentionally interact with it. The human hand position is tracked by an RGB-D camera online. By combining the Minimum Jerk model with Semi-Adaptable Neural Networks we obtain a reliable prediction of the human hand trajectory and final target in a short amount of time. The proposed algorithm was tested in a multi-movements scenario with FANUC LR Mate 200iD/7L industrial robot.

Tactile sensors have been increasingly used to support rigid robot grippers in object grasping and manipulation. However, rigid grippers are often limited in their ability to handle compliant, delicate, or irregularly shaped objects. In recent years, grippers made from soft and flexible materials have become increasingly popular for certain manipulation tasks, e.g., grasping, due to their ability to conform to the object shape without the need for precise control. Although promising, such soft robot grippers currently suffer from the lack of available sensing modalities. In this work, we introduce a soft and stretchable sensing skin and incorporate it into the two fingers of a shape-memory actuated soft gripper. The onboard sensing skin includes a 9-axis inertial measurement unit (IMU) and five discrete pressure sensors per finger. We use this sensorized soft gripper to study grasp success and stability of over 2585 grasps with various objects using several machine learning methods. Our experiments show that LSTMs were the most accurate predictors of grasp success and stability, compared to SVMs, FFNNs, and ST-HMP. We also evaluated the effects on performance of each sensor's data, and the success rates for individual objects. The results show that the accelerometer data of the IMUs has the largest contribution to the overall grasp prediction, which we attribute to its ability to detect precise movements of the gripper during grasping.

Service robots have developed considerably in recent years. This study aims to propose two approaches for controlling the motion of a service robot as it pours liquid precisely from an unknown container into another unknown container without the need of any external tools. To realize this task, we must resolve two sub-tasks, which are determining the poured volume and controlling the pouring action. Our first proposal concentrates on the target container. The poured volume is calculated using a model of target container and the height of liquid in the target container. The action is controlled using a proportional-derivative controller, which considers the angular speed of the pouring container as a process variable and the poured volume as a control variable. Our second method concentrates on the pouring container. The poured volume is calculated using the relation between the angle of the pouring container and poured volume. The action is controlled with a simple proportional controller that takes the angular speed of the pouring container as a process variable and target angle as a control variable. A point cloud is used to model the two containers. These two methods were implemented in a dualarm robot system for testing, and the results show that both methods are effective for controlling precise pouring tasks.

The seminal work cited in [1],[2] showed, for the first time, that flight stability of quadcopters would be possible in case of one or even multiple rotor failures. However, the quadcopter can remain airborne only by going through a spinning maneuver about an axis, fixed w.r.t the vehicle (i.e., resolved yaw). Furthermore, positional control can be achieved by periodically tilting this axis. This paper builds upon this concept with two major improvements: (1) introducing a precise aerodynamic model of propellers that takes the flapping torque due to unbalanced lifting force in the advancing and retreating blades subjected to freestream, into account, and (2) adding to the stability and flight efficiency of the quadcopter by introducing symmetric fixed tilting angles to the trust vectors. In our previous work [3], it was shown how the flight stability and energy efficiency can be improved by introducing fixed tilting angles in the thrust vectors. For controlled crash landing in case of one rotor failure, where a resolved yaw maneuver would be inevitable, introducing a titling angle in rotors can generate a reasonable resolved-rate-yaw spinning speed to keep the quadcopter airborne at a lower rotational speed of the blades by taking advantage of the freestream generated by spinning. This tilting angle would also lead to passive stability in yaw motion of the quadcopter before the failure. Our hypothesis was successfully tested via simulations.

Robust 3D perception using LiDAR is of prime importance for robotics, and its fundamental core lies in precise object modelling resisting to noise and outliers. In this paper, a precise 3D object modelling algorithm is designed especially for the intelligent vehicles. The proposed algorithm is advantageous by leveraging the crucial traffic geometrical prior of road surface profile, and both the noise and outliers are elegantly handled by robust correntropy-based metric. More specifically, the road surface correction (RSC) method transforms each individual LiDAR measurement from its locally planar road surface to a globally ideal plane. This procedure essentially guarantees the reduction of vehicle's motion from arbitrary 3D motion to physically feasible 2D motion. To deal with the noise and outliers, a correntropy-based multi-frame matching (CorrMM) algorithm is proposed which has a robust objective function with respect to point-to-plane residual error. An efficient solver inspired by M-estimator and retraction technique on Lie group is developed, which elegantly converts the optimization of highly non-linear objective function into a simple quadratic programming (QP) problem. Extensive experimental results validate that the proposed algorithm attains more crisper 3D object models than several state-of-the-art algorithms on a challenging real traffic dataset.

Visual servoing, i.e. control with visual information, is a valuable capability in many robotic applications. In particular, position based visual servoing (PBVS) estimates position information from the observed image to generate visual servo control. However, the estimation of the position of an unknown target using monocular images is still difficult due to the complexity of the image information. For the target estimation problem, we propose to integrate three complementary techniques for monocular visual servoing. First, to estimate the probability of a target's existence, the learning model with spatial features from convolution neural network is proposed. Second, the extended Kalman filter based on epipolar geometry estimates the 3D position of the target; moreover, from this 3D position, the perception model is trained online by self-generated virtual ground-truth. Finally, visual servo control is generated, and the resulting movement helps to construct epipolar geometry. Finally. the experimental validation is performed in a challenging setting involving occlusion and target's shape change.