IROS 2019 Papers

We present a dataset for evaluating the performance of visual-inertial odometry (VIO) systems employing an onboard light source. The dataset consists of 39 sequences, recorded in mines, tunnels, and other dark environments, totaling more than 160 minutes of stereo camera video and IMU data. In each sequence, the scene is illuminated by an onboard light of approximately 1300, 4500, or 9000 lumens. We accommodate both direct and indirect visual odometry methods by providing the geometric and photometric camera calibrations (i.e. response, attenuation, and exposure times). In contrast with existing datasets, we also calibrate the light source itself and publish data for inferring more complex light models. Ground-truth position data are available for a subset of sequences, as captured by a Leica total station. All remaining sequences start and end at the same position, permitting the use of total accumulated drift as a metric for evaluation. Using our proposed benchmark, we analyze the performance of several start-of-the-art VO and VIO frame-works. The full dataset, including sensor data, calibration sequences, and evaluation scripts, is publicly available online at http://arpg.colorado.edu/research/oivio.

This paper presents a cognitive assistant system for emergency medical services (EMS) that can serve as a rescue robot or virtual assistant, helping with improving situational awareness of the first responders through automated collection and analysis of data from the incident scene and providing suggestions to them. The proposed system relies on a Behavior Tree (BT) framework that combines the knowledge of EMS protocol guidelines with speech recognition, natural language processing, and machine learning methods to (i) extract critical information from responders' conversations and verbalized observations, (ii) infer the incident context, and (iii) decide on safe and effective response interventions to perform. We use a data-set of 8302 real EMS call records from an urban, high volume regional ambulance agency in the U.S. to evaluate the responsiveness and cognitive ability of the system and assess the safety of the suggestions provided to the responders. The experimental results show that the developed cognitive assistant achieves an average top-3 accuracy of 89% in selecting the correct EMS protocols and an average F1-score of 71% in suggesting the protocol specific interventions while providing transparency and evidence for the suggestions.

Continuum robots composed of elastic backbones have a broad application prospect in the narrow and restricted environment because they overcome the disadvantages of traditional articulated robots, such as being bulky and inflexible. Statics plays an important role in the planning and control of the continuum robot composed of the elastic backbone. Pseudo-Rigid Body (PRB) theory has shown great potential in the description of flexible body statics. The PRB 3R model accurately describes the large deformation of the flexible body and has high computational efficiency. However, PRB 3R models mostly focus on the planar static modeling, and there are few applications in three-dimensional (3D) statics. In this paper, a 3D static modeling method of cable-driven continuum robot based on PRB 3R theory is proposed. By introducing the equilibrium constraint equations of resultant force/moment and bending plane normal of the elastic backbone, the state of the continuum robot is determined. The 3D static equations established by the proposed method take into account the comprehensive effects of the elastic force, external force, gravity and friction. A static verification experiment system of the cable-driven continuum robot is designed to verify the proposed method. The accuracy of the proposed method is verified by comparison with experimental data. The maximum position error between simulation and experimental results is 7.6%.

In this paper a hybrid method for estimation of 6 degree-of-freedom (DOF) laparoscopic instrument tip force/torques in robotic-assisted minimally invasive surgery systems is proposed. The method is implemented on an in-house developed hyper-redundant (11-DOF) surgical robotic forceps prototype. This surgical robot is composed of two modules with a 7-DOF Kuka IIWA manipulator for performing 4-DOF Remote Centre of Motion (RCM) about the trocar and an articulated 4-DOF parallel wrist/gripper mechanism attached to the distal end of the Kuka robot for intra-corporeal manipulation. The hybrid sensor-based/sensorless method fuses torque estimates from the parallel wrist mechanism with measurements from a force sensor attached between the wrist base and the Kuka tool frame. With this approach it is possible to obtain force/torque estimates on all Cartesian axes (x, y, z, yaw, pitch, roll) of the forceps tip. The prototype is one of the first systems designed for robotic surgery that can achieve accurate force estimation on all 6 axes of the instrument tip. Experiment validation results with a force sensor verify the efficacy of the proposed method.

Omnidirectional catadioptric systems are popular in robotic applications thanks to their large field of view. For 3D scene reconstruction in a single shot, usually two different catadioptric cameras are needed. More cameras may contribute to better reconstruction while larger mounting space and higher power cost are required. In this paper, a single camera multi-stereo catadioptric system with vertical and horizontal baseline structure is proposed. It features achieving multi-pair of central or non-central omnidirectional stereos in a compact manner. To make the 3D reconstruction process general and adaptive to various types of system configurations, a flexible calibration and reconstruction algorithm pipeline is presented. The algorithm features approximating the system into multiple central sub-cameras and carrying out the stereo matching in a spherical representation. In addition, an effective 3D point cloud fusion algorithm is proposed to optimize the reconstruction results from multiple stereo pairs. The experiment carried out with synthetic and real data verified the feasibility and effectiveness of our system.

3D printing in medical technology is mainly used to produce customized devices for surgical planning or surgical guides and templates. However, for laparoscopic systems, 3D printing is hardly used. A challenge is the transmission of the necessary forces required in laparoscopic surgery.We designed and manufactured a 3D printed customizable manipulator system for Single Incision Laparoscopic Surgery (SILS), fulfilling the force-related requirements in laparoscopic surgery.The Single Incision Laparoscopic Manipulator System can apply forces at its tip up to 5.6 N with acceptable operating forces not exceeding 30 N. We showed the functionality of the manipulator system in a manipulation experiment in comparison to standard SILS instruments.

Obtaining accurate data about the environment in which a robot is located is a crucial matter when it comes to autonomous navigation and other robotic applications. A popular method of acquiring this information is to use sonar-rings, where a robot is fitted with multiple simple ultrasound transducers pointed in the directions where an object can appear. However, in a time where accurate 3D data is gaining importance, other sensing modalities are becoming more popular because of the ability to measure dense 3D point clouds. In these point clouds not only the horizontal plane is measured, but objects in the elevation planes can also be registered, which can be very interesting and makes applications such as 3D SLAM or object recognition possible. In this paper we present a way to extract complex 3D point cloud data from the entire surrounding sphere using multiple interconnected eRTIS sensors. These advanced imaging sonar sensors offer the flexibility of the popular sonar-ring in combination with the benefits of some of the competing sensing modalities. The setup presented here uses less sonar sensors (and thus less external hardware) while obtaining more information from the complete frontal hemispheres of each individual sensor. This setup is discussed, along with the issues that arise when using complex imaging sonar sensors in a network, and is tested in an indoor and outdoor environment. At the end of this paper is a discussion of the obtained results.

In this paper we present a novel approach termed 3D Move to See (3DMTS) which is based on the principle of finding the next best view using a 3D camera array and a robotic manipulator to obtain multiple samples of the scene from different perspectives. Distinct from traditional visual servoing and next best view approaches, the proposed method uses simultaneously-captured multiple views, scene segmentation and an objective function applied to each perspective to estimate a gradient representing the direction of the next best view in a “single shot”. The method is demonstrated within simulation and on a real robot containing a custom 3D camera array for the challenging scenario of robotic harvesting in a highly occluded and unstructured environment. We show, on a real robotic platform, that by moving the eye-in-hand camera using the gradient of an objective function leads to a locally optimal view of the object of interest, even amongst occlusions. The overall performance of the 3DMTS approach obtains a mean increase in target size of 29.3% compared to a baseline method using a single RGB-D camera, which obtained 9.17%. The results demonstrate qualitatively and quantitatively that the 3DMTS method performed better in most scenarios, and yielded three times the target size compared to the baseline method. Increasing the target size in the image given occlusions can improve robotic systems detecting key object features for further manipulation tasks, such as grasping and harvesting.

The complementary characteristics of active and passive depth sensing techniques motivate the fusion of the LiDAR sensor and stereo camera for improved depth perception. Instead of directly fusing estimated depths across LiDAR and stereo modalities, we take advantages of the stereo matching network with two enhanced techniques: Input Fusion and Conditional Cost Volume Normalization (CCVNorm) on the LiDAR information. The proposed framework is generic and closely integrated with the cost volume component that is commonly utilized in stereo matching neural networks. We experimentally verify the efficacy and robustness of our method on the KITTI Stereo and Depth Completion datasets, obtaining favorable performance against various fusion strategies. Moreover, we demonstrate that, with a hierarchical extension of CCVNorm, the proposed method brings only slight overhead to the stereo matching network in terms of computation time and model size.

This paper presents a three-degrees-of-freedom (DOF) gravity compensation mechanism for the waists of cooperative robots whose centers of mass change according to the arm poses. It comprises 2-DOF and 1-DOF gravity compensation mechanisms in series, and is capable of natural and agile motions such as bobbing and weaving. The 2-DOF mechanism for the upper part of the waist acts as a universal joint. To compensate for the gravity of this 2-DOF motion, a simple and robust link mechanism with one compressive spring is proposed. Moreover, for the complete gravity compensation regardless of the arm poses, a three-dimensional (3D) parallel link structure that maintains the upper body parallel to the ground is introduced. The 1-DOF gravity compensation mechanism, which provides the motions of the hip and thigh, also has a parallelogram structure to maintain the waist parallel to the ground. The implemented 3-DOF waist mechanism can compensate for up to 23 kg and its own weight. It has an adjusting mechanism that can easily calibrate the amount of compensation in case the weight of the upper body changes. The large workspace enables the upper body to move in any direction in a 3D space without singularity. Therefore, the robot with this waist can touch the ground to pick up objects on the ground as well as reach an object at a height of 2145 mm.

In this paper a force controller via energy tanks is implemented for novel applications in aerial contour follow. This control approach allows the aerial vehicle to trace out a boundary whilst in continuous contact with a surface by means of an actively compliant manipulator. This represents the first step towards tactile-based aerial navigation, which can be used to complement more traditional mapping approaches such as visual SLAM. Key results show that the energy-based approach can be used to apply a continuous shear force through the manipulator while the vehicle remains in contact with the surface of interest. Results also show the robustness and repeatability of this approach for prolonged aerial interaction, and the potential for future use in more complex, un-modeled environments.

Visual localization has attracted considerable attention due to its low-cost and stable sensor, which is desired in many applications, such as autonomous driving, inspection robots and unmanned aerial vehicles. However, current visual localization methods still struggle with environmental changes across weathers and seasons, as there is significant appearance variation between the map and the query image. The crucial challenge in this situation is that the percentage of outliers, i.e. incorrect feature matches, is high. In this paper, we derive minimal closed form solutions for 3D-2D localization with the aid of inertial measurements, using only 2 point matches or 1 point match and 1 line match. These solutions are further utilized in the proposed 2-entity RANSAC, which is more robust to outliers as both line and point features can be used simultaneously and the number of matches required for pose calculation is reduced. Furthermore, we introduce three feature sampling strategies with different advantages, enabling an automatic selection mechanism. With the mechanism, our 2-entity RANSAC can be adaptive to the environments with different distribution of feature types in different segments. Finally, we evaluate the method on both synthetic and real-world datasets, validating its performance and effectiveness in inter-session scenarios.