IROS 2019 Papers

This work presents the design, modeling, and fabrication of a whisker-like sensor capable of measuring the whisker's angular displacement as well as the applied moments at the base of the whisker. The sensor takes advantage of readily accessible and low-cost 3D magnetic sensors to transduce whisker deflections, and a planar serpentine spring structure at the whisker base is used to provide a mechanical suspension for the whisker to rotate. The sensor prototype was characterized, calibrated, and compared with analytical models of the spring system and the magnetic field. The prototype showed a moment sensing range of 1.1N·mm when deflected up to 19.7°. The sensitivity of the sensor was 0.38°/LSB for the angular displacement sensing, and 0.021 Nmm/LSB for the moment sensing. A fully integrated system is demonstrated to display real-time information from the whisker on a graphical interface.

Accurate master-slave control is important for Robot-Assisted Microsurgery (RAMS). This paper presents a handheld master controller for the operation and training of RAMS. A 9-axis Inertial Measure Unit (IMU) and a micro camera are utilized to form the sensing system for the handheld controller. A new hybrid marker pattern is designed to achieve reliable visual tracking, which integrated QR codes, Aruco markers, and chessboard vertices. Real-time multi-sensor fusion is implemented to further improve the tracking accuracy. The proposed handheld controller has been verified on an in-house microsurgical robot to assess its usability and robustness. User studies were conducted based on a trajectory following task, which indicated that the proposed handheld controller had comparable performance with the Phantom Omni, demonstrating its potential applications in microsurgical robot control and training.

Underwater active landmarks (UALs) are widely used for short-range underwater navigation in underwater robotics tasks. Detection of UALs is challenging due to large variance of underwater illumination, water quality and change of camera viewpoint. Moreover, improvement of detection accuracy relies upon statistical diversity of images used to train detection models. We propose a generative adversarial network, called Tank-to-field GAN (T2FGAN), to learn generative models of underwater images, and use the learned models for data augmentation to improve detection accuracy. To this end, first a T2FGAN is trained using images of UALs captured in a tank. Then, the learned model of the T2FGAN is used to generate images of UALs according to different water quality, illumination, pose and landmark configurations (WIPCs). In experimental analyses, we first explore statistical properties of images of UALs generated by T2FGAN under various WIPCs for active landmark detection. Then, we use the generated images for training detection algorithms. Experimental results show that training detection algorithms using the generated images can improve detection accuracy. In field experiments, underwater docking tasks are successfully performed in a lake by employing detection models trained on datasets generated by T2FGAN.

This paper presents a gear-driven prosthetic hand designed for toddlers with transradial amputation. The hand design considers three main issues: weight, cost, and operability. The prosthetic hand and the cosmetic silicon glove are made based on the dimensions of a real hand. The simple, stable, and reliable gear-driven transmission helps to reduce the weight and the cost. A small actuator is embedded in the palm. During the grasp, the four fingers and the thumb flexes and extends as a unit to provide a wide range of holding area. The kinematics and static analysis in grasping was performed and the simulation results were compared with measured data. The motion performance and practical operability of the proposed hand was verified experimentally by a test system and a transradial subject.

The visual-inertial navigation system (VINS) has been a practical approach for state estimation in recent years. In this paper, we propose a general GPS-aided omnidirectional visual-inertial state estimator capable of operating in ubiquitous environments and platforms. Our system consists of two parts: 1) the pre-processing of omnidirectional cameras, IMU, and GPS measurements, and 2) the sliding window based nonlinear optimization for accurate state estimation. We test our system in different conditions including an indoor office, campus roads, and challenging open water surface. Experiment results demonstrate the high accuracy of our approach than state-of-the-art VINSs in all scenarios. The proposed odometry achieves drift ratio less than 0.5% in 1200 m length outdoors campus road in overexposure conditions and 0.65% in open water surface, without a loop closure, compared with a centimeter accuracy GPS reference.

We present a fully-integrated sensing and control system which enables mobile manipulator robots to execute building tasks with millimeter-scale accuracy on building construction sites. The approach leverages multi-modal sensing capabilities for state estimation, tight integration with digital building models, and integrated trajectory planning and whole-body motion control. A novel method for high-accuracy localization updates relative to the known building structure is proposed. The approach is implemented on a real platform and tested under realistic construction conditions. We show that the system can achieve sub-cm end-effector positioning accuracy during fully autonomous operation using solely onboard sensing.

The 3D depth estimation and relative pose estimation problem within a decentralized architecture is a challenging problem that arises in missions that require coordination among multiple vision-controlled robots. The depth estimation problem aims at recovering the 3D information of the environment. The relative localization problem consists of estimating the relative pose between two robots, by sensing each other's pose or sharing information about the perceived environment. Most solutions for these problems use a set of discrete data without taking into account the chronological order of the events. This paper builds on recent results on continuous estimation to propose a framework that estimates the depth and relative pose between two non-holonomic vehicles. The basic idea consists in estimating the depth of the points by explicitly considering the dynamics of the camera mounted on a ground robot, and feeding the estimates of 3D points observed by both cameras in a filter that computes the relative pose between the robots. We evaluate the convergence for a set of simulated scenarios and show experimental results validating the proposed framework.

The handling of liquid filled containers is a challenging task in industrial manufacturing processes. Without an appropriate trajectory sloshing of the liquid occurs during point-to-point motion. This paper presents a time optimal trajectory planning approach that is used for feedforward motion control and prevents the liquid filled container from sloshing. For this, a model of the liquid inside the container is included into the formulation of a constrained dynamic optimization problem to achieve rest-to-rest motion while simultaneously avoiding obstacles. The presented approach is validated by simulation and on an experimental setup using an industrial robot handling the container.

Modern agricultural robots like drones have been studied in automatic yield estimation in recent years. Fruit counting is a fundamental task in the automatic yield estimation, on which significant progress has been achieved by detection-based methods and segmentation-regression-based methods. However, for clustered-fruit counting, the existing methods lack advantages on the localization of small and occluded fruits or discrete number regression. In addition, it is observed that existing deep neural network based counting methods have high variances on fruit density map estimation. Aiming at solving these two problems and decreasing the regression variance, in this paper, we propose a density-map-estimation model with DropBlock regularization. For evaluating the proposed model, we propose a new Clustered-Fruit dataset. Extensive experiments show that the proposed model is effective and outperforms the state-of-the-art counting methods on the Clustered-Fruit dataset. Our dataset is available at Clustered-Fruit.

For an autonomous corridor following task where the environment is continuously changing, several forms of environmental noise prevent an automated feature extraction procedure from performing reliably. Moreover, in cases where pre-defined features are absent from the captured data, a well defined control signal for performing the servoing task fails to get produced. In order to overcome these drawbacks, we present in this work, using a convolutional neural network (CNN) to directly estimate the required control signal from an image, encompassing feature extraction and control law computation into one single end-to-end framework. In particular, we study the task of autonomous corridor following using a CNN and present clear advantages in cases where a traditional method used for performing the same task fails to give a reliable outcome. We evaluate the performance of our method on this task on a Wheelchair Platform developed at our institute for this purpose.

In this work, we tackle the problem of child engagement estimation while children freely interact with a robot in a friendly, room-like environment. We propose a deep learning-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed regularly in the room, fuse the results and feed them to a deep Neural Network (NN) trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next-generation social robots as an attention monitoring tool during various Child Robot Interaction (CRI) tasks both for Typically Developed (TD) children and children affected by autism (ASD).

We address the problem of developing precision, quasi-static control strategies for fingertip manipulation in robot hands. In general, analytically specifying useful object transition maps, or hand-object Jacobians, for scenarios in which there is uncertainty in some key aspect of the hand-object system is difficult or impossible. This could be in scenarios with standard fully-actuated hands where, for instance, there is not an accurate model of the contact conditions, or in scenarios with fewer control inputs than mechanical degrees of freedom (such as underactuated hands or those that are controlled by synergies or impedance controlled frameworks), since the output space is of higher dimension than the input space. In this work, we develop a method for extracting object transition maps by tracking the state of the grasp frame. We begin by modeling a compliant, underactuated hand and its mechanical properties through an energy-based approach. From this energy model, we provide controlled actuation inputs to change the state of the grasp frame. We observe the response from these actions and develop a regression map of the action-reaction pairs, where the map is subject to our intent for grasp frame movement and the regional relationship between the contacts. Once the regression model is developed, we perform within-hand planning of the grasp frame with newly introduced objects. This approach is agnostic to the global geometry of the object and is able to adapt when undesirable contact conditions, such as sliding, occur. The learning-based methodology estimates the non-linearities representative in the properties of the system. We test our framework physically on an adapted Yale Openhand Model O. By transferring the learned model from simulation to the physical hand without adaptation, we show that this energy modeling approach is robust to inaccuracies in parameter estimation. We demonstrate its efficacy in a handwriting task.

Object detection, extended to recognize and localize indoor structural features, is used to enable a quadrotor drone to autonomously navigate through indoor environments. The video stream from a monocular front-facing camera on-board a quadrotor drone is fed to an off-board system that runs a Convolutional Neural Network (CNN) object detection algorithm to identify specific features such as dead-ends, doors, and intersections in hallways. Using pixel-scale dimensions of the bounding boxes around the recognized objects, the distance to intersections, dead-ends and doorways can be estimated accurately using a Support Vector Regression (SVR) model to generate flight control commands for consistent real-time autonomous navigation at flight speeds approaching 2 m/s.

In this paper, we focus on a rain removal task from a single image of the urban street scene for autonomous driving in rain. We develop a Convolutional Neural Network which takes a rainy image as input, and directly recovers a clean image in the presence of rain streaks, atmospheric veiling effect (haze, fog, mist) caused by distant rain streak accumulation. We propose a synthetic dataset containing images of urban street scenes with different rain intensities, orientations and haziness levels for training and evaluation. We evaluate our method quantitatively and qualitatively on the synthetic data. Experiments show that our model outperforms state-of-the-art methods. We also test our method qualitatively on the real-world data. Our model is fast and it takes 0.05s for an image of 1024 × 512. Our model can be seamlessly integrated with existing image-based high-level perception algorithms for autonomous driving in rain. Experiment results show that our deraining method improves semantic segmentation and object detection largely for autonomous driving in rain.

Caging is a promising tool which allows a robot to manipulate an object without directly reasoning about the contact dynamics involved. Furthermore, caging also provides useful guarantees in terms of robustness to uncertainty, and often serves as a way-point to a grasp. However, caging is traditionally difficult to integrate as part of larger manipulation frameworks, where caging is not the goal but an intermediate condition. In this paper, we develop a convex-combinatorial model to characterize caging from an optimization perspective. More specifically, we derive a set of sufficient constraints to enclose the configuration of the object in a compact-connected component of its free-space. The convex-combinatorial nature of this approach provides guarantees on optimality and convergence, and its optimization nature makes it versatile for further applications on robot manipulation tasks. To the best of our knowledge, this is the first optimization-based approach to formulate the caging condition.

Visual Servoing has been widely investigated in the last decades as it provides a powerful strategy for robot control. Thanks to the direct feed-back from a set of sensors, it allows to reduce the impact of some modeling errors and to perform tasks even in uncertain environments. The commonly exploited approach in this field is to use a model that expresses the rate of change of a set of features as a function of sensor twist. These schemes are commonly used to obtain a velocity command, which needs to be tracked by a low-level controller. Another approach that can be exploited consists in going one step further and to consider an acceleration model for the features. This strategy allows also to obtain a natural and direct link with the dynamic model of the controlled system. This study aims at comparing the use of velocity and acceleration-based models in feed-back linearization for Visual Servoing. We consider the case of a redundant manipulator and discuss what this implies for both control techniques. By means of simulations, we show that controllers based on features acceleration give better results than those based on velocity in presence of noisy feedback signals.

Designing reinforcement learning (RL) problems that can produce delicate and precise manipulation policies requires careful choice of the reward function, state, and action spaces. Much prior work on applying RL to manipulation tasks has defined the action space in terms of direct joint torques or reference positions for a joint-space proportional derivative (PD) controller. In practice, it is often possible to add additional structure by taking advantage of model-based controllers that support both accurate positioning and control of the dynamic response of the manipulator. In this paper, we evaluate how the choice of action space for dynamic manipulation tasks affects the sample complexity as well as the final quality of learned policies. We compare learning performance across three tasks (peg insertion, hammering, and pushing), four action spaces (torque, joint PD, inverse dynamics, and impedance control), and using two modern reinforcement learning algorithms (Proximal Policy optimization and Soft Actor-Critic). Our results lend support to the hypothesis that learning references for a task-space impedance controller significantly reduces the number of samples needed to achieve good performance across all tasks and algorithms.

While robots are more and more deployed among people in public spaces, the impact of cyber-security attacks is significantly increasing. Most of consumer and professional robotic systems are affected by multiple vulnerabilities and the research in this field is just started. This paper addresses the problem of automatic detection of anomalous behaviors possibly coming from cyber-security attacks. The proposed solution is based on extracting system logs from a set of internal variables of a robotic system, on transforming such data into images, and on training different Autoencoder architectures to classify robot behaviors to detect anomalies. Experimental results in two different scenarios (autonomous boats and social robots) show effectiveness and general applicability of the proposed method.

Laryngeal cancer treatments, while curative, often lead to voice impairment. Minimally invasive surgical methods that facilitate greater preservation of healthy tissue have recently emerged, but they are still limited in important ways. In this work, we describe a device that combines the advantages of the two primary minimally invasive approaches: the high quality incision and reduced post-operative pain achievable with transoral laser microsurgery and the superior visualization and tissue manipulability afforded by transoral robotic surgery. Our 11 mm diameter scanning system connects to focusing optics and a fiber optic laser source and can direct a laser beam across a 18×10 mm plane with controllable trajectories at speeds up to 7m/s. We describe its design and benchtop validation and present avenues for further development within a clinical environment. While oncological treatment is a natural first application area for this technology, we anticipate that it may also yield important benefits for the minimally invasive treatment of benign laryngeal diseases.