IROS 2020 Papers

Variable impedance control is advantageous for physical human-robot interaction to improve safety, adaptability and many other aspects. This paper presents a gain-scheduled variable stiffness control approach under strict frequency-domain constraints. Firstly, to reduce conservativeness, we characterize and constrain the impedance rendering, actuator saturation, disturbance/noise rejection and passivity requirements into their specific frequency bands. This relaxation makes sense because of the restricted frequency properties of the interactive robots. Secondly, a gain-scheduled method is taken to regulate the controller gains with respect to the desired stiffness. Thirdly, the scheduling function is parameterized via a nonsmooth optimization method. Finally, the proposed approach is validated by simulations, experiments and comparisons with a gain-fixed passivity-based PID method.

This paper presents the results of the experimental tests performed to validate the functionality of a variable pitch system (VPS), designed for pitch attitude control of the novel underwater robotic vehicle explorer UX-1. The VPS is composed of a mass suspended from a central rod mounted across the hull. This mass is rotated around the transverse axis of the vehicle in order to perform a change in the inclination angle for navigation in vertical mine shafts. In this work, the equations of motion are first derived with a quaternion attitude representation, and are then extended to include the dynamics of the VPS. The performance of the VPS is demonstrated in real underwater experimental tests that validate the pitch angle control independently, and coupled with the heave motion control system.

Performing various in-hand manipulation tasks, without learning each individual task, would enable robots to act more versatile, while reducing the effort for training. However, in general it is difficult to achieve stable in-hand manipulation, because the contact state between the fingertips becomes difficult to model, especially for a robot hand with anthropomorphically shaped fingertips. Rich tactile feedback can aid the robust task execution, but on the other hand it is challenging to process high-dimensional tactile information. In the current paper we use two fingers of the Allegro hand, and each fingertip is anthropomorphically shaped and equipped not only with 6-axis force-torque (F/T) sensors, but also with uSkin tactile sensors, which provide 24 tri-axial measurements per fingertip. A convolutional neural network is used to process the high dimensional uSkin information, and a long short-term memory (LSTM) handles the time-series information. The network is trained to generate two different motions ("twist" and "push"). The desired motion is provided as a task-parameter to the network, with twist defined as -1 and push as +1. When values between -1 and +1 are used as the task parameter, the network is able to generate untrained motions in-between the two trained motions. Thereby, we can achieve multiple untrained manipulations, and can achieve robustness with high-dimensional tactile feedback.

The properties and applications of auxetics have been widely explored in the past years. Through proper utilization of auxetic structures, designs with unprecedented mechanical and structural behaviors can be produced. Taking advantage of this, we present the development of novel and low-cost 3D structures inspired by a simple auxetic unit. The core part, which we call the body in this paper, is a 3D realization of 2D rotating squares. This body structure was formed by joining four similar structures through softer material at the vertices. A monolithic structure of this kind is accomplished through a custom-built multi-material 3D printer. The model works in a way that, when torque is applied along the face of the rotational squares, they tend to bend at the vertex of the softer material, and due to the connected-ness of the design, a proper opening and closing motion is achieved. To demonstrate the potential of this part as an important component for robots, two applications are presented: a soft gripper and a crawling robot. Vacuum-driven actuators move both the applications. The proposed gripper combines the benefits of two types of grippers whose fingers are placed parallel and equally spaced to each other, in a single design. This gripper is adaptable to the size of the object and can grasp objects with large and small cross- sections alike. A novel bending actuator, which is made of soft material and bends in curvature when vacuumed, provides the grasping nature of the gripper. Crawling robots, in addition to their versatile nature, provide a better interaction with humans. The designed crawling robot employs negative pressure-driven actuators to highlight linear and turning locomotion.

The field of soft robotics has evolved as a domain for developing light, compliant and safe actuators. However, one of the challenges in the field is the lack of repeatable fabrication techniques as well as customizability that restricts the application of soft robots. We present a fabrication technique using sacrificial molding to fabricate pneumatic channels that are repeatable and less prone to variability. This technique enables the monolithic fabrication of actuators which eliminates conventional failure modes such as delamination. We then use embedded endostructures manufactured using Fused Deposition Modelling (FDM) 3D printers to customize the behavior of bending actuator by altering local mechanical characteristics. Finite element analysis (FEA) was used as a tool to tune the choice of materials and the geometry of the 3D printed layers based on the required application. We analyze the effect of the mechanical properties of the endostructures on actuator behavior and its utility in improving customizability. We analyzed the behavior of actuators with a variety of endostructures using visual markers. As predicted by the FEA and Euler-Bernoulli beam theory, the behavior of the actuators was seen to be influenced by the mechanical properties of the endostucture. Thus, we present a new methodology for tuning the mechanical properties of Soft Pneumatic Actuators (SPAs), which is simple and efficient to predict as well as easy to execute.

The ability of detecting materials plays an important role in robotic applications. The robot can incorporate the information from contactless material detection and adapt its behavior in how it grasps an object or how it walks on specific surfaces. In this, paper we apply machine learning on impedance spectra from capacitive proximity sensors for material detection. The unique spectra of certain materials only differ slightly and are subject to noise and scaling effects during each measurement. A best-fit classification approach to pre-recorded data is therefore inaccurate. We perform classification on ten different materials and evaluate different classification algorithms ranging from simple k-NN approaches to artificial neural networks, which are able to extract the material specific information from the impedance spectra.

In recent years, human pose estimation has seen great improvements by the use of neural networks. However, these approaches are unsuitable for safety-critical applications such as human-robot interaction (HRI), as no guarantees are given whether a produced detection is correct or not and false detections with high confidence scores are produced on a regular basis. In this work, we propose a method to identify and eliminate false detections by comparing keypoint detections from different neural networks and assigning a 'Don't know' label in the case of a mismatch. Our approach is driven by the principle of software diversity, a technique recommended by the safety standard IEC 61508-7 [1] for dealing with software implementation faults. We evaluate our general concept on the MPII human pose dataset [2] using available ground truth data to calculate a suitable threshold for our keypoint comparison, reducing the number of false detections by approx. 61%. For the application at runtime, where no ground truth data is available, we introduce a method to calculate the needed threshold directly from keypoint detections. In further experiments, it was possible to reduce the number of false detections by approx. 75%. Eliminating keypoints by comparison also lowers the correct detection rate, which we maintained above 75% in all experiments. As this effect is limited and non-critical regarding safety we believe that the proposed approach can lead the way to a safe use of neural networks for human pose estimation in the future.

We present an unsupervised learning pipeline for dense depth, optical flow and egomotion estimation for autonomous driving applications, using the event-based output of the Dynamic Vision Sensor (DVS) as input. The backbone of our pipeline is a bioinspired encoder-decoder neural network architecture - ECN. To train the pipeline, we introduce a covariance normalization technique which resembles the lateral inhibition mechanism found in animal neural systems.Our work is the first monocular pipeline that generates dense depth and optical flow from sparse event data only, and is able to transfer from day to night scenes without any additional training. The network works in self-supervised mode and has just 150k parameters. We evaluate our pipeline on the MVSEC self driving dataset and present results for depth, optical flow and and egomotion estimation. Thanks to the efficient design, we are able to achieve inference rates of 300 FPS on a single Nvidia 1080Ti GPU. Our experiments demonstrate significant improvements upon works that used deep learning on event data, as well as the ability to perform well during both day and night.

Crops are an important source of food and other products. In conventional farming, tractors apply large amounts of agrochemicals uniformly across fields for weed control and plant protection. Autonomous farming robots have the potential to provide environment-friendly weed control on a per plant basis. A system that reliably distinguishes crops, weeds, and soil under varying environment conditions is the basis for plant-specific interventions such as spot applications. Such semantic segmentation systems, however, often show a performance decay when applied under new field conditions. In this paper, we therefore propose an effective approach to unsupervised domain adaptation for plant segmentation systems in agriculture and thus to adapt existing systems to new environments, different value crops, and other farm robots. Our system yields a high segmentation performance in the target domain by exploiting labels only from the source domain. It is based on CycleGANs and enforces a semantic consistency domain transfer by constraining the images to be pixel-wise classified in the same way before and after translation. We perform an extensive evaluation, which indicates that we can substantially improve the transfer of our semantic segmentation system to new field environments, different crops, and different sensors or robots.

In this paper, we propose an unsupervised deep learning framework with Bayesian inference for improving the accuracy of per-pixel depth prediction from monocular RGB images. The proposed framework predicts confidence map along with depth and pose information for a given input image. The depth hypotheses from previous frames are propagated forward and fused with the depth hypothesis of the current frame by using Bayesian inference mechanism. The ground truth information required for training the confidence map prediction is constructed using image reconstruction loss thereby obviating the need for explicit ground truth depth information used in supervised methods. The resulting unsupervised framework is shown to outperform the existing state-of-the-art methods for depth prediction on the publicly available KITTI outdoor dataset. The usefulness of the proposed framework is further established by demonstrating a real-world robotic pick-and-place application where the pose of the robot end-effector is computed using the depth predicted from an eye-in-hand monocular camera. The design choices made for the proposed framework is justified through extensive ablation studies.

This work uses quadratic programming to perform torque control on an industrial collaborative robot, while keeping defined constraints. Limits for rotational and translational coordinates are considered at position, velocity and acceleration level. Although the problem of having hardware and safe limitations has been considered before. Solutions usually rely on functions that need a proper tuning. The proposed control scheme is tested to work on a real robot to avoid not only static but also dynamic obstacles without the need of any empirical tuning. The method is tested also under physical human robot interaction (pHRI) showing smooth behaviour of the robot despite of external forces.

In this paper, we propose a unified calibration method for multi-camera multi-LiDAR systems. Only using a single planar checkerboard, the captured checkerboard frames by each sensor are classified as either global frames if they are observed by at least two sensors, or a local frame if observed by a single camera. Both global and local frames of each camera are used to estimate its intrinsic parameters, whereas the global frames between sensors are for computing their relative poses. In contrast to the previous methods that simply combine the pairwise poses (e.g., camera-to-camera or camera-to-LiDAR) that are separately estimated, we further optimize the sensor poses in the system globally using all observations as the constraints in the optimization problem. We find that the point-to-plane distances are effective as camera-to-LiDAR constraints where the points are 3D positions of the checkerboard corners and the planes are estimated from the LiDAR point-cloud. Also, abundant corner observations in the local frames enable the joint optimization of intrinsic and extrinsic parameters in a unified framework. The proposed calibration method utilizes entire observations in a unified global optimization framework, and it significantly reduces the error caused by a simple composition of the relative sensor poses. We extensively evaluate the proposed algorithm qualitatively and quantitatively using real and synthetic datasets. We plan to make the implementation open to the public with the paper publication.

This paper proposes a methodology for real-time depth estimation of underwater monocular camera images, fusing measurements from a single-beam echosounder. Our system exploits the echosounder's detection cone to match its measurements with the detected feature points from a monocular SLAM system. Such measurements are integrated in a monocular SLAM system to adjust the visible map points and the scale. We also provide a novel calibration process to determine the extrinsic between camera and echosounder to have reliable matching. Our proposed approach is implemented within ORB-SLAM2 and evaluated in a swimming pool and in the ocean to validate image depth estimation improvement. In addition, we demonstrate its applicability for improved underwater color correction. Overall, the proposed sensor fusion system enables inexpensive underwater robots with a monocular camera and echosounder to correct the depth estimation and scale in visual SLAM, leading to interesting future applications, such as underwater exploration and mapping.

We present a novel Multi-Relational Graph Convolutional Network (MRGCN) based framework to model on-road vehicle behaviors from a sequence of temporally ordered frames as grabbed by a moving monocular camera. The input to MRGCN is a multi-relational graph where the graph's nodes represent the active and passive agents/objects in the scene, and the bidirectional edges that connect every pair of nodes are encodings of their Spatio-temporal relations.We show that this proposed explicit encoding and usage of an intermediate spatio-temporal interaction graph to be well suited for our tasks over learning end-end directly on a set of temporally ordered spatial relations. We also propose an attention mechanism for MRGCNs that conditioned on the scene dynamically scores the importance of information from different interaction types.The proposed framework achieves significant performance gain over prior methods on vehicle-behavior classification tasks on four datasets. We also show a seamless transfer of learning to multiple datasets without resorting to fine-tuning. Such behavior prediction methods find immediate relevance in a variety of navigation tasks such as behavior planning, state estimation, and applications relating to the detection of traffic violations over videos.

Manipulation tasks in daily life, such as pouring water, unfold through human intentions. Being able to process contextual knowledge from these Activities of Daily Living (ADLs) over time can help us understand manipulation intentions, which are essential for an intelligent robot to transition smoothly between various manipulation actions. In this paper, to model the intended concepts of manipulation, we present a vision dataset under a strictly constrained knowledge domain for both robot and human manipulations, where manipulation concepts and relations are stored by an ontology system in a taxonomic manner. Furthermore, we propose a scheme to generate a combination of visual attentions and an evolving knowledge graph filled with commonsense knowledge. Our scheme works with real-world camera streams and fuses an attention-based Vision-Language model with the ontology system. The experimental results demonstrate that the proposed scheme can successfully represent the evolution of an intended object manipulation procedure for both robots and humans. The proposed scheme allows the robot to mimic human-like intentional behaviors by watching real-time videos. We aim to develop this scheme further for real-world robot intelligence in Human-Robot Interaction.

3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking. Project videos and code are at https://jianrenw.github.io/Self-Supervised-3D-Data-Association/.

Spatial mapping of surface roughness is a critical enabling technology for automating adaptive sanding operations. We leverage GelSight sensors to convert the problem of surface roughness measurement into a vision classification problem. By combining GelSight sensors with Optitrack positioning systems we attempt to develop an accurate spatial mapping of surface roughness that can compare to human touch, the current state of the art for large scale manufacturing. To perform the classification, we propose the use of Bayesian neural networks in conjunction with uncertainty-aware prediction. We compare the sensor and network with a human baseline for both absolute and relative texture classification. To establish a baseline, we collected performance data from humans on their ability to classify materials into 60, 120, and 180 grit sanded pine boards. Our results showed that the probabilistic network performs at the level of human touch for absolute and relative classifications. Using the Bayesian approach enables establishing a confidence bound on our prediction. We were able to integrate the sensor with Optitrack to provide a spatial map of sanding grit applied to pine boards. From this result, we can conclude that GelSight with Bayesian neural networks can learn accurate representations for sanding, and could be a significant enabling technology for closed loop robotic sanding operations.

In classical computer vision, rectification is an integral part of multi-view depth estimation. It typically includes epipolar rectification and lens distortion correction. This process simplifies the depth estimation significantly, and thus it has been adopted in CNN approaches. However, rectification has several side effects, including a reduced field of view (FOV), resampling distortion, and sensitivity to calibration errors. The effects are particularly pronounced in case of significant distortion (e.g., wide-angle fisheye cameras). In this paper, we propose a generic scale-aware self-supervised pipeline for estimating depth, euclidean distance, and visual odometry from unrectified monocular videos. We demonstrate a similar level of precision on the unrectified KITTI dataset with barrel distortion comparable to the rectified KITTI dataset. The intuition being that the rectification step can be implicitly absorbed within the CNN model, which learns the distortion model without increasing complexity. Our approach does not suffer from a reduced field of view and avoids computational costs for rectification at inference time. To further illustrate the general applicability of the proposed framework, we apply it to wide-angle fisheye cameras with 190° horizontal field of view. The training framework UnRectDepthNet takes in the camera distortion model as an argument and adapts projection and unprojection functions accordingly. The proposed algorithm is evaluated further on the KITTI rectified dataset, and we achieve state-of-the-art results that improve upon our previous work FisheyeDistanceNet [1]. Qualitative results on a distorted test scene video sequence indicate excellent performance1.

In this paper we introduce the first reinforcement learning (RL) based robotic navigation method which utilizes ultrasound (US) images as an input. Our approach combines state-of-the-art RL techniques, specifically deep Q-networks (DQN) with memory buffers and a binary classifier for deciding when to terminate the task.Our method is trained and evaluated on an in-house collected data-set of 34 volunteers and when compared to pure RL and supervised learning (SL) techniques, it performs substantially better, which highlights the suitability of RL navigation for US-guided procedures. When testing our proposed model, we obtained a 82.91% chance of navigating correctly to the sacrum from 165 different starting positions on 5 different unseen simulated environments.