IROS 2021 Papers

Being able to safely operate for extended periods of time in dynamic environments is a critical capability for autonomous systems. This generally involves the prediction and understanding of motion patterns of dynamic entities, such as vehicles and people, in the surroundings. Many motion prediction methods in the literature implicitly account for environmental factors by learning on observed motion in a fixed environment, and are designed to make predictions in the same environment. In this paper, we address the problem of generating likely motion trajectories for novel environments, represented as occupancy grid maps, where motion has not been observed. We introduce the Occupancy-Conditional Trajectory Network (OTNet) framework, capable of transferring the previously observed motion patterns in known environments to new environments. OTNet provides a functional representation for motion trajectories and utilises neural networks to learn occupancy-conditional distributions over the function parameters. We empirically demonstrate our method’s ability to generate complex multi-modal trajectory patterns in both simulated and real-world environments.

Dexterous manipulation tasks often require contact switching, where fingers make and break contact with the object. We propose a method that plans trajectories for dexterous manipulation tasks involving contact switching using contact-implicit trajectory optimization (CITO) augmented with a high-level discrete contact sequence planner. We first use the high-level planner to find a sequence of finger contact switches given a desired object trajectory. With this contact sequence plan, we impose additional constraints in the CITO problem. We show that our method finds trajectories approximately 7 times faster than a general CITO baseline for a four-finger planar manipulation scenario. Furthermore, when executing the planned trajectories in a full dynamics simulator, we are able to more closely track the object pose trajectories planned by our method than those planned by the baselines.

Recently, a new category of bio-inspired legged robots moving directly on the seabed have been proposed to complement the abilities of traditional underwater vehicles and to enhance manipulation and sampling tasks. So far, only tele-operated use of underwater legged robots has been reported and in this paper we attempt to fill such gap by presenting the first step towards autonomous area inspection. First, we present a 3 dimensional single-legged model for underwater hopping locomotion and derive a path following control strategy. Later, we adapt such control strategy to an underwater hexapod robot SILVER2 on the robotic simulator Webots. Finally, we simulate a full autonomous mission consisting in the inspection of an area over a pre-defined path, target recognition, transition to a safer gait and target approach. Our results show the feasibility of the approach and encourage the implementation of the presented control strategy on the robot SILVER2.

We describe a framework for changing-contact robot manipulation tasks, which require the robot to make and break contacts with objects and surfaces. The discontinuous interaction dynamics of such tasks make it difficult to construct and use a single dynamics model or control strategy for such tasks. For any target motion trajectory, our framework incrementally improves its prediction of when contacts will occur. This prediction and a model relating approach velocity to impact force modify the velocity profile of the motion sequence such that it is C∞ smooth, and help achieve a desired force on impact. We implement this framework by building on our hybrid force-motion variable impedance controller for continuous-contact tasks. We evaluate our framework in the illustrative context of a robot manipulator performing sliding tasks involving multiple contact changes with surfaces of different properties.

Elderly and mobility impaired people need special attention during bathing activities, since these tasks are demanding in body flexibility. Our aim is to build an assistive robotic bathing system, in order to increase the independence and safety of this procedure. Towards this end, the expertise of professional carers for bathing sequences and appropriate motions have to be adopted, in order to achieve natural, physical human - robot interaction. In this paper a Navigation Function (NF) approach is proposed in order to reproduce the way an expert clinical carer executes the bathing activities by means of construction repulsive potential fields ("virtual obstacles") for an assistive bath robot. The produced vector field, constructed based on the demonstration procedure, is used for real-time motion behavior planning tasks, which exploits the visual information from Depth sensors and the advantages of the NF approach, to estimate the reference pose for the end- effector of the assistive robotic system. The proposed method guarantees globally asymptotic convergence to the learned from demonstration washing motion, within the deformable and moving body-part limits, while in addition, restricted areas on the body surface are avoided. The proposed method is evaluated using real experimental data, obtained from human subjects during pouring water task demonstration.

Improving robot systems via newly-developed sensing devices, control algorithms, or state estimators in order to obtain safe and efficient human-robot interaction as well as tactile manipulation skills requires standardized performance measurement protocols for objective comparison. Common protocols to evaluate robot motion performance are currently defined in EN ISO 9283:1998. For tactile and safety performance, however, no common metrics were agreed on nor standardized yet. In this paper, we propose a set of quantifiable performance criteria for robot performance analysis, objectifying robot force sensing, force control, and collision detection/reaction performance. We introduce the corresponding measurement setups and protocols, demonstrate and experimentally validate each with a Universal Robot UR10e and UR5e as well as a Franka Emika Panda robot arm. The proposed performance criteria, metrics, and experimental setups constitute the basis of a fully tactile performance and safety benchmarking framework that allows to objectively evaluate tactile robot performance via reproducible reference tests.

There has been an increasing demand to automate the non-patient care matters so that the clinical staff can focus on delivering patient care. For example, out-patients undergoing chemotherapy increases their toilet usage frequency due to the treatment. As they are undergoing chemotherapy, their output waste contains a level of chemical. This task is compulsory yet troublesome and time-consuming so it is often desired to be removed from the nursing staff for them to focus on patient care. Hence, in this paper, we propose a manipulator system to automatically dispose the bedpan used by patients undergoing chemotherapy. The main technical challenge lies in the removal of the bedpan from the commode as the interaction of the grasping is highly dynamic, along with the different conditions of the bedpans. To address this manipulation issue, a Residual Reinforcement Learning (RRL) method that leverages vision-based commode pose estimation and the reinforcement learning (RL)-based uncertainty compensation for improvement of the grasping accuracy is proposed to increase the robustness of the disposal. The experiments conducted show that the manipulator can dispose the bedpan without human intervention and the proposed method achieves a 100 % success rate while the traditional method without RL is only 50 %.

Endoscopic laser tools have been recently proposed in order to overcome the limitations of state-of-the-art laser tools, by integrating fiber-coupled lasers into flexible endoscopic systems. One of the main challenges in designing such endoscopic tools consists in the focusing of the laser, that requires to be frequently adjusted reducing the reliability of the system and increasing surgeons’ mental workload. To avoid these problems, compact auto-focusing tools have been recently developed, taking advantage of MEMS varifocal mirrors (VM) to allow integration with endoscopic tools. In this paper, we integrate such VM-based tool with a distance sensing algorithm based on 3D surface reconstruction to achieve a complete autofocusing system. We evaluate the performance of the proposed integrated system by ablating lines on plaster block targets at variable distance and comparing the obtained ablation depth and width with that of a fixed focus system. Preliminary results show that the proposed system is able to keep the laser in focus resulting in uniform ablation lines for distance ranges from 14mm to 22mm.

This paper proposes a novel neural network-based control policy to enable a mobile robot to navigate safety through environments filled with both static obstacles, such as tables and chairs, and dense crowds of pedestrians. The network architecture uses early fusion to combine a short history of lidar data with kinematic data about nearby pedestrians. This kinematic data is key to enable safe robot navigation in these uncontrolled, human-filled environments. The network is trained in a supervised setting, using expert demonstrations to learn safe navigation behaviors. A series of experiments in detailed simulated environments demonstrate the efficacy of this policy, which is able to achieve a higher success rate than either standard model-based planners or state-of-the-art neural network control policies that use only raw sensor data.

Lumbar injection is an image-guided procedure performed manually for diagnosis and treatment of lower back pain and leg pain. Previously, we have developed and verified an MR-Conditional robotic solution to assisting the needle insertion process. Drawing on our clinical experiences, a virtual remote center of motion (RCM) constraint is implemented to enable our robot to mimic a clinician’s hand motion to adjust the needle tip position in situ. Force and image data are collected to study the needle behavior in gel phantoms during this motion, and a mechanics-based needle-tissue interaction model is proposed and evaluated to further examine the underlying physics. This work extends the commonly-adopted notion of an RCM for flexible needles, and introduces new motion parameters to describe the needle behavior. The model parameters can be tuned to match the experimental result to sub-millimeter accuracy, and this proposed needle manipulation method presents a safer alternative to laterally translating the needle during in situ needle adjustments.

Deep neural networks are the leading solution to the object detection problem. However, challenges arise when applying these networks to the kind of real-time, first-person video data that a robotic platform must process: specifically, detections may not be consistent from frame to frame, and objects may frequently appear at viewpoints that are particularly challenging for the model, resulting in inaccurate detections. In this paper, we present our approach for addressing these challenges for our particular vision problem: diver detection onboard autonomous underwater vehicles (AUVs). We begin by producing and releasing a dataset of approximately 105,000 annotated images of divers sourced from videos in order to address the challenge of learning a wide variety of object rotations and translations. This is one of the largest and most varied diver detection datasets ever created, and we compare models trained and tested on both our dataset and a previous dataset to demonstrate that our dataset improves the state-of-the-art in diver detection. Then, in order to choose an object detection model that produces detections that are consistent from frame to frame, we evaluate several state-of-the-art object detection models on the temporal stability of their detections in addition to the typical accuracy and efficiency metrics, mean average precision (mAP) and frames per second. Importantly, our results showed that models with the highest mAP do not also have the highest temporal stability.

In the future, extraterrestrial expeditions will not only be conducted by rovers but also by flying robots. The technical demonstration drone Ingenuity, that just landed on Mars, will mark the beginning of a new era of exploration unhindered by terrain traversability. Robust self-localization is crucial for that. Cameras that are lightweight, cheap and information-rich sensors are already used to estimate the ego-motion of vehicles. However, methods proven to work in man-made environments cannot simply be deployed on other planets. The highly repetitive textures present in the wastelands of Mars pose a huge challenge to descriptor matching based approaches.In this paper, we present an advanced robust monocular odometry algorithm that uses efficient optical flow tracking to obtain feature correspondences between images and a refined keyframe selection criterion. In contrast to most other approaches, our framework can also handle rotation-only motions that are particularly challenging for monocular odometry systems. Furthermore, we present a novel approach to estimate the current risk of scale drift based on a principal component analysis of the relative translation information matrix. This way we obtain an implicit measure of uncertainty. We evaluate the validity of our approach on all sequences of a challenging real-world dataset captured in a Mars-like environment and show that it outperforms state-of-the-art approaches. The source code is publicly available at: https://github.com/DLR-RM/granit.

Human trajectory prediction has received increased attention lately due to its importance in applications such as autonomous vehicles and indoor robots. However, most existing methods make predictions based on human-labeled trajectories and ignore the errors and noises in detection and tracking. In this paper, we study the problem of human trajectory forecasting in raw videos, and show that the prediction accuracy can be severely affected by various types of tracking errors. Accordingly, we propose a simple yet effective strategy to correct the tracking failures by enforcing prediction consistency over time. The proposed "re-tracking" algorithm can be applied to any existing tracking and prediction pipelines. Experiments on public benchmark datasets demonstrate that the proposed method can improve both tracking and prediction performance in challenging real-world scenarios. The code and data are available at https://git.io/retracking-prediction.

Studies have shown that picking techniques play an important role in determining fruit quality at harvest (e.g. bruising, stem retention, etc). When picking fruit such as apples and pears, professional pickers use active perception, incorporating both visual and tactile input about fruit orientation, stem location, and the fruit’s immediate surroundings. This combination of tactile, visual, and force feedback is what enables human workers to execute dynamic movements that quickly and efficiently remove fruit from the tree without damage. However, much of the prior work on robotic fruit picking has formulated the harvesting problem as a position-control problem, using visual feedback for closed-loop end-effector placement while disregarding feedback on physical contact. As a first step towards more intelligent fruit picking — combining proprioception, localized sensing, and observed forces — we have developed a custom end-effector with multiple in-hand sensors, including tactile sensors on the fingertips. This paper presents the mechatronic design of the device as well as results from multiple outdoor picking trials with a Honeycrisp apple tree. Preliminary results show that, with multi-modal sensing, fruit slip, fruit separation from the tree, and fruit release from the hand can be detected.

Physical human-robot interactions (pHRI) are less efficient and communicative than human-human interactions, and a key reason is a lack of informative sense of touch in robotic systems. Interpreting human touch gestures is a nuanced, challenging task with extreme gaps between human and robot capability. Among prior works that demonstrate human touch recognition capability, differences in sensors, gesture classes, feature sets, and classification algorithms yield a conglomerate of non-transferable results and a glaring lack of a standard. To address this gap, this work presents 1) four proposed touch gesture classes that cover an important subset of the gesture characteristics identified in the literature, 2) the collection of an extensive force dataset on a common pHRI robotic arm with only its internal wrist force-torque sensor, and 3) an exhaustive performance comparison of combinations of feature sets and classification algorithms on this dataset. We demonstrate high classification accuracies among our proposed gesture definitions on a test set, emphasizing that neural network classifiers on the raw data outperform other combinations of feature sets and algorithms. Accompanying video is here.1

This paper presents a learning-based Model Predictive Control (MPC) methodology incorporating nonlinear predictions with robotics applications in mind. In particular, MPC is combined with feedback linearization for computational efficiency and Gaussian Process Regression (GPR) is used to model unknown system dynamics and nonlinearities. In this method, MPC predicts future states by leveraging a GPR model and optimizes a sequence of inputs over feedback linearized states. The controller was tested in simulation by using a two-link planar robot in the presence of model uncertainty. With respect to trajectory-tracking error, the proposed controller outperformed a conventional Proportional-Derivative Inverse Dynamics controller and a GPR-augmented version. Although a fully nonlinear MPC formulation achieved slightly better performance, the proposed controller had an average control calculation time that was 82× faster.

Generating robot motion that fulfills multiple tasks simultaneously is challenging due to the geometric constraints imposed on the robot. In this paper, we propose to solve multi-task problems through learning structured policies from human demonstrations. Our structured policy is inspired by RMPflow, a framework for combining subtask policies on different spaces. The policy structure provides the user an interface to 1) specifying the spaces that are directly relevant to the completion of the tasks, and 2) designing policies for certain tasks that do not need to be learned. We derive an end-to-end learning objective that is suitable for the multi-task problem, emphasizing the distance between generated motions and demonstrations measured on task spaces. Furthermore, the motion generated from the learned policy class is guaranteed to be stable. We validate the effectiveness of our proposed learning framework through qualitative and quantitative evaluations on three robotic tasks on a 7-DOF Rethink Sawyer robot.

Existing autonomous parking solutions usually require special signs, pre-built maps or accurate ranging sensors to achieve reliable perception of the parking environment, but these methods are difficult to popularize because they either require preconditions or are expensive for production cars. In this paper, we propose a vision-only autonomous parking solution based on only six cameras. Through the appropriate depth estimation algorithms, our method obtains the pixel level depth of the image, and constructs a dense point cloud, so as to realize the fine perception of the parking environment. An improved Radon transform based parking space detection method are applied for better parking space detection method. Our proposed method achieves processing speed of above 5 Hz on a intermediate level computing platform. Furthermore, we demonstrate the practicability of the proposed system in real-world parking lots.

Constrained battery life on current Unmanned Aerial Vehicles (drones) limits the time they can operate and distance they can travel. We address this challenge by harvesting solar power to enable duty-cycled operation on a palm-sized drone. We present a scaling analysis that suggests that more solar power can be collected per unit mass of the drone as scale reduces, favoring small drones. By charging from the sun, the drone can operate for more than a single charging cycle, enabling extended mission time, and long-distance travel. To realize this, we design a high efficiency charging circuit and introduce two innovations. The first is a photovoltaic array that passively folds down while in flight to reduce air drag and automatically opens during landing due to the ground effect. The second is a sensor system and controller that autonomously finds suitable charging sites that are flat and well-lit. The drone can be fully charged in 3 hrs using the solar array and charging circuit with an average efficiency of 90.84%. Each charge enables a 4.7 min flight, allowing the drone to travel up to 1.2 km in a day. We also discuss how this platform could be used to take periodic measurements for smart agriculture or wildlife tracking, rapidly deploy wireless networks, or deploy microrobots in the future.