IROS 2020 Papers

Automated surgical gestures classification and recognition are important precursors for achieving the goal of objective evaluation of surgical skills. Many works have been done to discover and validate metrics based on the motion of instruments that can be used as features for automatic classification of surgical gestures. In this work, we present a series of angular metrics that can be used together with Cartesian-based metrics to better describe different surgical gestures. These metrics can be calculated both in Cartesian and joint space, and they are used in this work as features for automatic classification of surgical gestures. To evaluate the proposed metrics, we introduce a novel surgical dataset that contains both Cartesian and joint spaces data acquired with da Vinci Research Kit (dVRK) while a single expert operator is performing 40 subsequent suturing exercises. The obtained results confirm that the application of metrics in the joint space improves the accuracy of automatic gesture classification.

Lightweight robots are known to be intrinsically elastic in their joints. The established classical approaches to control such systems are mostly based on motor-side coordinates since the joints are comparatively stiff. However, that inevitably introduces errors in the coordinates that actually matter: the ones on the link side. Here we present a new joint-torque controller that uses feedback of the link-side positions. Passivity during interaction with the environment is formally shown as well as asymptotic stability of the desired equilibrium in the regulation case. The performance of the control approach is experimentally validated on DLR's new generation of lightweight robots, namely the SARA robot, which enables this step from motor-side-based to link-sided-based control due to sensors with higher resolution and improved sampling rate.

We study a problem in vision-aided navigation in which an autonomous agent has to traverse a specified path in minimal time while ensuring extraction of a steady stream of visual percepts with low latency. Vision-aided robots extract motion estimates from the sequence of images of their on-board cameras by registering the change in bearing to landmarks in their environment. The computational burden of the latter procedure grows with the range of apparent motion undertaken by the projections of the landmarks, incurring a lag in pose estimates that should be minimized while navigating at high speeds. This paper addresses the problem of selecting a desired number of landmarks in the environment, together with the time parametrization of the path, to allow the agent execute it in minimal time while both (i) ensuring the computational burden of extracting motion estimates stays below a set threshold and (ii) respecting the actuation constraints of the agent. We provide two efficient approximation algorithms for addressing the aforementioned problem. Also, we show how it can be reduced to a mixed integer linear program for which there exist well-developed optimization packages. Ultimately, we illustrate the performance of our algorithms in experiments using a quadrotor.

Robots navigating autonomously need to perceive and track the motion of objects and other agents in its surroundings. This information enables planning and executing robust and safe trajectories. To facilitate these processes, the motion should be perceived in 3D Cartesian space. However, most recent multi-object tracking (MOT) research has focused on tracking people and moving objects in 2D RGB video sequences. In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance. Our system is built with recent neural networks for re-identification, 2D and 3D detection and track description, combined into a joint probabilistic data-association framework within a multi-modal recursive Kalman architecture. As part of our work, we release the JRDB dataset, a novel large scale 2D+3D dataset and benchmark, annotated with over 2 million boxes and 3500 time consistent 2D+3D trajectories across 54 indoor and outdoor scenes. JRDB contains over 60 minutes of data including 360o cylindrical RGB video and 3D pointclouds in social settings that we use to develop, train and evaluate JRMOT. The presented 3D MOT system demonstrates state-of-the-art performance against competing methods on the popular 2D tracking KITTI benchmark and serves as first 3D tracking solution for our benchmark. Real-robot tests on our social robot JackRabbot indicate that the system is capable of tracking multiple pedestrians fast and reliably. We provide the ROS code of our tracker at https://sites.google.com/view/jrmot.

We propose a method to annotate segmentation masks accurately and automatically using invisible marker for object manipulation. Invisible marker is invisible under visible (regular) light conditions, but becomes visible under invisible light, such as ultraviolet (UV) light. By painting objects with the invisible marker, and by capturing images while alternately switching between regular and UV light at high speed, massive annotated datasets are created quickly and inexpensively. We show a comparison between our proposed method and manual annotations. We demonstrate semantic segmentation for deformable objects including clothes, liquids, and powders under controlled environmental light conditions. In addition, we show demonstrations of liquid pouring tasks under uncontrolled environmental light conditions in complex environments such as inside the office, house, and outdoors. Furthermore, it is possible to capture data while the camera is in motion so it becomes easier to capture large datasets, as shown in our demonstration.

This study proposes a novel 7-DOF robotic manipulator called CoSMo-Arm for high torque density multi-link robotic platform based on a concentrically stacked modular actuator (CoSMoA) and an extended coaxial spherical joint module (E-CoSMo) introduced in previous researches. The CoSMoA is an actuator module designed to improve thermal characteristics by stacking the motor actuator parts to share the heat dissipation device of the adjacent actuator module, thereby theoretically amplifying motor performance by approximately 3.2 times. The E-CoSMo is a parallel joint mechanism connected to the end of the CoSMoA to create point-centered rotational four degrees of freedom. This joint module has a large range of motion in specific rotational directions and maximum output of approximately up to four times to the actuator output in the specific workspace. The CoSMo-Arm is designed to take advantages of these novel concept modules, having a higher payload than its own weight. To verify the benefits of the proposed mechanism, we performed kinematic analysis and dynamics simulations. From experimental verifications of the real prototype, the feasibility and validity are confirmed as a multi-DOF robotic manipulator.

Fine needle insertions into a lung are challenging in terms of the needle deflection due to the breathing motion. Although previous related works neglected the effect for the needle deflection due to the breathing motion by patients stopping the breath during the insertion, they have to suffer from the discomfort. This paper proposes the intermittent insertion control method to decrease needle deflection adapting the lung deformation due to the breathing motion. The novelty of this method is to allow for accurate needle insertion without stopping the breath, which will contribute to decreasing the discomfort and the amount of radiation exposure for patients. The intermittent insertion is to move forward the fine needle during a certain time frame that the needle deflection barely occurs since the lung is not deformed by the diaphragm motion. The feasibility of the proposed method was validated through a polyvinyl chloride (PVC) phantom and ex vivo experiments. The results showed that the deflection can be suppressed up to 1.3 mm and 3.9 mm in the PVC phantom and ex vivo experiments, respectively.

In this paper, we present a novel approach for classification of unseen object instances from interactive tactile feedback. Furthermore, we demonstrate the utility of a low resolution tactile sensor array for tactile perception that can potentially close the gap between vision and physical contact for manipulation. We contrast our sensor to high-resolution camera-based tactile sensors. Our proposed approach interactively learns a one-class classification model using 3D tactile descriptors, and thus demonstrates an advantage over the existing approaches, which require pre-training on objects. We describe how we derive 3D features from the tactile sensor inputs, and exploit them for learning one-class classifiers. In addition, since our proposed method uses unsupervised learning, we do not require ground truth labels. This makes our proposed method flexible and more practical for deployment on robotic systems. We validate our proposed method on a set of household objects and results indicate good classification performance in real-world experiments.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

Robotic technology is increasingly considered the major mean for fruit picking. However, picking fruits in a dense cluster imposes a challenging research question in terms of motion/path planning as conventional planning approaches may not find collision-free movements for the robot to reach-and-pick a ripe fruit within a dense cluster. In such cases, the robot needs to safely push unripe fruits to reach a ripe one. Nonetheless, existing approaches to planning pushing movements in cluttered environments either are computationally expensive or only deal with 2-D cases and are not suitable for fruit picking, where it needs to compute 3-D pushing movements in a short time. In this work, we present a path planning algorithm for pushing occluding fruits to reach-and-pick a ripe one. Our proposed approach, called Interactive Probabilistic Movement Primitives (I-ProMP), is not computationally expensive (its computation time is in the order of 100 milliseconds) and is readily used for 3-D problems. We demonstrate the efficiency of our approach with pushing unripe strawberries in a simulated polytunnel. Our experimental results confirm I-ProMP successfully pushes table top grown strawberries and reaches a ripe one.

This paper presents the design of an Interacting Multiple Model (IMM) filter for improved navigation performance of Micro Aerial Vehicles (MAVs). The paper considers a navigation system that incorporates rotor drag dynamics and proposes a strategy to overcome the sensitivity of the system to external wind disturbances. Two error state Kalman filters are incorporated in an IMM filtering framework. The first filter has a model that uses conventional Inertial Navigation System (INS) mechanization equations, while the second filter considers a dynamic model with rotor drag forces of the MAV. In order to support the two error state Kalman filters, the generic IMM algorithm [1] is modified for error state implementation, handle dissimilar state definitions, and adaptive switching during operation. Numerical simulations and experimental validation using the EuRoC dataset are conducted to evaluate the performance of the proposed IMM filter design for changing flight conditions and external wind disturbance scenarios.

For multiple robots performing exploration in a previously unmapped environment, such as planetary exploration, maintaining accurate localization and building a consistent map are vital. If the robots do not have a map to localize against and do not explore the same area, they may not be able to find visual loop closures to constrain their relative poses, making traditional SLAM impossible. This paper presents a method for using UWB ranging sensors in multi-robot SLAM, which allows the robots to localize and build a map together even without visual loop closures. The ranging measurements are added to the pose graph as edges and used in optimization to estimate the robots' relative poses. This method builds a map using all robots' observations that is consistent and usable. It performs similarly to visual loop closures when they are available, and provides a good map when they are not, which other methods cannot do. The method is demonstrated on PUFFER robots, developed for autonomous planetary exploration, in an unstructured environment.

Realization of industry-scale, goal-driven, autonomous systems with AI planning technology faces several challenges: flexibly specifying planning goal states in varying situations, synthesizing plans in large state spaces, re-planning in dynamic situations, and facilitating humans to supervise, give feedback and intervene. In this paper, we present Intent-driven Strategic Tactical Planning (ISTP) to address these challenges. We demonstrate its efficacy through its application for radio base station inspection across several locations using drones. The inspection task involves capturing images, thermal images or signal measurements - called knowledge-objects - of various components of the base stations for downstream processing. In the ISTP approach, an operator indicates her goals by flying the drone to different components of interest. These goals are generalized to capture the intent of the operator, which are then instantiated in new situations to generate goals dynamically. Towards planning and re-planning in large state spaces to achieve these goals efficiently, we extend the Strategic-Tactical Planning paradigm. All the components of ISTP are integrated in an intuitive UI and demonstrated through a real life use-case built on the UNITY simulator platform.

Autonomous navigation and exploration in confined spaces are currently setting new challenges for robots. The presence of narrow passages, flammable atmosphere, dust, smoke, and other hazards makes the mapping and navigation tasks extremely difficult. To tackle these challenges, robots need to make intelligent decisions, maximising information while maintaining the safety of the system and their surroundings. In this paper, we present a suite of reasoning mechanisms along with a software architecture for exploration tasks that can be used to underpin the behavior of a broad range of robots operating in confined spaces. We present an autonomous navigation module that allows the robot to safely traverse known areas of the environment and extract features of the unknown frontier regions. An exploration component, by reasoning about these frontiers, provides the robot with the ability to venture into new spaces. From low-level sensory input and contextual information, the robot incrementally builds a semantic network that represents known and unknown parts of the environment and then uses a logic-based, high-level reasoner to interrogate such a network and decide the best course of actions. We evaluate our approach against several mine-like challenging scenarios with different characteristics using a small drone. The experimental results indicate that our method allows the robot to make informed decisions on how to best explore the environment while preserving safety.

In this paper we present a novel strategy for motion planning of autonomous robotic arms in Robotic Minimally Invasive Surgery (R-MIS). We consider a scenario where several laparoscopic tools must move and coordinate in a shared environment. The motion planner is based on a Model Predictive Controller (MPC) that predicts the future behavior of the robots and allows to move them avoiding collisions between the tools and satisfying the velocity limitations. In order to avoid the local minima that could affect the MPC, we propose a strategy for driving it through a sequence of waypoints. The proposed control strategy is validated on a realistic surgical scenario.

As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.†

Traditional soft robots require separate sensors and actuators to precisely control their motion. A twisted-and-coiled actuator (TCA) is a new artificial muscle with both actuation and self-sensing capability that can simultaneously serve both as a sensor and an actuator allowing to control the motion of TCAs without external sensors. This paper investigates the integrated sensing and actuation for TCAs, and the self-sensing function is realized by only measuring the TCA's electrical resistance change. The closed-loop control of a single TCA is realized, and an innervated soft finger that can respond to external load without extra sensors is demonstrated. Our results will lay a foundation for integrated sensing and control by directly using the actuator, paving the way for self-contained smart robotic systems (e.g., untethered soft robots).

Inspection for structural properties (surface stiffness and coefficient of restitution) is crucial for understanding and performing aerial manipulations in unknown environments, with little to no prior knowledge on their state. Inspection-on-the-fly is the uncanny ability of humans to infer states during manipulation, reducing the necessity to perform inspection and manipulation separately. This paper presents an infrastructure for inspection-on-the-fly method for aerial manipulators using hybrid physical interaction control. With the proposed method, structural properties (surface stiffness and coefficient of restitution) can be estimated during physical interactions. A three-stage hybrid physical interaction control paradigm is presented to robustly approach, acquire and impart a desired force signature onto a surface. This is achieved by combining a hybrid force/motion controller with a model-based feed-forward impact control as intermediate phase. The proposed controller ensures a steady transition from unconstrained motion control to constrained force control, while reducing the lag associated with the force control phase. And an underlying Operational Space dynamic configuration manager permits complex, redundant vehicle/arm combinations. Experiments were carried out in a mock-up of a Dept. of Energy exhaust shaft, to show the effectiveness of the inspection-on-the-fly method to determine the structural properties of the target surface and the performance of the hybrid physical interaction controller in reducing the lag associated with force control phase.

Given an articulated robot arm, we present a method to identify two regions with non-empty interiors. The first region is a subset of the configuration space where every point in the region is manipulable. The second region is a subset of the workspace where every point in the region is reachable by the end-effector. Our method expresses the kinematic state of the robot arm using the maximal coordinates, so that the kinematic constraints take polynomial forms. We then reformulate the optimization-based inverse kinematics (IK) algorithm as gradient flows. Finally, we use sum-of-squares (SOS) programming to certify the convergence of each gradient flow. Our main result shows that the feasibility of an SOS programming problem is a sufficient condition for the manipulability and reachability of the sublevel sets of polynomial functions. Our method can be used to certify manipulable or reachable regions by solving a set of linear matrix inequalities (LMIs) or to maximize the volume of a region by solving a set of bilinear matrix inequalities (BMIs). These identified regions can then be used in various motion planning problems as hard safety constraints.