IROS 2019 Papers

Polyp dissection requires very accurate detection of the region of interest and high-precision cutting with adequate safety margins. Robot-assisted polyp dissection is a solution to accomplish high-quality intervention. This paper proposes a method to constrain the robot to follow an accurate dissection path based on Virtual Fixtures (VF). The VFs are created via specific control points obtained directly from images of the surgical scene and are updated by the vision algorithm. The VF constraints can autonomously adapt themselves to environment changing during the surgical intervention. The entire pipeline is validated through experiments on the da Vinci Research Kit (dVRK) robot.

Assistive Robotic Manipulators (ARMs) play an important role for people with upper-limb disabilities and the elderly by helping them complete Activities of Daily Living (ADLs). However, as the objects to handle in ADLs differ in size, shape and manipulation constraints, many two or three fingered end-effectors of ARMs have difficulty robustly interacting with these objects. In this paper, we propose vision-based control of a 5-fingered manipulator (Schunk SVH), automatically changing its approach based on object classification using computer vision combined with deep learning. The control method is tested in a simulated environment and achieves a more robust grasp with the properly shaped five-fingered hand than with a comparable three-fingered gripper (Barrett Hand) using the same control sequence. In addition, the final optimal grasp pose (x, y, and θ) is learned through a deep regressor in the penultimate stage of the grasp. This method correctly identifies the optimal grasp pose in 78.35% of cases when considering all three parameters for an object included in the training set, but in a different setting than that of the training set.

In this paper, we focus on the problem of vision-based localization for ground robotic applications. In recent years, camera only or camera-IMU (inertial measurement unit) based localization methods are widely studied, in terms of theoretical properties, algorithm design, and real-world applications. However, we experimentally find that none of existing methods is able to perform high-precision and robust localization for ground robots in large-scale complicated 3D environments. To this end, in this paper, we propose a novel vision-based localization algorithm dedicatedly designed for ground robots, by fusing measurements from a camera, an IMU, and the wheel odometer. The first contribution of this paper is that we propose a novel algorithm for approximating the motion manifold for ground robots by parametric representation and performing pose integrating via IMU and wheel odometer measurements. Secondly, we propose a complete localization algorithm, by using a sliding-window based estimator. The estimator is designed based on iterative optimization to fuse measurements from multiple sensors on the proposed manifold representation. We show that, based on a variety of real-world experiments, the proposed algorithm outperforms a number of the state-of-the-art vision based localization algorithms by a significant margin, especially when deployed in large-scale complicated environments.

It is still an open problem to develop a reduced order model of bipedal walking that closely represents the complex dynamics of humanoid robots. In this paper, we propose control methodologies, removing both the constant CoM height constraint and the constant centroidal angular momentum constraint. We define a capturability criterion. and propose an enhanced intrinsically stable model predict control to fulfill this new capturability criterion. Then the angular momentum can be controlled. The results of simulations using humanoid robot HRP-4 show the proposed methods can improve the stability of bipedal locomotion on uneven terrains.

This paper introduces an arbitrary-region based shape-control methodology for a swarm-robotics framework to conquer the traditional obstacle-avoidance problem. In this control approach, during the movement of a team of robots through an unknown occluded environment, a virtual boundary for the robotic crowd that could pass through even a very narrow passage, as a swarm, has been created based on the agents' /robots' sensing information. The primary challenge is to force each and every agent to fit into the arbitrary boundary, thus created, so that they can form a swarm and eventually progress towards the goal while avoiding obstacles. Consequently, the process maximizes the inter-agent cohesion. Furthermore, the virtual region is subdivided into so-called cells designated by a non-overlapping circular geometry. The utmost endeavor is to make an agent place itself at the center of a circular cell and a mechanism has been devised, where each agent is duly attracted and eventually is resident at the center of each cell completely blanketing the entire virtual region. To place each agent inside the stipulated contour, a novel circle-packing algorithm has been proposed such that the contour is fitted with a number of identical circles. Finally, the simulation results along with hardware implementation demonstrate the effectiveness of the proposed control technique.

We consider the problem of autonomous mobile robot exploration in an unknown environment taking into account the robot's mapping rate, map uncertainty, and state estimation uncertainty. This paper presents an exploration framework built upon segment-aided pose SLAM adapted for better active localization. We build on our previous work on expectation maximization (EM) exploration, which explicitly models unknown landmarks as latent variables and predicts their expected uncertainty, to resolve the lack of landmark state in denser instances of SLAM. The proposed system comprises path generation, place recognition forecasting, belief propagation and utility evaluation using a virtual map. We analyze the performance in simulated experiments, showing that our algorithm maintains higher coverage speed in exploration as well as lower mapping and localization error. The real-time applicability is demonstrated on an unmanned ground vehicle.

Existing autonomous vehicle (AV) navigation algorithms treat lane recognition, obstacle avoidance, local path planning, and lane following as separate functional modules which result in driving behavior that is incompatible with human drivers. It is imperative to design human-compatible navigation algorithms to ensure transportation safety. We develop a new tightly-coupled perception-planning framework that combines all these functionalities to ensure human-compatibility. Using GPS-camera-lidar sensor fusion, we detect actual lane boundaries (ALBs) and propose availability-reasonability-feasibility (ARF) threefold tests to determine if we should generate virtual lane boundaries (VLBs) or follow ALBs. If needed, VLBs are generated using a dynamically adjustable multi-objective optimization framework that considers obstacle avoidance, trajectory smoothness (to satisfy vehicle kinodynamic constraints), trajectory continuity (to avoid sudden movements), GPS following quality (to execute global plan), and lane following or partial direction following (to meeting human expectation). Consequently, vehicle motion is more human compatible than existing approaches. We have implemented our algorithm and tested under open source data with satisfying results.

The time complexity of making observations and loop closures in a graph-based visual SLAM system is a function of the number of views stored [1], [2]. Clever algorithms, such as approximate nearest neighbor search, can make this function sub-linear. Despite this, over time the number of views can still grow to a point at which the speed and/or accuracy of the system becomes unacceptable, especially in computation- and memory-constrained SLAM systems. However, not all views are created equal. Some views are rarely observed, because they have been created in an unusual lighting condition, or from low quality images, or in a location whose appearance has changed. These views can be removed to improve the overall performance of a SLAM system. In this paper, we propose a method for pruning views in a visual SLAM system to maintain its speed and accuracy for long term use.

In this paper, we present a multi-camera visual odometry (VO) system for an autonomous vehicle. Our system mainly consists of a virtual LiDAR and a pose tracker. We use a perspective transformation method to synthesize a surroundview image from undistorted fisheye camera images. With a semantic segmentation model, the free space can be extracted. The scans of the virtual LiDAR are generated by discretizing the contours of the free space. As for the pose tracker, we propose a visual odometry system fusing both the feature matching and the virtual LiDAR scan matching results. Only those feature points located in the free space area are utilized to ensure the 2D-2D matching for pose estimation. Furthermore, bundle adjustment (BA) is performed to minimize the feature points reprojection error and scan matching error. We apply our system to an autonomous vehicle equipped with four fisheye cameras. The testing scenarios include an outdoor parking lot as well as an indoor garage. Experimental results demonstrate that our system achieves a more robust and accurate performance comparing with a fisheye camera based monocular visual odometry system.

In this paper, we present a multi-camera sensor system along with its control algorithm for automated visual inspection from a moving vehicle. To accomplish this task, we propose a unique hardware configuration consisting of a frontal stereo vision system, six lateral cameras motorized to tilt, and a GPS/IMU sensor mounted on the roof of a car. From the frontal stereo system, we detect electric poles and estimate their corresponding 3D positions. Based on this 3D estimation, the tilt angles of the motorized lateral cameras are controlled in real-time to capture high resolution images of the equipment - typically installed a few meters above the road surface. In addition, inertial odometry information from the GPS/IMU module is utilized for pose estimation, object localization, and re-identification among cameras. Experimental results demonstrate the efficiency and robustness of our system for automated electric equipment maintenance, which can reduce human effort significantly.

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for variable impedance control in end-effector space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.

Hybrid legged-wheeled robots are able to adapt their leg configuration and height to vary their footprint polygons and go over obstacles or traverse narrow spaces. In this paper, we present a variable configuration wheeled motion planner based on the A* algorithm. It takes advantage of the agility of hybrid wheeled-legged robots and plans paths over low-lying obstacles and in narrow spaces. By imposing a symmetry on the robot polygon, the computed plans lie in a low-dimensional search space that provides the robot with configurations to safely negotiate obstacles by expanding or shrinking its footprint polygon. The introduced autonomous planner is demonstrated using simulations and real-world experiments with the CENTAURO robot.

A simulation algorithm to predict expected upper limb movements of prosthesis users performing activities of daily living (ADL) was developed. It is quite challenging to determine the right type and fit of a prosthesis and provide appropriate training to properly utilize it. The amputee care team typically uses prior experiences to provide prescription and training customized for each individual. It is also very difficult to anticipate expected and undesired compensatory motions due to reduced degrees of freedom of a prosthesis user. We have developed a tool to predict and visualize the expected upper limb movements resulting from using a prescribed prosthesis and its suitability to the needs of the amputee. It is expected to help clinicians make decisions such as the type of the prosthesis, and whether to include a wrist joint, based on the impact it will have on the rest of the joints. The main focus of this work is to use robotics-based methods to simulate human use of prostheses and identify the expected posture of the limited joints on the upper limbs. Unlike other works, this paper does not discuss the control of the prosthesis but the posture of the body. A weighted least-norm inverse kinematics algorithm was used to develop a robotics-based model of the upper limbs and torso. Motion capture data from the subjects were used to determine the weighting matrix the algorithm required. Results show that this approach provides human-like simulation of joint motions and matches the motion capture data. The algorithm uses the individual’s anthropometrics and level of amputation to create a personalized kinematic model of the upper body and the joint motions during ADLs. A graphic user interface (GUI) was created to allow the clinician to input the relevant data resulting in arm movements of the prospective prosthesis user. A custom-made visualization software was developed to display an animation performing the simulated motion. It should be stressed that this work d... Show More

Recognizing abnormal events such as traffic violations and accidents in natural driving scenes is essential for successful autonomous driving and advanced driver assistance systems. However, most work on video anomaly detection suffers from two crucial drawbacks. First, they assume cameras are fixed and videos have static backgrounds, which is reasonable for surveillance applications but not for vehicle-mounted cameras. Second, they pose the problem as one-class classification, relying on arduously hand-labeled training datasets that limit recognition to anomaly categories that have been explicitly trained. This paper proposes an unsupervised approach for traffic accident detection in first-person (dashboard-mounted camera) videos. Our major novelty is to detect anomalies by predicting the future locations of traffic participants and then monitoring the prediction accuracy and consistency metrics with three different strategies. We evaluate our approach using a new dataset of diverse traffic accidents, AnAn Accident Detection (A3D), as well as another publicly-available dataset. Experimental results show that our approach outperforms the state-of-the-art. Code and the dataset developed in this work are available at: https:llgithub.comlMoonBtvdltad-IROS2019.

In surgical workflow analysis and training in robot-assisted surgery, automatic task segmentation could significantly reduce the manual labeling time and enhance robot learning efficiency. This paper presents an unsupervised segmentation approach to automatically segment a given surgical task without manual intervention. A new segmentation method is presented, which relies only on bimanual kinematic trajectories without the need for prior information about the data. Specifically, surgical tasks are segmented by fusing trajectories' spatiotemporal and variance properties. To demonstrate the effectiveness of the proposed method, detailed experiments were first conducted on our dataset. We segmented trajectories of three different surgical stitches and observed an average F1 score of 77.9% against the ground truths. The same trajectories were then added with different levels of noises and the segmentation comparison was made with four other methods. The proposed algorithm had demonstrated its robustness against the noises. Finally, to assess its generalization ability, the method was evaluated on publicly available JIGSAWS dataset and an average F1 score of 75.5% was achieved.

Current robotic terrain mapping techniques require expensive sensor suites to construct an environmental representation. In this work, we present a cube-shaped robot that can roll through unstructured terrain and construct a detailed topographic map of the surface that it traverses in real time with low computational and monetary expense. Our approach devolves many of the complexities of locomotion and mapping to passive mechanical features. Namely, rolling movement is achieved by sequentially inflating latex bladders that are located on four sides of the robot to destabilize and tip it. Sensing is achieved via arrays of fine plastic pins that passively conform to the geometry of underlying terrain, retracting into the cube. We developed a topography by shade algorithm to process images of the displaced pins to reconstruct terrain contours and elevation. We experimentally validated the efficacy of the proposed robot through object mapping and terrain locomotion tasks.

Human-robot shared control (SC) has largely been studied in two complementary forms, namely divisible shared control (DSC) and interactive shared control (ISC). DSC enables clean division of the human and the robot subtasks, thus enabling them to work independently, while ISC allows for flexible intervention to improve the collaborative performance or experience. This paper presents a unified scheme that combines both forms of SCs to attain the benefits of flexibility as well as ease of use when human and robot jointly work on a task together. Based on the idea that flexibility should be embedded in every task constraint that the robot is controlling, we connect ISC into the robot subtask providing a soft boundary between the divided orthogonal subspaces allowing human to access the robot subtask and intervene whenever necessary. We also propose a new simple yet effective Cartesian stiffness adaptation law that enables the robot to modify its endpoint stiffness in the robot's control subspace in the presence of disagreement from the human. Simulations and real robot studies for a teleoperated path-following scenario were performed to demonstrate the flexibility of the unified shared control (USC), which allows the robot to dynamically adapt its task based on the operator's intentions.

In this paper, we propose walking balance control based on Caputurability. The proposed method consists of five strategies: (i) moving Zero Moment Point (ZMP) in the support polygon (ii) landing position modification (iii) landing timing modification (iv) angular momentum control (v) falling detection and fall control. Walking pattern generation calculates the ZMP so that the Capture Point (CP) reaches the position of the supporting foot at the end of the double support phase. Owing to the asymmetry of the reachable landing region, landing timing modification is different in the sagittal and lateral planes, and the step time is extended in the lateral plane depending on the direction of disturbances. The torque around the center of gravity to avoid falling is realized through whole-body inverse kinematics with constraints on the angular momentum. In addition, we propose falling detection considering the reachable landing region. We verified the effectiveness of the proposed method through experiments in which the biped robot was disturbed by pushing during tether-free walking. The robot could prevent breakdown by detecting possible falling and performed knee bending motions to suppress damage.

Legibility can be defined as the ability of a robot to communicate its intent to the user. Legibility is relatively little investigated in multi-robot systems, but, in the literature, studies exist where the trajectory of manipulators is analyzed as a factor to improve the collaboration between robots and users. In this paper, we focus on the legibility of a group of mobile robots. To this end, we consider a set of motion-variables: trajectory, dispersion and stiffness. They are typical parameters that determine the motion of a group of robots. To analyze the effect of the motion-variables over legibility, Fisher's exact test and ANOVA (analysis of variance) were carried out. The data for the statistical analysis cover a full factorial plan and they were collected in a virtual reality set-up, where the users shared the environment with a group of robots. We investigate two aspects of legibility: the correctness and the rapidity of communication, namely if the communication happens correctly and how fast it happens. Trajectory was found to be relevant to correctly communicate the intention of the robots, while stiffness and dispersion were relevant for the rapidity of legibility.