IROS 2022 论文列表

Surgical scene segmentation is fundamentally crucial for prompting cognitive assistance in robotic surgery. However, pixel-wise annotating surgical video in a frame-by-frame manner is expensive and time consuming. To greatly reduce the labeling burden, in this work, we study semi-supervised scene segmentation from robotic surgical video, which is practically essential yet rarely explored before. We consider a clinically suitable annotation situation under the equidistant sampling. We then propose PGV-CL, a novel pseudo-label guided cross-video contrast learning method to boost scene segmentation. It effectively leverages unlabeled data for a trusty and global model regularization that produces more discriminative feature representation. Concretely, for trusty representation learning, we propose to incorporate pseudo labels to instruct the pair selection, obtaining more reliable representation pairs for pixel contrast. Moreover, we expand the representation learning space from previous image-level to cross-video, which can capture the global semantics to benefit the learning process. We extensively evaluate our method on a public robotic surgery dataset EndoVis18 and a public cataract dataset CaDIS. Experimental results demonstrate the effectiveness of our method, consistently outperforming the state-of-the-art semi-supervised methods under different labeling ratios, and even surpassing fully supervised training on EndoVis18 with 10.1% labeling. Our code is available at https://github.com/yangyu-cuhk/PGV-CL.

Visual perception tasks often require vast amounts of labelled data, including 3D poses and image space segmen-tation masks. The process of creating such training data sets can prove difficult or time-intensive to scale up to efficacy for general use. Consider the task of pose estimation for rigid objects. Deep neural network based approaches have shown good performance when trained on large, public datasets. However, adapting these networks for other novel objects, or fine-tuning existing models for different environments, requires significant time investment to generate newly labelled instances. Towards this end, we propose ProgressLabeller as a method for more efficiently generating large amounts of 6D pose training data from color images sequences for custom scenes in a scalable manner. ProgressLabeller is intended to also support transparent or translucent objects, for which the previous methods based on depth dense reconstruction will fail. We demonstrate the effectiveness of ProgressLabeller by rapidly create a dataset of over 1M samples with which we fine-tune a state-of-the-art pose estimation network in order to markedly improve the downstream robotic grasp success rates. Progresslabeller is open-source at https://github.com/huijieZH/ProgressLabeller

Harvesting valuable ocean data, ranging from climate and marine life analysis to industrial equipment monitoring, is an extremely challenging real-world problem. Sparse underwater sensor networks are a promising approach to scale to larger and deeper environments, but these have difficulty offloading their data without external assistance. Traditionally, offloading data has been achieved by costly, fixed communication infrastructure. In this paper, we propose a planning under uncertainty method that enables an autonomous underwater vehicle (AUV) to adaptively collect data from smart sensor networks in underwater environments. Our novel solution exploits the ability of sensor nodes to provide the AUV with time-of-flight acoustic localisation, and is able to prioritise nodes with the most valuable data. In both simulated experiments and a real-world field trial, we demonstrate that our method outperforms the type of hand-designed behaviours that has previously been used in the context of underwater data harvesting.

Robots deployed in settings such as warehouses and parking lots must cope with frequent and substantial changes when localizing in their environments. While many previous localization and mapping algorithms have explored methods of identifying and focusing on long-term features to handle change in such environments, we propose a different approach - can a robot understand the distribution of movable objects and relate it to observations of such objects to reason about global localization? In this paper, we present probabilistic object maps (POMs), which represent the distributions of movable objects using pose-likelihood sample pairs derived from prior trajectories through the environment and use a Gaussian process classifier to generate the likelihood of an object at a query pose. We also introduce POM-Localization, which uses an observation model based on POMs to perform inference on a factor graph for globally consistent long-term localization. We present empirical results showing that POM-Localization is indeed effective at producing globally consistent localization estimates in challenging real-world environments and that POM-Localization improves trajectory estimates even when the POM is formed from partially incorrect data.

With advances in image processing and machine learning, it is now feasible to incorporate semantic information into the problem of simultaneous localisation and mapping (SLAM). Previously, SLAM was carried out using lower level geometric features (points, lines, and planes) which are often view-point dependent and error prone in visually repetitive environments. Semantic information can improve the ability to recognise previously visited locations, as well as maintain sparser maps for long term SLAM applications. However, SLAM in repetitive environments has the critical problem of assigning measurements to the landmarks which generated them. In this paper, we use k-best assignment enumeration to compute marginal assignment probabilities for each measurement landmark pair, in real time. We present numerical studies on the KITTI dataset to demonstrate the effectiveness and speed of the proposed framework.

Information about environmental stiffness is useful for robotic tasks involving interactions with unstructured and unknown environments. However, online estimation remains a challenge. Owing to the nature of its calculation algorithm, a large amount of noise may be generated, depending on the response value of the force and position. In this study, we propose a variable gain filter that predicts the degree of such noise using a probabilistic approach and reflects only reliable data in the estimation. We show experimentally that the proposed method improves the accuracy of the stiffness estimation without degrading the estimation time constant.

Advanced social cognitive skills enhance the effectiveness of human-robot interactions. Research shows that an important precursor to the development of these abilities in humans is Theory of Mind (ToM) - the ability to attribute mental states to oneself and to others. In this work, we endow robots with ToM abilities and propose a ToM-based approach to proactive robotic assistance by appealing to epistemic planning techniques. Our evaluation shows that robots implementing our approach and demonstrating ToM are measurably more helpful and perceived by humans as more socially intelligent compared to robots with a deficit in ToM.

For the proper functioning of mobile manipulator-type autonomous robot performing complicated tasks in a human-robot coexistence environment, tasks and motions must be planned simultaneously. In such environments, a human and robot should collaborate with each other. Therefore, the robot must act in accordance with the human and avoid useless actions duplicated with those of humans. However, any action undertaken by a human has uncertainty, and thus, predicting them correctly is challenging. This study proposed probabilistic task and motion planning considering both deterministic and probabilistic environment changes caused by robot and human actions temporarily and spatially, respectively. First, the environmental changes were modeled, where the robot is capable of recognizing the possibility of environmental changes. Second, in task planning, the probabilities of each environmental change owing to human actions was minimized. Finally, in motion planning, a movement path connecting each task in a planned order was planned, thereby enabling the robot to perform actions not duplicated with those by a human. Furthermore, the plans generated were compared without considering possibility of human actions and the effectiveness of the proposed method was verified. Consequently, the proposed method was confirmed to reduce the time required for finishing the tasks.

Soft robots are promising for manipulation tasks thanks to their compliance, safety, and high degree of freedom. However, the commonly used bidirectional continuum segment design means soft robotic manipulators only function in a limited hemispherical workspace. This work increases a soft robotic arm's workspace by designing, fabricating, and controlling an additional soft prismatic actuator at the base of the soft arm. This actuator consists of pneumatic artificial muscles and a piston, making the actuator back-driveable. We increase the task space volume by 116%, and we are now able to perform manipulation tasks that were previously impossible for soft robots, such as picking and placing objects at different positions on a surface and grabbing an object out of a container. By combining a soft robotic arm with a prismatic joint, we greatly increase the usability of soft robots for object manipulation. This work promotes the use of integrated and modular soft robotic systems for practical manipulation applications in human-centered environments.

In recent years, unmanned aerial vehicles (UAVs) are used for numerous inspection and video capture tasks. Manually controlling UAVs in the vicinity of obstacles is challenging, however, and poses a high risk of collisions. Even for autonomous flight, global navigation planning might be too slow to react to newly perceived obstacles. Disturbances such as wind might lead to deviations from the planned trajectories. In this work, we present a fast predictive obstacle avoidance method that does not depend on higher-level localization or mapping and maintains the dynamic flight capabilities of UAVs. It directly operates on LiDAR range images in real time and adjusts the current flight direction by computing angular potential fields within the range image. The velocity magnitude is subsequently determined based on a trajectory prediction and time-to-contact estimation. Our method is evaluated using Hardware-in-the-Loop simulations. It keeps the UAV at a safe distance to obstacles, while allowing higher flight velocities than previous reactive obstacle avoidance methods that directly operate on sensor data.

Apple picking is a challenging manipulation task, but it is difficult to test solutions due to the limited window of time that apples are in season. Previous methods have built simulations of apple trees, but simulations rarely capture soft contact and deformation well, both of which are common in fruit picking. In this paper we present and validate a physical proxy that replicates the mechanics of a real world apple pick. This proxy, in conjunction with a novel hand with multiple sensors, enables large-scale capture of sensor data for data collection and testing. To validate our approach, we train a Long Short-Term Memory network to classify a pick as successful or failed based on sensor feedback from the robot hand. We show that a network trained on the proxy performs as well, or even better, than a network trained solely on real apple trees, with accuracies up to 90%. We determine which sensors are most important for pick classification and also demonstrate that our proxy preserves the most important sensor feature data for pick classification. For our hand, the most informative sensor was the finger's servomotor effort.

This paper addresses the problem of water sampling by using a multirotor UAV with a cable-suspended mechanism. In order to ensure the safe execution of the sampling procedure and the stabilization of the vehicle, the disturbances, induced by the water flow and transferred through the cable, have to be identified. Specifically, an estimate of the disturbances is extracted by integrating a depth sensor, a load cell, an ultrasonic sensor and a downward-looking camera into the UAV's sensor suite and fusing the respective measurements. Gaussian Processes are afterwards employed so as to learn the uncertain disturbances in real time and in a non-parametric manner. The predicted disturbances are incorporated into a geometric control scheme which is capable of stabilizing the UAV above the desired sampling position while compensating for the aforementioned disturbances. The performance of the proposed control strategy is demonstrated through both simulation and experimental results.

Legged robots can traverse a wide variety of terrains, some of which may be challenging for wheeled robots, such as stairs or highly uneven surfaces. However, quadruped robots face stability challenges on slippery surfaces. This can be resolved by adjusting the robot's locomotion by switching to more conservative and stable locomotion modes, such as crawl mode (where three feet are in contact with the ground always) or amble mode (where one foot touches down at a time) to prevent potential falls. To tackle these challenges, we propose an approach to learn a model from past robot experience for predictive detection of potential failures. Accordingly, we trigger gait switching merely based on proprioceptive sensory information. To learn this predictive model, we propose a semi-supervised process for detecting and annotating ground truth slip events in two stages: We first detect abnormal occurrences in the time series sequences of the gait data using an unsupervised anomaly detector, and then, the anomalies are verified with expert human knowledge in a replay simulation to assert the event of a slip. These annotated slip events are then used as ground truth examples to train an ensemble decision learner for predicting slip probabilities across terrains for traversability. We analyze our model on data recorded by a legged robot on multiple sites with slippery terrain. We demonstrate that a potential slip event can be predicted up to 720 ms ahead of a potential fall with an average precision greater than 0.95 and an average F-score of 0.82. Finally, we validate our approach in real-time by deploying it on a legged robot and switching its gait mode based on slip event detection.

Pouring liquids accurately into containers is one of the most challenging tasks for robots as they are unaware of the complex fluid dynamics and the behavior of liquids when pouring. Therefore, it is not possible to formulate a generic pouring policy for real-time applications. In this paper, we propose PourNet, as a generalized solution to pouring different liquids into containers. PourNet is a hybrid planner that uses deep reinforcement learning, for end-effector planning, and Nonlinear Model Predictive Control, for joint planning. In this work, we introduce a novel simulation environment using Unity3D and NVIDIA-Flex to train our agents. By effective choice of the state space, action space and the reward functions, we allow for a direct sim-to-real transfer of the learned skills without additional training. In the simulation, PourNet outperforms state-of-the-art by an average of 4.9g deviation for water-like, and 9.2g deviation for honey-like liquids. In the real-world scenario using Kinova Movo Platform, PourNet achieves an average pouring deviation of 2.3g for dish soap when using a novel pouring container. The average pouring deviation measured for water was 5.5g. All comprehensive experiments and the simulation environment is available at: http://cxdcxd.github.io/RRS/.

A treadmill system has a large potential to provide humans with an augmented walking experience in real-life without a spatial limitation. However, a treadmill gait is different from walking on level ground. In previous studies, the adaptive belt speed control of a treadmill was developed to achieve a self-paced walking for making the users' treadmill gait similar to their level ground gait. Such studies have focused on steady-state walking and regulating the user's position on the treadmill. A normal gait can be divided into an acceleration state after gait initiation, a steady state, and a deceleration state for stopping. The objective of this study is to develop a treadmill system with a wire tension application enabling a human to experience a similar gait to a level ground gait during the transition phase from an acceleration state to a steady state. We developed a treadmill 4 m long × 1 m wide. To allow a user to move on the treadmill during the gait acceleration phase, an insensitive zone where a user can move without the treadmill belt drive was set. In addition, the treadmill was equipped with a wire traction system to apply a traction force canceling the effect of the belt floor acceleration of the treadmill when the belt speed of the treadmill changes. Through an experiment with six participants, the proposed treadmill system allowed the users to move in an acceleration state with the same head acceleration pattern as with level ground walking and cancel the inertial effect with the wire traction, which enabled the users to transition to a steady state from an acceleration state.

When humans grasp objects in the real world, we often move our arms to hold the object in a different pose where we can use it. In contrast, typical lab settings only study the stability of the grasp immediately after lifting, without any subsequent re-positioning of the arm. However, the grasp stability could vary widely based on the object's holding pose, as the gravitational torque and gripper contact forces could change completely. To facilitate the study of how holding poses affect grasp stability, we present PoseIt, a novel multi-modal dataset that contains visual and tactile data collected from a full cycle of grasping an object, re-positioning the arm to one of the sampled poses, and shaking the object. Using data from PoseIt, we can formulate and tackle the task of predicting whether a grasped object is stable in a particular held pose. We train an LSTM classifier that achieves 85% accuracy on the proposed task. Our experimental results show that multi-modal models trained on PoseIt achieve higher accuracy than using solely vision or tactile data and that our classifiers can also generalize to unseen objects and poses. The PoseIt dataset is publicly released here: https://github.com/CMURoboTouch/PoseIt.

High-precision camera re-localization technology in a pre-established 3D environment map is the basis for many tasks, such as Augmented Reality, Robotics and Autonomous Driving. The point-based visual re-localization approaches are well-developed in recent decades, but are insufficient in some feature-less cases. In this paper, we design a complete pipeline for camera pose refinement with points and lines, which contains the innovatively designed line extracting CNN named VLSE, the line matching and the pose optimization approaches. We adopt a novel line representation and customize a hybrid convolution block based on the Stacked Hourglass network [1], to detect accurate and stable line features on images. Then we apply a geometric-based strategy to obtain precise 2D-3D line correspondences using epipolar constraint and reprojection filtering. A following point-line joint cost function is constructed to optimize the camera pose with the initial coarse pose from the pure point-based localization. Sufficient experiments are conducted on open datasets, i.e, line extractor on Wireframe and YorkUrban, localization performance on InLoc ducl and duc2, to confirm the effectiveness of our point-line joint pose optimization method.

This paper studies the problem of constructing polytopic representations for planar regions from depth camera readings. This problem is of great importance for terrain mapping in complicated environment and has great potentials in legged locomotion applications. To address the polytopic planar region characterization problem, we propose a two-stage solution scheme. At the first stage, the planar regions embedded within a sequence of depth images are extracted individually and then merged to establish a terrain map containing only planar regions in a selected frame. To simplify the representations of the planar regions that are applicable to foothold planning for legged robots, we further approximate the extracted planar regions via convex polytopes at the second stage. With the polytopic representation, the proposed approach achieves a great balance between accuracy and simplicity. Experimental validations with RGB-D cameras are conducted to demonstrate the performance of the proposed scheme. The proposed scheme successfully characterizes the planar regions via polytopes with acceptable accuracy. More importantly, the run time of the overall scheme is less than 10ms (i.e., > 100Hz) throughout the tests, which strongly illustrates the advantages of our approach developed in this paper.

For enabling efficient, large-scale coordination of unmanned aerial vehicles (UAV s) under the labeled setting, in this work, we develop the first polynomial time algorithm for the reconfiguration of many moving bodies in three-dimensional spaces, with provable 1. xx asymptotic makespan optimality guarantee under high robot density. More precisely, on an m_{1} \times m_{2} \times m_{3}m_{1} \times m_{2} \times m_{3} grid, m_{1}\geq m_{2}\geq m_{3}m_{1}\geq m_{2}\geq m_{3}, our method computes solutions for routing up to \displaystyle \frac{m_{1}m_{2}m_{3}}{3}\displaystyle \frac{m_{1}m_{2}m_{3}}{3} uniquely labeled robots with uniformly randomly distributed start and goal configurations within a makespan of m_{1}+2m_{2}+2m_{3}+o(m_{1})m_{1}+2m_{2}+2m_{3}+o(m_{1}), with high probability. Because the makespan lower bound for such instances is m_{1}+m_{2}+m_{3}-o(m_{1})m_{1}+m_{2}+m_{3}-o(m_{1}), also with high probability, as m_{1}\displaystyle \rightarrow\infty, \frac{m_{1}+2m_{2}+2m_{3}}{m_{1}+m_{2}+m_{3}}m_{1}\displaystyle \rightarrow\infty, \frac{m_{1}+2m_{2}+2m_{3}}{m_{1}+m_{2}+m_{3}} optimality guarantee is achieved. \displaystyle \frac{m_{1}+2 m_{2}+2m_{3}}{m_{1}+m_{2}+m_{3}}\in\left(1, \displaystyle \frac{5}{3}\right]\displaystyle \frac{m_{1}+2 m_{2}+2m_{3}}{m_{1}+m_{2}+m_{3}}\in\left(1, \displaystyle \frac{5}{3}\right], yielding 1. xx optimality. In contrast, it is well-known that multi-robot path planning is NP-hard to optimally solve. In numerical evaluations, our method readily scales to support the motion planning of over 100, 000 robots in 3D while simultaneously achieving 1. xx optimality. We demonstrate the application of our method in coordinating many quadcopters in both simulation and hardware experiments.