ICRA 2021 Papers

The inner/low-level control loop of most industrial robotic manipulators is protected from any modification by a closed control architecture. The only way to specify joint inputs to them is through position or velocity commands. Furthermore, the inner controller is unknown/uncertain, as it is not revealed to the user. This makes it very difficult to determine the configuration of the inner controller, including the structure and values of the control gains. As a result, integrating external sensory feedback systems into a closed architecture system becomes a difficult task because the interaction of the external feedback loop with the inner control loop can affect the stability of the overall system. In this paper, a stable control strategy is proposed to generate the joint velocity commands for industrial robotic manipulators with uncertain closed control architecture. Based on the proposed method, external feedback controllers can be added regardless of the inner control loop configuration. The proposed external (or outer) control loop approach provides a high degree of design flexibility by enabling smooth implementation of various modern control applications on industrial robots. Unlike previous studies, the proposed control method is not limited to a particular configuration of the controller in the inner loop, nor to the structure of its control gains. The proposed controller design is validated on the UR 5e industrial manipulator.

Complex pipeline network should be inspected regularly for safety. In general, these tasks are often completed by large pipe-climbing robots or customized equipment. Most of them are not effective, and cannot work on pipes with uncertain barriers. Moreover, some pipes are mounted in constrained scenarios, so bulky robots are not applicable. This paper presents a tethered soft-rigid pipe-climbing robot to fill the gap. Two indispensable actions, i.e. embracing pipe and moving along it, are realized by a soft component and a 3D printed wheel mechanism. The latter includes two forces: thrust force from the compressed air and tractive force from wheels, to drive the robot comprehensively, and a lightweight body (only 160g) benefits agile motion. In operation, pressure exerted on the soft component enables the robot to embrace pipes of different diameters, with controllable adhesion force, and locomotion force is also regulated. Inspired from vehicle, an elastic damper is attached between the wheel structure and the robot body, which can effectively alleviate vibration when crossing barriers. In addition, theoretical models are constructed to analyse and control thrust force, and the locomotion performance is analysed by dynamics model. Experiments demonstrate that this robot can perform rapid climbing at a speed of 1.09m/s in load-free scenarios, and it can move at a maximum speed of 0.828m/s with 500g load. Reconstruction of a flexible pipe using the robot is also demonstrated.

Soft robotic grippers possess high structural compliance and adaptability for grasping objects with unknown and irregular shapes and sizes. To enable more dexterous manipulation, soft sensors with similar mechanical properties to common elastomer materials are desired to be integrated into soft grippers. In this paper, we develop ionic hydrogel-based strain and tactile sensors and integrate these sensors into a three-finger soft gripper for learning-based object recognition. Such hydrogel-based sensors have excellent conductivity, high stretchability and toughness, good ambient stability, and unique anti-freezing property, and can be readily attached to a soft gripper at desired locations for strain and tactile sensing. Based on a deep-learning model, we demonstrate the capability of the sensory soft gripper for object grasping and recognition at both room and freezing temperatures, and achieve high recognition accuracy close to 100% for 10 typical objects. With these abilities, our gripper can find interesting applications such as sorting food or chemicals in low temperature storage and cold chain transportation, or manipulating equipment in polar area.

Composites are increasingly becoming a material of choice in the aerospace and automotive industries. Currently, many composite parts are produced by manually laying up sheets on complex molds. Composite sheet layup requires executing two main tasks: (1) grasping a sheet and (2) draping it on the mold. Automating the layup process requires automation of these two tasks. This paper is focused on the automation of the grasping task using robots. This requires an automated generation of grasp plans to enable robots to hold the sheet during the draping process. We present a simulation-based approach for determining robot grasp locations on the composite sheets. We also present an intervention controller that uses a real-time sheet tracking system during plan execution and can prevent failures. We demonstrate the performance of the developed system using a large complex part.

Electromyography (EMG) is a wearable, noninvasive, commonly used method for measuring the human muscular activations from the surface of the skin. In this work, we present a pilot study that focuses on the formulation of a shared control framework to facilitate the simplified execution of Electromyography (EMG) based telemanipulation tasks with a robotic platform. The framework combines a Random Forests (RF) regression method with a compliance controller that relies on the force measurements collected with a force-torque sensor. The RF regression efficiently maps the myoelectric activations of the human muscles to corresponding human wrist positions. Then, a teleoperation process is used to control the robot arm end-effector’s position, utilizing the human wrist position estimations. The examined application involves semi-autonomous cleaning of a whiteboard surface with the proposed framework. The compliance controller guarantees that a desired contact force will always be maintained on the whiteboard surface during task execution. This ensures that any EMG based decoding inaccuracies will not drive the robot away from the cleaning plane. Essentially, the system projects the EMG based estimation on the cleaning plane. The shared control framework offers robust performance, with minimal training and calibration required.

This paper presents a new self-supervised system for learning to detect novel and previously unseen categories of objects in images. The proposed system receives as input several unlabeled videos of scenes containing various objects. The frames of the videos are segmented into objects using depth information, and the segments are tracked along each video. The system then constructs a weighted graph that connects sequences based on the similarities between the objects that they contain. The similarity between two sequences of objects is measured by using generic visual features, after automatically re-arranging the frames in the two sequences to align the viewpoints of the objects. The graph is used to sample triplets of similar and dissimilar examples by performing random walks. The triplet examples are finally used to train a siamese neural network that projects the generic visual features into a low-dimensional manifold. Experiments on three public datasets, YCB-Video, CORe50 and RGBD-Object, show that the projected low-dimensional features improve the accuracy of clustering unknown objects into novel categories, and outperform several recent unsupervised clustering techniques.

This paper presents a method for LiDAR sensor-based traversability analysis for autonomous mobile robots in urban environments. Although urban environments are structured environments, a typical terrain comprises hazardous regions for mobile robots. Therefore, a reliable method for detecting traversable regions is required to prevent robots from getting stuck in the middle of the road. Conventional approaches require considerable efforts to obtain a model for traversability analysis for a specific robot or environment. In particular, learning-based methods require explicit training data. This paper introduces a method for traversability mapping based on a self-training algorithm to eliminate the hand labeling process. A neural network was applied to the underlying classifier of the self-training algorithm. With our approach, the model can be learned with even weakly labeled data obtained from robot-specific parameters and the robot’s footprint. In practical experiments, the self-trained model performed better performance than the existing supervised learning method. Moreover, as the fraction of unlabeled data increased, the performance also increased. Therefore, the demonstrations in the urban environments indicate the effectiveness of the proposed method for traversability mapping.

We introduce a self-supervised method for systematically choosing traversable terrain while autonomously navigating a vehicle to a goal position in an unknown off-road environment. Leveraging the color discriminant bias of off-road terrain types, and using images from a vehicle-mounted camera, we employ a viewpoint transformation that maintains the spatial layout of the terrain to cluster terrain types by color and register corresponding traversability features to guide future navigation decisions. As it navigates, our algorithm also generates training images for use in contemporary end-to-end navigation schemes. Our test results demonstrate the advantages of our approach over classical near-to-far approaches in off-road environments with unknown traversability characteristics, and highlight its fit to supervised semantic segmentation schemes that require foreknowledge of traversability characteristics for labeling, which are limited by insufficient data and suffer pixel-level class imbalance. We detail the techniques for clustering, feature registration, path planning and navigation; and demonstrate the method. Finally, we study the effectiveness of non-discretionary self-supervised data labeling.

Creating robots that can perform general-purpose service tasks in a human-populated environment has been a longstanding grand challenge for AI and Robotics research. One particularly valuable skill that is relevant to a wide variety of tasks is the ability to locate and retrieve objects upon request. This paper models this skill as a Scavenger Hunt (SH) game, which we formulate as a variation of the NP-hard stochastic traveling purchaser problem. In this problem, the goal is to find a set of objects as quickly as possible, given probability distributions of where they may be found. We investigate the performance of several solution algorithms for the SH problem, both in simulation and on a real mobile robot. We use Reinforcement Learning (RL) to train an agent to plan a minimal cost path, and show that the RL agent can outperform a range of heuristic algorithms, achieving near optimal performance. In order to stimulate research on this problem, we introduce a publicly available software stack and associated website that enable users to upload scavenger hunts which robots can download, perform, and learn from to continually improve their performance on future hunts.

Autonomous vehicles need to handle various traffic conditions and make safe and efficient decisions and maneuvers. However, on the one hand, a single optimization/sampling-based motion planner cannot efficiently generate safe trajectories in real time, particularly when there are many interactive vehicles near by. On the other hand, end-to-end learning methods cannot assure the safety of the outcomes. To address this challenge, we propose a hierarchical behavior planning framework with a set of low-level safe controllers and a high-level reinforcement learning algorithm (H-CtRL) as a coordinator for the low-level controllers. Safety is guaranteed by the low-level optimization/sampling-based controllers, while the high-level reinforcement learning algorithm makes H-CtRL an adaptive and efficient behavior planner. To train and test our proposed algorithm, we built a simulator that can reproduce traffic scenes using real-world datasets. The proposed HCtRL is proved to be effective in various realistic simulation scenarios, with satisfying performance in terms of both safety and efficiency.

Craniomaxillofacial reconstruction with patientspecific customized craniofacial implants (CCIs) is most commonly performed for large-sized skeletal defects. Because the exact size of skull resection may not be known prior to the surgery, in single-stage cranioplasty, an oversized CCI is prefabricated and resized intraoperatively with a manual-cutting process provided by a surgeon. The manual resizing, however, may be inaccurate and significantly add to the operating time. This paper introduces a fast and non-contact approach for intraoperatively determining the exact contour of the skull resection and automatically resizing the implant to fit the resection area. Our approach includes four steps: First, we acquire a patient’s defect information using a handheld 3D scanner. Second, the scanned defect is aligned to the CCI by registering the scanned defect to the preoperative CT model. Third, a cutting toolpath is generated from the scanned defect model by extracting the resection contour. Lastly, a cutting robot resizes the oversized CCI to fit the resection area. To evaluate the resizing performance of our method, we generated six different resection shapes for the cutting experiments. We compared the performance of our method to the performance of surgeon’s manual resizing and an existing technique that collects the defect contour with an optical tracking system. The results show that our proposed method improves the resizing accuracy by 56% compared to the surgeon’s manual modification and 42% compared to the optical tracking method.

In this paper, we present a robotic system to automatically perform defect inspection tasks over free-form specular surfaces, which the image acquisition sub-system is equipped with a 6-DOF robot manipulator to achieve flexible scanning. Given the mesh model of the workpiece, we implement K-means based region segmentation algorithm on the point cloud after preprocessing. Then, we take the smooth regions as input to plan the scanning path. A projection registration method that robustly localizes the object in the robot’s frame is proposed for real-time workpiece localization. According to the optical features of the high-resolution line scan, we design an image processing pipeline to detect defects from the captured images. We report a detailed experimental study to validate the proposed methodology.

Effective social robots should leverage humor’s unique ability to improve relationship connections and dispel stress, but current robots possess limited (if any) humorous abilities. In this paper, we aim to supplement one aspect of autonomous robots by giving robotic systems the ability to "read the room" to assess how their humorous statements are received by nearby people in real time. Using a dataset of the audio of crowd responses to a robotic comedian over multiple performances (first presented in past work), we establish human-labeled joke success ground truths and compare individual human rater accuracy against the outputs of lightweight Machine Learning (ML) approaches that are easy to deploy in real-time joke assessment. Our results indicate that all three ML approaches (naïve Bayes, support vector machines, and single-hidden-layer feedforward neural networks) performed significantly better than the baseline approach used in our past work. In particular, support vector machines and neural network approaches are comparable to a human rater in the task of assessing if a joke failed or not in certain cases. The products of this work will inform self-assessment techniques for robots and help social robotics researchers test their own assessment methods on realistic data from human crowds.

In this work, a novel Dynamic Movement Primitive (DMP) formulation is proposed which supports reversibility, i.e. backwards reproduction of a learned trajectory. Apart from sharing all favourable properties of the original DMP, decoupling the teaching of position and velocity profiles and bidirectional drivability along the encoded path are also supported. Original DMP have been extensively used for encoding and reproducing a desired motion pattern in several robotic applications. However, they lack reversibility, which is a useful and expedient property that can be leveraged in many scenarios. The proposed formulation is analyzed theoretically and its practical usefulness is showcased in an assembly by insertion experimental scenario.

Real-time guidewire segmentation and endpoint localization play a pivotal role in robot-assisted minimally invasive surgery, which is helpful to reduce radiation dose and procedure time. Nevertheless, the tasks often come with the challenge of limited computational resources. For this purpose, a real-time multi-task framework with two stages is developed. In the first stage, a Fast Attention-fused Network (FAD-Net) is proposed to obtain accurate guidewire segmentation masks. In the second stage, a lightweight localization network and a post-processing algorithm are designed to robustly predict the guidewire endpoint position. Quantitative and qualitative evaluations on intraoperative X-ray sequences from 30 patients demonstrate that the developed framework outperforms the previously-published results for the tasks, achieving state-of-the-art performance. Moreover, the inference rate of the developed framework is approximately 10.6 FPS, which meets the real-time requirement of X-ray fluoroscopy. These results indicate the proposed approach has the potential to be integrated into the robotic navigation framework for endovascular interventions, enabling robotic-assisted minimally invasive surgery.

This paper offers a multi-layer planning approach for autonomous surface vessels (ASVs) that must adhere to good seamanship practices and the International Regulations for Prevention of Collisions at Sea (COLREGS) [1]. The approach combines novel situational awareness logic with motion primitive-based planners in a receding horizon framework. Further, ship domain and ship arena concepts are used to develop risk metrics that capture COLREGS compliance and the notion of good seamanship. By relying on metrics-driven motion planning as opposed to rule-based conditions, the proposed framework scales naturally to non-trivial single-vessel and multi-vessel situations. The planner is evaluated using adaptive, simulation-based testing to statistically compare the performance to other standard methods. Finally, proof-of-concept field experiments are presented on a subscale platform.

A method that enables an industrial robot to accomplish the peg-in-hole task for holes in concrete is proposed. The proposed method involves slightly detaching the peg from the wall, when moving between search positions, to avoid the negative influence of the concrete’s high friction coefficient. It uses a deep neural network (DNN), trained via reinforcement learning, to effectively find holes with variable shape and surface finish (due to the brittle nature of concrete) without analytical modeling or control parameter tuning. The method uses displacement of the peg toward the wall surface, in addition to force and torque, as one of the inputs of the DNN. Since the displacement increases as the peg gets closer to the hole (due to the chamfered shape of holes in concrete), it is a useful parameter for inputting in the DNN. The proposed method was evaluated by training the DNN on a hole 500 times and attempting to find 12 unknown holes. The results of the evaluation show the DNN enabled a robot to find the unknown holes with average success rate of 96.1% and average execution time of 12.5 seconds. Additional evaluations with random initial positions and a different type of peg demonstrate the trained DNN can generalize well to different conditions. Analyses of the influence of the peg displacement input showed the success rate of the DNN is increased by utilizing this parameter. These results validate the proposed method in terms of its effectiveness and applicability to the construction industry.

Path planning and collision avoidance are challenging in complex and highly variable environments due to the limited horizon of events. In literature, there are multiple model- and learning-based approaches that require significant computational resources to be effectively deployed and they may have limited generality. We propose a planning algorithm based on a globally stable passive controller that can plan smooth trajectories using limited computational resources in challenging environmental conditions. The architecture combines the recently proposed fractal impedance controller with elastic bands and regions of finite time invariance. As the method is based on an impedance controller, it can also be used directly as a force/torque controller. We validated our method in simulation to analyse the ability of interactive navigation in challenging concave domains via the issuing of via-points, and its robustness to low bandwidth feedback. A swarm simulation using 11 agents validated the scalability of the proposed method. We have performed hardware experiments on a holonomic wheeled platform validating smoothness and robustness of interaction with dynamic agents (i.e., humans and robots). The computational complexity of the proposed local planner enables deployment with low-power micro-controllers lowering the energy consumption compared to other methods that rely upon numeric optimisation.

Load-carrying capability is an essential criterion in legged robots' practical application. This paper proposes an unpowered hydraulic auxiliary system to improve the legged robot's loading capability and energy efficiency. For humans, it has been widely hypothesized that intra-abdominal pressure can reduce potential injurious compressive force imposed on spinal discs when a person lifts heavy objects. Inspired by this human biomechanical phenomenon, we design a novel loading-carrying strategy using hydraulic cylinders, valves and accumulators. Different from ordinary powered hydraulic systems, this design provides continuous support force to share the load applied on knee joint actuator without consuming extra energy. The bent-leg theoretical model is constructed to validate the design and analysis. A bipedal hydraulic-assisted electric leg prototype (HyELeg) is fabricated and tested for squatting and walking gaits. The results show that with hydraulic assistance, the prototype can save energy by 45.9% for squatting with a load of 125% of its own body weight, and the walking performance is enhanced by 16.9% in energy efficiency with carrying a load of 67.5% of its own body weight.