IROS 2021 论文列表

Cooperative transportation by multi-aerial robots has the potential to support various payloads and improve fail- safe against dropping. Furthermore, changing the attachment positions of robots according payload characteristics increases the stability of transportation. However, there are almost no transportation systems capable of scaling to the payload weight and size and changing the optimal attachment positions. To address this issue, we propose a cooperative transportation sys- tem comprising autonomously executable software and suitable hardware, covering the entire process, from pre-takeoff setting to controlled flight. The proposed system decides the formation of the attachment positions by prioritizing controllability based on the center of gravity obtained from Bayesian estimations with robot pairs. We investigated the cooperative transportation of an unknown payload larger than that of whole carrier robots through numerical simulations. Furthermore, we performed cooperative transportation of an unknown payload (with a weight of about 3.2 kg and maximum length of 1.76 m) using eight robots. The proposed system was found to be versatile with regard to handling unknown payloads with different shapes and center-of-gravity positions.

We present a novel application of Learning from Demonstration to realize a fully autonomous bi-manual surgical suturing task, including needle pick up, insertion, re-grasping, extraction and hand-over. Surgical action primitives are learned from a single human demonstration and encoded into an action library from which they are pulled to compose more elaborate tasks at planning/execution time. The method is demonstrated in a non-clinical setting, using unmodified surgical instruments with a custom surgical robot system. We use stereo vision to automatically detect the suture needle and entry points to close the control loop and generalize tasks to different task conditions. The suturing task is shown to generalize well to differing initial conditions with a success rate of 17 % for the full task, a mean subtask success rate of 75 % and mean needle insertion error of 3.3 mm over the course of 46 trial task executions at human speed. Failures could all be attributed to erroneous vision-based detection, pose estimation and robot calibration.

In this paper, we aim to recognize function points of category-agnostic objects and perform object manipulation. To recognize function points of various shapes, it is necessary to train with a large amount of training data. Also, it is necessary to take into account not only visual information but also physics and interaction between objects. To solve these problems, we are working on the automatic generation of training data by detecting function points from a physical simulation. In the proposed system, we add simulation with target task operation and goal state, which allows a robot to acquire the target function point recognizer. We also use GAN to generate various random shapes and render them with random domains, and train Deep Neural Networks on these data. These enable the robot to recognize function points of unseen objects in the real world and realize manipulation.

High definition (HD) maps have demonstrated their essential roles in enabling full autonomy, especially in complex urban scenarios. As a crucial layer of the HD map, lane-level maps are particularly useful: they contain geometrical and topological information for both lanes and intersections. However, large scale construction of HD maps is limited by tedious human labeling and high maintenance costs, especially for urban scenarios with complicated road structures and irregular markings. This paper proposes an approach based on semantic-particle filter to tackle the automatic lane-level mapping problem in urban scenes. The map skeleton is firstly structured as a directed cyclic graph from online mapping database OpenStreetMap. Our proposed method then performs semantic segmentation on 2D front-view images from ego vehicles and explores the lane semantics on a birds-eye-view domain with true topographical projection. Exploiting OpenStreetMap, we further infer lane topology and reference trajectory at intersections with the aforementioned lane semantics. The proposed algorithm has been tested in densely urbanized areas, and the results demonstrate accurate and robust reconstruction of the lane-level HD map.

Many essential services for autonomous vehicles, e.g., navigation on high-quality maps, are designed based on the understanding of traffic conditions, e.g., travel time/speed on road segments, traffic flow, etc. However, most existing traffic condition models lack the consideration of the differentiation for vehicles with different types (e.g., personal vehicles or trucks) and thus they cannot satisfy some type-specific services, e.g., traffic-condition-based routing for autonomous vehicles with different types. To address this challenge, we design a novel vehicular mobility based sensing model called mDrive to predict the travel speed on the road segments, which is targeted for different types of vehicles by utilizing the camera data obtained from the traffic cameras equipped in the road intersections only, without any in-vehicle GPS devices. mDrive addresses the type-aware traffic speed prediction problem with sparse sensors based on three correlations: (1) the spatial correlation of travel speed on the connected road segments; (2) the temporal correlation of travel speed on the consecutive time slots; (3) the type correlation of different vehicular types’ speed on the same road segment. We implement mDrive on traffic camera data from the Chinese city Suzhou and evaluate it by using the detailed GPS data from personal vehicles, taxis, and trucks, with road contextual data as ground truth. The experiment show mDrive outperforms state-of-the-art methods by reducing 6.2% mean relative error on average for all types of vehicles.

Automated planning enables robots to find plans to achieve complex, long-horizon tasks, given a planning domain. This planning domain consists of a list of actions, with their associated preconditions and effects, and is usually manually defined by a human expert, which is very time-consuming or even infeasible. In this paper, we introduce a novel method for generating this domain automatically from human demonstrations. First, we automatically segment and recognize the different observed actions from human demonstrations. From these demonstrations, the relevant preconditions and effects are obtained, and the associated planning operators are generated. Finally, a sequence of actions that satisfies a user-defined goal can be planned using a symbolic planner. The generated plan is executed in a simulated environment by the TIAGo robot. We tested our method on a dataset of 12 demonstrations collected from three different participants. The results show that our method is able to generate executable plans from using one single demonstration with a 92% success rate, and 100% when the information from all demonstrations are included, even for previously unseen stacking goals.

In this paper, we introduce an automata-based framework for planning with relaxed specifications. User relaxation preferences are represented as weighted finite state edit systems that capture permissible operations on the specification, substitution and deletion of tasks, with complex constraints on ordering and grouping. We propose a three-way product automaton construction method that allows us to compute minimal relaxation policies for the robots using shortest path algorithms. The three-way product automaton captures the robot’s motion, specification satisfaction, and available relaxations at the same time. Additionally, we consider a bi-objective problem that balances temporal relaxation of deadlines within specifications with changing and deleting tasks. Finally, we present the runtime performance and a case study that highlights different modalities of our framework.

The process of capturing a well-composed photo is difficult and it takes years of experience to master. We propose a novel pipeline for an autonomous agent to automatically capture an aesthetic photograph by navigating within a local region in a scene. Instead of classical optimization over heuristics such as the rule-of-thirds, we adopt a data-driven aesthetics estimator to assess photo quality. A reinforcement learning framework is used to optimize the model with respect to the learned aesthetics metric. We train our model in simulation with indoor scenes, and we demonstrate that our system can capture aesthetic photos in both simulation and real world environments on a ground robot. To our knowledge, this is the first system that can automatically explore an environment to capture an aesthetic photo with respect to a learned aesthetic estimator. Source code is at https://github.com/HadiZayer/AutoPhoto

Perceiving obstacles and avoiding collisions is fundamental to the safe operation of a robot system, particularly when the robot must operate in highly dynamic human environments. Proximity detection using on-robot sensors can be used to avoid or mitigate impending collisions. However, existing proximity sensing methods are orientation and placement dependent, resulting in blind spots even with large numbers of sensors. In this paper, we introduce the phenomenon of the Leaky Surface Wave (LSW), a novel sensing modality, and present AuraSense, a proximity detection system using the LSW. AuraSense is the first system to realize no-dead-spot proximity sensing for robot arms. It requires only a single pair of piezoelectric transducers, and can easily be applied to off-the-shelf robots with minimal modifications. We further introduce a set of signal processing techniques and a lightweight neural network to address the unique challenges in using the LSW for proximity sensing. Finally, we demonstrate a prototype system consisting of a single piezoelectric element pair on a robot manipulator, which validates our design. We conducted several micro benchmark experiments and performed more than 2000 on-robot proximity detection trials with various potential robot arm materials, colliding objects, approach patterns, and robot movement patterns. AuraSense achieves 100% and 95.3% true positive proximity detection rates when the arm approaches static and mobile obstacles respectively, with a true negative rate over 99%, showing the real-world viability of this system.

Safe and proactive planning in robotic systems generally requires accurate predictions of the environment. Prior work on environment prediction applied video frame prediction techniques to bird’s-eye view environment representations, such as occupancy grids. ConvLSTM-based frameworks used previously often result in significant blurring of the predictions, loss of static environment structure, and vanishing of moving objects, thus hindering their applicability for use in safety-critical applications. In this work, we propose two extensions to the ConvLSTM architecture to address these issues. We present the Temporal Attention Augmented ConvLSTM (TAAConvLSTM) and Self-Attention Augmented ConvLSTM (SAAConvLSTM) frameworks for spatiotemporal occupancy grid prediction, and demonstrate improved performance over baseline architectures on the real-world KITTI and Waymo datasets. We provide our implementation at https: //github.com/sisl/AttentionAugmentedConvLSTM.

Understanding a controller’s performance in different scenarios is crucial for robots that are going to be deployed in safety-critical tasks. If we do not have a model of the dynamics of the world, which is often the case in complex domains, we may need to approximate a performance function of the robot based on its interaction with the environment. Such a performance function gives us insights into the behaviour of the robot, allowing us to fine-tune the controller with manual interventions. In high-dimensionality systems, where the action-state space is large, fine-tuning a controller is non-trivial. To overcome this problem, we propose a performance function whose domain is defined by external features and parameters of the controller. Attainment regions are defined over such a domain defined by feature-parameter pairs, and serve the purpose of enabling prediction of successful execution of the task. The use of the feature-parameter space –in contrast to the action-state space– allows us to adapt, explain and fine-tune the controller over a simpler (i.e., lower dimensional) space. When the robot successfully executes the task, we use the attainment regions to gain insights into the limits of the controller, and its robustness. When the robot fails to execute the task, we use the regions to debug the controller and find adaptive and counterfactual changes to the solutions. Another advantage of this approach is that we can generalise through the use of Gaussian processes regression of the performance function in the high-dimensional space. To test our approach, we demonstrate learning an approximation to the performance function in simulation, with a mobile robot traversing different terrain conditions. Then, with a sample-efficient method, we propagate the attainment regions to a physical robot in a similar environment.

We propose a new approach to investigate and quantify dynamic grasp performance. Oftentimes, existing approaches to grasp analysis assess a grasp’s quality in a static situation. We build upon such considerations to also account for the dynamic nature of most grasp operations. In particular, these typically do not, in practice, occur in a static setting. Robotic grasping is indeed commonly involved in, for instance, pick-and-place operations which involve movement and thus a dynamic aspect. We investigate grasp quality over such movements, affording consideration not only to the gripper’s and grasp configuration, but also to their trajectory. More specifically, we explore the relationship from the gripper’s base acceleration to the stability of the grasped object (assessed using the relative acceleration of the object with respect to that of the gripper), using linear approximations of the corresponding dynamics. From such relations, we construct a grasp’s robustness metric, which accounts for the movements involved in the considered scenario. Numerical simulations are used to compare achieved results with those obtained using alternate existing methods. We illustrate merit of the proposed metric by exploring robustness of a given grasp under different trajectories.

Assembly orders of components have direct influence on feasibility and efficiency of assembly process in manufacturing and are usually defined by experienced operators. To automate the assembly sequence generation process, we present a method using the idea of case-based reasoning, which can take advantage of experience of a reference assembly to generate the assembly sequence of a new assembly. First, a novel assembly representation method named assembly graph is present in which nodes indicating components’ 3D shape information and edges indicating the geometry constraints. Second, a similar components retrieve process is conducted based on assembly graph representation. Then, the assembly sequence is generated by applying the assembly order of retrieved components to the new ones. Next, the generated sequence is revised to satisfy the inherent constraints in the new assembly which is formulated as a contact graph. Finally, the revised sequence is stored into a case library with corresponding assembly model. We apply the proposed method to generate assembly sequences for chair assemblies and experimental results show its effectiveness and flexibility.

This paper proposes a robot assembly planning method by automatically reading the graphical instruction manuals designed for humans. Essentially, the method generates an Assembly Task Sequence Graph (ATSG) by recognizing a graphical instruction manual. An ATSG is a graph describing the assembly task procedure by detecting types of parts included in the instruction images, completing the missing information automatically, and correcting the detection errors automatically. To build an ATSG, the proposed method first extracts the information of the parts contained in each image of the graphical instruction manual. Then, by using the extracted part information, it estimates the proper work motions and tools for the assembly task. After that, the method builds an ATSG by considering the relationship between the previous and following images, which makes it possible to estimate the undetected parts caused by occlusion using the information of the entire image series. Finally, by collating the total number of each part with the generated ATSG, the excess or deficiency of parts are investigated, and task procedures are removed or added according to those parts. In the experiment section, we build an ATSG using the proposed method to a graphical instruction manual for a chair and demonstrate the action sequences found in the ATSG can be performed by a dual-arm robot execution. The results show the proposed method is effective and simplifies robot teaching in automatic assembly.

This article presents a novel software architecture enabling the analysis of assembly actions from fine-grained hand motions. Unlike previous works that compel humans to wear ad-hoc devices or visual markers in the human body, our approach enables users to move without additional burdens. Modules developed are able to: (i) reconstruct the 3D motions of body and hands keypoints using multi-camera systems; (ii) recognize objects manipulated by humans, and (iii) analyze the relationship between the human motions and the manipulated objects. We implement different solutions based on OpenPose and Mediapipe for body and hand keypoint detection. Additionally, we discuss the suitability of these solutions for enabling real-time data processing. We also propose a novel method using Long Short-Term Memory (LSTM) deep neural networks to analyze the relationship between the detected human motions and manipulated objects. Experimental validations show the superiority of the proposed approach against previous works based on Hidden Markov Models (HMMs).

Developments in human-robot teaming have given rise to significant interest in training methods that enable collaborative agents to safely and successfully execute tasks alongside human teammates. While effective, many existing methods are brittle to changes in the environment and do not account for the preferences of human collaborators. This ineffectiveness is typically due to the complexity of deployment environments and the unique personal preferences of human teammates. These complications lead to behavior that can cause task failure or user discomfort. In this work, we introduce Plan Augmentation and Repair through SEmantic Constraints (PARSEC): a novel algorithm that utilizes a semantic hierarchy to enable novice users to quickly and effectively select constraints using natural language that correct faulty behavior or adapt skills to their preferences. We show through a case study that our algorithm efficiently finds corrective constraints that match the user’s intent, providing a path for novice users to exploit the advantages of constrained motion planning combined with human-in-the-loop skill training.

Recently, mobile robots have become important tools in various industries, especially in logistics. Deep reinforcement learning emerged as an alternative planning method to replace overly conservative approaches and promises more efficient and flexible navigation. However, deep reinforcement learning approaches are not suitable for long-range navigation due to their proneness to local minima and lack of long term memory, which hinders its widespread integration into industrial applications of mobile robotics. In this paper, we propose a navigation system incorporating deep-reinforcement-learning- based local planners into conventional navigation stacks for long-range navigation. Therefore, a framework for training and testing the deep reinforcement learning algorithms along with classic approaches is presented. We evaluated our deep-reinforcement-learning-enhanced navigation system against various conventional planners and found that our system outperforms them in terms of safety, efficiency and robustness.

A formulation of the area defense and surveillance problem for one intruder and one defense and surveillance robot and its corresponding solution using control barrier functions is presented. The defense robot must follow the intruder as it moves through a rectangular region in the plane, ensuring that the position of the intruder is also within a rectangular region attached to the surveillance robot. The proposed reactive and closed-form control laws depend on the positions of the robots, their maximum speeds, and the size of the rectangular regions. We show the application and effectiveness of our results in experiments with real robots.

Visual monitoring operations underwater require both observing the objects of interest in close-proximity, and tracking the few feature-rich areas necessary for state estimation. This paper introduces the first navigation framework, called AquaVis, that produces on-line visibility-aware motion plans that enable Autonomous Underwater Vehicles (AUVs) to track multiple visual objectives with an arbitrary camera configuration in real-time. Using the proposed pipeline, AUVs can efficiently move in 3D, reach their goals while avoiding obstacles safely, and maximizing the visibility of multiple objectives along the path within a specified proximity. The method is sufficiently fast to be executed in real-time and is suitable for single or multiple camera configurations. Experimental results show the significant improvement on tracking multiple automatically-extracted points of interest, with low computational overhead and fast re-planning times.Accompanying short video: https://youtu.be/JKO bbrIZyU