IROS 2021 Papers

Shape control has become a prominent research field as it enables the automation of tasks in many applications. Overall, deforming an object to a desired target shape by using few grippers is a major challenge. The limited information about the object dynamics, the need to combine small and large deformations in order to achieve certain target shapes and the non-linear nature of most deformable objects are factors that significantly hamper shape control performance. In this paper, we propose a shape control method for multi-robot manipulation of large-strain deformable objects. Our approach is based on multi-scale Laplacian descriptors that feed an FMM (Fast Marching Method) for elastic shape contour matching. The FMM's resulting path and the Laplacian operator are used to define a control strategy for the robot grippers. Simulation experiments carried out with an ARAP (As Rigid As Possible) deformation model provide satisfactory results.

We address the problem of assigning a team of drones to autonomously capture a set desired shots of a dynamic target in the presence of obstacles. We present a two-stage planning pipeline that generates offline an assignment of drone to shots and locally optimizes online the viewpoint. Given desired shot parameters, the high-level planner uses a visibility heuristic to predict good times for capturing each shot and uses an Integer Linear Program to compute drone assignments. An online Model Predictive Control algorithm uses the assignments as reference to capture the shots. The algorithm is validated in hardware with a pair of drones and a remote controlled car.

In this paper, we propose an evolutionary algorithm for solving the multi-robot orienteering problem where a team of cooperative robots aims to maximize the total information collected by visiting a subset of given nodes within a fixed budget on travel costs. Multi-robot orienteering problems are relevant to applications such as logistic delivery services, precision agriculture, and environmental sampling and monitoring. We consider the case where the information gain at each node is related to the service time each robot spends at the node. As such, we address a variant of the Orienteering Problem where the collected rewards are a function of the time a robot spends at a given location. We present a genetic algorithm solver to this cooperative Team Orienteering Problem with service-time dependent rewards. We evaluate the approach over a diverse set of node configurations and for different team sizes. Lastly, we evaluate the effects of team heterogeneity on overall task performance through numerical simulations.

A multi-modal framework to generate user intention distributions when operating a mobile vehicle is proposed in this work. The model learns from past observed trajectories and leverages traversability information derived from the visual surroundings to produce a set of future trajectories, suitable to be directly embedded into a perception-action shared control strategy on a mobile agent, or as a safety layer to supervise the prudent operation of the vehicle. We base our solution on a conditional Generative Adversarial Network with Long-Short Term Memory cells to capture trajectory distributions conditioned on past trajectories, further fused with traversability probabilities derived from visual segmentation with a Convolutional Neural Network. The proposed data-driven framework results in a significant reduction in error of the predicted trajectories (versus the ground truth) from comparable strategies in the literature (e.g. Social-GAN) that fail to account for information other than the agent’s past history. Experiments were conducted on a dataset collected with a custom wheelchair model built onto the open-source urban driving simulator CARLA, proving also that the proposed framework can be used with a small, unannotated dataset.

Motivated by the goal of achieving long-term drift-free camera pose estimation in complex scenarios, we propose a global positioning framework fusing visual, inertial and Global Navigation Satellite System (GNSS) measurements in multiple layers. Different from previous loosely- and tightly-coupled methods, the proposed multi-layer fusion allows us to delicately correct the drift of visual odometry and keep reliable positioning while GNSS degrades. In particular, local motion estimation is conducted in the inner-layer, solving the problem of scale drift and inaccurate bias estimation in visual odometry by fusing the velocity of GNSS, pre-integration of Inertial Measurement Unit (IMU) and camera measurement in a tightly-coupled way. The global localization is achieved in the outer-layer, where the local motion is further fused with GNSS position and course in a long-term period in a loosely-coupled way. Furthermore, a dedicated initialization method is proposed to guarantee fast and accurate estimation for all state variables and parameters. We give exhaustive tests of the proposed framework on indoor and outdoor public datasets. The mean localization error is reduced up to 63%, with a promotion of 69% in initialization accuracy compared with state-of-the-art works. We have applied the algorithm to Augmented Reality (AR) navigation, crowd sourcing high-precision map update and other large-scale applications.

This paper proposes a collaborative policy framework via relational graph reasoning for multi-agent systems to accomplish adversarial tasks. A relational graph reasoning module consisting of an agent graph reasoning module and an opponent graph module, is designed to enable each agent to learn mixture state representation to enhance the effectiveness of the policy. In particular, for each agent, the agent graph reasoning module is designed to infer different underlying influences from different opponents and generate agent-level state representation. The opponent graph reasoning module is creatively designed for the opponents to reason relations from their surrounding objects including the agents and the opponents based on their latent features and then predict the future state of the opponents. It forms an opponent-level state representation. Besides, in order to effectively predict the state of the opponents, an intrinsic reward based on prediction error is designed to motivate the policy learning. Furthermore, interactions among agents are utilized to transmit messages and fuse information to promote the cooperative behaviors among the agents. Finally, various representative simulations on two multi-agent adversarial tasks are conducted to demonstrate the superiority and effectiveness of the proposed framework by comparison with existing methods.

Predicting the motion of observed entities benefits humans almost seamlessly. The same benefits can be proliferated to mobile autonomous systems if we have a reliable, real-time solution to predict the motion of any object of interest, be it the host’s own motion or that of an observed foreign object. In this work, a novel Multi-Variable State Prediction (MVSP) methodology is devised for real-time trajectory prediction. MVSP incorporates cascaded stages of HMM with Viterbi algorithm and probabilistic quantization for accurately predicting the motion characteristics of the moving object. The overall scheme is employed to predict the motion of moving objects in a 3D space. The proposed approach is verified on both synthetically generated data sequences and data-sets captured from real-life experiments. For a practical scenario, the experiments resulted in an RMS error of 0.6m for a predicted distance of ~18m demonstrating the effectiveness and accuracy of the proposed methodology.

We address the problem of motion control for safe physical interaction, and in particular finding new ways for impedance controller parameters’ adaptation to ensure better safety with minimum possible lose in robot performance. We propose an exteroception-based dynamically updated safety metric that takes into account current robot state and inertia as well as external objects’ mass, shape, material properties, velocity, sensor confidence and existing sampling rates. We also present how this metric can be applied to design a finite state machine of the multi-stage controller, which allows us to prioritize either safety of performance by setting different energy and power constraints with smooth transition in between free motion and interaction modes.

The goal of this work is to propose a way of dealing with physical interactions for collaborative robots that will ensure the safety of a human operator and improve the performance of a common task by implementing multiple robot behavior scenarios. In this scope, all collisions of a robotic arm are detected and analyzed to chooses an appropriate reaction strategy. The points of contact on the robot’s surface for each collision are estimated, the external forces are identified and collisions are classified by the set of predefined categories. Based on these categories and the current robot state, the algorithm chose an appropriate behavior scenario.All presented algorithms are based only on proprioceptive sensors information and were tested in a simulated environment and on the real collaborative robots KUKA iiwa and Universal Robots UR10e. The result for contact localization showed 4 cm mean accuracy, the classification algorithm was able to identify collisions with hard and soft objects with 98% accuracy for KUKA iiwa 14.

Modeling induction motor dynamics is a crucial problem in the industry. The previous works mainly model the dynamics based on the physical model assumption and state equation. However, due to the complex internal structure of motors, the traditional methods cannot estimate dynamics precisely. To address this issue, we adopt a deep learning-based approach that takes the time-series motor data measured from the sensor to estimate the dynamics without making any assumptions about the motor’s interior. In this paper, we propose a multi-scale feature aggregation with self-attention network (MASNet) to deal with modeling motor dynamics. First, our model extracts multi-scale features from motor signals by various convolutional kernel sizes. Then, the proposed adaptive feature aggregation module fuses different receptive signals effectively. To further refine these high-level motor features, two-stream components, containing Bidirectional LSTM and self-attention module, are applied to improve motor contextual information. Moreover, we present a novel temporal relative loss to enhance consecutive signal consistency, which improves the performance of modeling dynamics. To deploy the service in the real-world scenario, our network is very lightweight and reduces the number of parameters from 0.62M to 0.09M (around 85% reduction) but outperforms state-of-the-art algorithms by extensive experiments on simulation and real-world motor datasets.

This paper investigates the task coordination of multi-robot where each robot has a private individual temporal logic task specification; and also has to jointly satisfy a globally given collaborative temporal logic task specification. To efficiently generate feasible and optimized task execution plans for the robots, we propose a hierarchical multi-robot temporal task planning framework, in which a central server allocates the collaborative tasks to the robots, and then individual robots can independently synthesize their task execution plans in a decentralized manner. Furthermore, we propose an execution plan adjusting mechanism that allows the robots to iteratively modify their execution plans via privacy-preserved inter-agent communication, to improve the expected actual execution performance by reducing waiting time in collaborations for the robots. The correctness and efficiency of the proposed method are analyzed and also verified by extensive simulation experiments.

The multi-robot coverage problem is an essential building block for systems that perform tasks like inspection, exploration, or search and rescue. We discretize the coverage problem to induce a spatial graph of locations and represent robots as nodes in the graph. Then, we train a Graph Neural Network controller that leverages the spatial equivariance of the task to imitate an expert open-loop routing solution. This approach generalizes well to much larger maps and larger teams that are intractable for the expert. In particular, the model generalizes effectively to a simulation of ten quadrotors and dozens of buildings in an urban setting. We also demonstrate the GNN controller can surpass planning-based approaches in an exploration task.

Robots operating in households must find objects on shelves, under tables, and in cupboards. In such environments, it is crucial to search efficiently at 3D scale while coping with limited field of view and the complexity of searching for multiple objects. Principled approaches to object search frequently use Partially Observable Markov Decision Process (POMDP) as the underlying framework for computing search strategies, but constrain the search space in 2D. In this paper, we present a POMDP formulation for multi-object search in a 3D region with a frustum-shaped field-of-view. To efficiently solve this POMDP, we propose a multi-resolution planning algorithm based on online Monte-Carlo tree search. In this approach, we design a novel octree-based belief representation to capture uncertainty of the target objects at different resolution levels, then derive abstract POMDPs at lower resolutions with dramatically smaller state and observation spaces. Evaluation in a simulated 3D domain shows that our approach finds objects more efficiently and successfully compared to a set of baselines without resolution hierarchy in larger instances under the same computational requirement. We demonstrate our approach on a mobile robot to find objects placed at different heights in two 10m2×2m regions by moving its base and actuating its torso.

In this paper, we propose a resource-efficient approach to provide an autonomous UAV with an on-board perception method to detect safe, hazard-free landing sites during flights over complex 3D terrain. We aggregate 3D measurements acquired from a sequence of monocular images by a Structure-from-Motion approach into a local, robot-centric, multi-resolution elevation map of the overflown terrain, which fuses depth measurements according to their lateral surface resolution (pixel-footprint) in a probabilistic framework based on the concept of dynamic Level of Detail. Map aggregation only requires depth maps and the associated poses, which are obtained from an on-board Visual Odometry algorithm. An efficient landing site detection method then exploits the features of the underlying multi-resolution map to detect safe landing sites based on slope, roughness, and quality of the reconstructed terrain surface. The evaluation of the performance of the mapping and landing site detection modules are analyzed independently and jointly in simulated and real-world experiments in order to establish the efficacy of the proposed approach.

A human hand can grasp a desired number of objects at once from a pile based solely on tactile sensing. To do so, a robot needs to make a grasp in a pile, sense the number of objects in the grasp before lifting, and predict how many will remain in the grasp after lifting. It is a very challenging problem because when making the prediction, the robotic hand is still in the pile and the objects in the grasp are not observable to vision systems. Moreover, some objects in the hand before lifting may fall out the grasp when the lifting starts because they were supported by other objects in the pile instead of the fingers. A robotic hand should sense how many objects are in a grasp using its tactile sensors before lifting. This paper presents novel multi-object grasping analyzing methods to solve this problem. They include a grasp volume calculation, tactile force analysis, and a data-driven deep learning approach. The methods have been implemented on a Barrett hand and then evaluated in simulations and a real setup with a robotic system. The evaluation results conclude that once the Barrett hand grasps multiple objects in the pile, the data-driven models can make a good prediction before lifting on how many objects will remain in the hand after lifting. The root-mean-square errors are 0.74 for balls and 0.58 for cubes in simulations, and 1.06 for balls and 1.45 for cubes in the real system.

Future planetary missions will rely on rovers that can autonomously explore and navigate in unstructured environments. An essential element is the ability to recognize places that were already visited or mapped. In this work, we leverage the ability of stereo cameras to provide both visual and depth information, guiding the search and validation of loop closures from a multi-modal perspective. We propose to augment submaps that are created by aggregating stereo point clouds, with visual keyframes. Point clouds matches are found by comparing CSHOT descriptors and validated by clustering, while visual matches are established by comparing keyframes using Bag-of-Words (BoW) and ORB descriptors. The relative transformations resulting from both keyframe and point cloud matches are then fused to provide pose constraints between submaps in our graph-based SLAM framework. Using the LRU rover, we performed several tests in both an indoor laboratory environment as well as a challenging planetary analog environment on Mount Etna, Italy, consisting of areas where either keyframes or point clouds alone failed to provide adequate matches demonstrating the benefit of the proposed multi-modal approach.

The Visibility-based Persistent Monitoring (VPM) problem seeks to find a set of trajectories (or controllers) for robots to persistently monitor a changing environment. Each robot has a sensor, such as a camera, with a limited field-of-view that is obstructed by obstacles in the environment. The robots may need to coordinate with each other to ensure no point in the environment is left unmonitored for long periods of time. We model the problem such that there is a penalty that accrues every time step if a point is left unmonitored. However, the dynamics of the penalty are unknown to us. We present a Multi-Agent Reinforcement Learning (MARL) algorithm for the VPM problem. Specifically, we present a Multi-Agent Graph Attention Proximal Policy Optimization (MA-G-PPO) algorithm that takes as input the local observations of all agents combined with a low resolution global map to learn a policy for each agent. The graph attention allows agents to share their information with others leading to an effective joint policy. Our main focus is to understand how effective MARL is for the VPM problem. We investigate five research questions with this broader goal. We find that MA-G-PPO is able to learn a better policy than the non-RL baseline in most cases, the effectiveness depends on agents sharing information with each other, and the policy learnt shows emergent behavior for the agents.

Cable-Driven Parallel Robots (CDPRs) are parallel robots with rigid links replaced by cables. As for most parallel robots the determination of the analytical solutions to the direct geometrico-static model (DGSM) is a difficult task that is often not feasible online. However, the knowledge of the moving-platform (MP) pose is necessary in order to control the CDPR, e.g. with visual servoing. When the MP pose measurement is not available, an estimation can be sufficient. This paper compares three estimation methods: (a) control-based; (b) image-based; and (c) model-based. The three methods are implemented experimentally with an open-loop velocity controller and a closed-loop visual servoing controller. Overall, very good results are shown with model-based and control-based methods for both controllers. Finally, it is shown that the visual servoing controller leads to a better accuracy of the robot than the velocity controller.

We propose a new deep learning framework to decompose monocular videos into 3D geometry (camera pose and depth), moving objects, and their motions, with no supervision. We build upon the idea of view synthesis, which uses classical camera geometry to re-render a source image from a different point-of-view to obtain supervisory signals, specified by a predicted relative 6-degree-of-freedom pose and depth map. However, the typical view synthesis equations rely on a strong assumption: that objects in scenes do not move. This rigid-world assumption limits the predictive power, and rules out learning about objects automatically. We propose a simple solution: minimize the synthesis error on small local regions of the image instead. While the scene as a whole may be non-rigid, it is always possible to find small regions that are approximately rigid, such as inside a moving object. Our network can learn a dense pose map describing poses for each local region. This represents a significantly richer model, including 6D object motions, with little additional complexity. We establish very competitive results on unsupervised odometry and depth prediction on KITTI. We also demonstrate new capabilities on EPIC-Kitchens, a challenging dataset of indoor videos, where there is no ground truth information for depth, odometry, object segmentation or motion - yet all are recovered automatically by our approach.