IROS 2019 Papers

This paper presents a systematic approach on realtime reconstruction of an underwater environment using Sonar, Visual, Inertial, and Depth data. In particular, low lighting conditions, or even complete absence of natural light inside caves, results in strong lighting variations, e.g., the cone of the artificial video light intersecting underwater structures, and the shadow contours. The proposed method utilizes the well defined edges between well lit areas and darkness to provide additional features, resulting into a denser 3D point cloud than the usual point clouds from a visual odometry system. Experimental results in an underwater cave at Ginnie Springs, FL, with a custom-made underwater sensor suite demonstrate the performance of our system. This will enable more robust navigation of autonomous underwater vehicles using the denser 3D point cloud to detect obstacles and achieve higher resolution reconstructions.

Common formulations to consider collision avoidance in trajectory optimization often use either preprocessed environments or only check and penalize collisions at discrete time steps. However, when only checking at discrete states, this requires either large margins that prevent manipulation close to obstacles or dense time discretization increasing the dimensionality of the optimization problem in complex environments. Nonetheless, collisions may still occur in the interpolation/transition between two valid states or in environments with thin obstacles. In this work, we introduce a computationally inexpensive continuous-time collision avoidance term in presence of static and moving obstacles. Our penalty is based on conservative advancement and harmonic potential fields and can be used as either a cost or constraint in off-the-shelf non-linear programming solvers. Due to the use of conservative advancement (collision checks) rather than distance computations, our method outperforms discrete collision avoidance based on signed distance constraints resulting in smooth motions with continuous-time safety while planning in discrete time. We evaluate our proposed continuous collision avoidance on scenarios including manipulation of moving targets, locomanipulation on mobile robots, manipulation trajectories for humanoids, and quadrotor path planning and compare penalty terms based on harmonic potential fields with ones derived from contact normals.

In the context of future manufacturing lines, removing fixtures will be a fundamental step to increase the flexibility of autonomous systems in assembly and logistic operations. Vision-based 3D pose estimation is a necessity to accurately handle objects that might not be placed at fixed positions during the robot task execution. Industrial tasks bring multiple challenges for the robust pose estimation of objects such as difficult object properties, tight cycle times and constraints on camera views. In particular, when interacting with objects, we have to work with close-range partial views of objects that pose a new challenge for typical view-based pose estimation methods.In this paper, we present a 3D pose estimation method based on a gradient-ascend particle filter that integrates new observations on-the-fly to improve the pose estimate. Thereby, we can apply this method online during task execution to save valuable cycle time. In contrast to other view-based pose estimation methods, we model potential views in full 6dimensional space that allows us to cope with close-range partial objects views. We demonstrate the approach on a real assembly task, in which the algorithm usually converges to the correct pose within 10-15 iterations with an average accuracy of less than 8mm.

We address one-shot imitation learning, where the goal is to execute a previously unseen task based on a single demonstration. While there has been exciting progress in this direction, most of the approaches still require a few hundred tasks for meta-training, which limits the scalability of the approaches. Our main contribution is to formulate one-shot imitation learning as a symbolic planning problem along with the symbol grounding problem. This formulation disentangles the policy execution from the inter-task generalization and leads to better data efficiency. The key technical challenge is that the symbol grounding is prone to error with limited training data and leads to subsequent symbolic planning failures. We address this challenge by proposing a continuous relaxation of the discrete symbolic planner that directly plans on the probabilistic outputs of the symbol grounding model. Our continuous relaxation of the planner can still leverage the information contained in the probabilistic symbol grounding and significantly improve over the baseline planner for the one-shot imitation learning tasks without using large training data.

As robots start to execute complex manipulation tasks, they are expected to improve their skill set over time as humans do. A prominent approach to accomplish this is having robots to keep models of their actions based on their experiences in order to improve their action executions in the future. In this paper, we present such a methodology where robots start to execute some actions with random parameters and record their generic execution logs with semantic annotations in a symbolic knowledge base for robots. Using the data inside logs, multivariate Gaussian mixture models are fitted to the high-level action parameters for later executions. These affordance models are being updated whenever a new execution is carried on. In essence, robots can use these continuously-updated probabilistic model for improving their actions To prove the applicability we demonstrate opening-a-fridge-door experiments with a PR2 robot.

A continuous collision checking method for a cable-driven parallel robot with an embarked robotic arm is proposed in this paper. The method aims at validating paths by checking for collisions between any pair of robot bodies (mobile platform, cables, and arm links). For a pair of bodies, an upper bound on their relative velocity and a lower bound on the distance between the bodies are computed and used to validate a portion of the path. These computations are done repeatedly until a collision is found or the path is validated. The method is integrated within the Humanoid Path Planner (HPP) software, tested with the cable-driven parallel robot CoGiRo, and compared to a discretized validation method.

We present a context-dependent bi-directional tree-search framework for point-to-point path planning for manipulators. Conceptually, our framework is composed of six modules: tree selection, focus selection, node selection, target selection, extend selection and connection type selection. Each module consists of a set of interchangeable strategies. By exploiting synergistic interaction between these strategies and selecting appropriate strategies based the contextual cues from the search state, we show an instance of our framework that computes high-quality solutions in a variety of complex scenarios with a low failure rate. We also show that some popular path planning methods in the literature can be easily represented in our framework. We compare our approach with these popular methods in a diverse set of test scenarios. We report a 15-fold reduction in failure rate coupled with at least a 26% drop in solution suboptimality when compared to the best of the alternative methods.

We present a novel autonomous driving system which uses the road contextual information and intentions of other road users for urban driving. Unlike highways, urban environments require the drivers to follow traffic signs and signals while using their best judgment for anomalous situations. In such scenarios, a self-driving car needs to understand and take into account the uncertainties in the environment to plan and decide its action accordingly. Our planner models the intentions of the surrounding vehicles leveraging a neural network, and integrates the road contextual information to reduce its environment uncertainties and also speed up the decision making process. We validate our planner in simulation and in a real urban environment. Our experimental results show that integrating intention inference and road contextual information for prediction, planning and decision making help improve safety and efficiency of our autonomous driving system.

Grasping and manipulating objects is an important human skill. Since most objects are designed to be manipulated by human hands, anthropomorphic hands can enable richer human-robot interaction. Desirable grasps are not only stable, but also functional: they enable post-grasp actions with the object. However, functional grasp synthesis for high degree-of-freedom anthropomorphic hands from object shape alone is challenging. We present ContactGrasp, a framework for functional grasp synthesis from object shape and contact on the object surface. Contact can be manually specified or obtained through demonstrations. Our contact representation is object-centric and allows functional grasp synthesis even for hand models different than the one used for demonstration. Using a dataset of contact demonstrations from humans grasping diverse household objects, we synthesize functional grasps for three hand models and two functional intents. The project webpage is https://contactdb.cc.gatech.edu/contactgrasp.html.

We present a reformulation of a contact-implicit optimization (CIO) approach that computes optimal trajectories for rigid-body systems in contact-rich settings. A hard-contact model is assumed, and the unilateral constraints are imposed in the form of complementarity conditions. Newton's impact law is adopted for enhanced physical correctness. The optimal control problem is formulated as a multi-staged program through a multiple-shooting scheme. This problem structure is exploited within the FORCES Pro framework to retrieve optimal motion plans, contact sequences and control inputs with increased computational efficiency. We investigate our method on a variety of dynamic object manipulation tasks, performed by a six degrees of freedom robot. The dynamic feasibility of the optimal trajectories, as well as the repeatability and accuracy of the task-satisfaction are verified through simulations and real hardware experiments on one of the manipulation problems.

Robotic automation is a key driver for the advancement of technology. The skills of human workers, however, are difficult to program and seem currently unmatched by technical systems. In this work we present a data-driven approach to extract and learn robot-independent contact skills from human demonstrations in simulation environments, using a Long Short Term Memory (LSTM) network. Our model learns to generate error-correcting sequences of forces and torques in task space from object-relative motion, which industrial robots carry out through a Cartesian force control scheme on the real setup. This scheme uses forward dynamics computation of a virtually conditioned twin of the manipulator to solve the inverse kinematics problem. We evaluate our methods with an assembly experiment, in which our algorithm handles part tilting and jamming in order to succeed. The results show that the skill is robust towards localization uncertainty in task space and across different joint configurations of the robot. With our approach, non-experts can easily program force-sensitive assembly tasks in a robot-independent way.

Research in the areas related to driving behavior, e.g., behavior modeling and prediction, requires datasets with highly interactive vehicle motions. Existing public vehicle motion datasets emphasize increasing the number of vehicles and time duration, but behavior-related researchers are suffering from two factors. First, strong interactions among vehicles are not well addressed and datasets are of relatively low-density to observe meaningful interactions. Second, most of the existing datasets are missing the map information with reference paths which is essential for driving-behavior-related research. To address this issue, a dataset with highly interactive vehicle motions is constructed in this paper. A variety of challenging driving scenarios such as unsignalized intersections and roundabouts are included. Reference paths are also constructed from motion data along with high-definition maps so that key features can be generated for both prediction and planning algorithms. Moreover, we propose a set of metrics to extract the interactive motions in different maps, including the minimum difference of time to collision point (MDTTC) and duration of waiting period. Such metrics are used to quantity the interaction density of the dataset. We also give several representative results on prediction and motion generation utilizing the constructed dataset to demonstrate how the dataset can facilitate research in the area of driving behavior.

There is a strong demand for covering a large area autonomously by multiple UAVs (Unmanned Aerial Vehicles) supported by a ground vehicle. Limited by UAVs' battery life and communication distance, complete coverage of large areas typically involves multiple take-offs and landings to recharge batteries, and the transportation of UAVs between operation areas by a ground vehicle. In this paper, we introduce a novel large-area-coverage planning framework which collectively optimizes the paths for aerial and ground vehicles. Our method first partitions a large area into sub-areas, each of which a given fleet of UAVs can cover without recharging batteries. UAV operation routes, or trails, are then generated for each sub-area. Next, the assignment of trials to different UAVs and the order in which UAVs visit their assigned trails are simultaneously optimized to minimize the total UAV flight distance. Finally, a ground vehicle transportation path which visits all sub-areas is found by solving an asymmetric traveling salesman problem (ATSP). Although finding the globally optimal trail assignment and transition paths can be formulated as a Mixed Integer Quadratic Program (MIQP), the MIQP is intractable even for small problems. We show that the solution time can be reduced to close-to-real-time levels by first finding a feasible solution using a Random Key Genetic Algorithm (RKGA), which is then locally optimized by solving a much smaller MIQP.

During swarm teleoperation, the operator may threaten the distance-dependent inter-robot communications and, with them, the connectivity of the slave swarm. To prevent the operator from disconnecting the swarm, this paper develops a constructive strategy to dynamically modulate the interconnections of, and the local damping injections at, all slave robots. Lyapunov-based set invariance analysis shows that the strategy preserves all interaction links in the tree network while synchronizing the slave swarm. By properly limiting the impact of the user command rather than rejecting it entirely, the proposed explicit gain update law enables the operator to guide the motion of the slave swarm to the extent to which it does not endanger swarm connectivity. An experiment illustrates that the proposed strategy can maintain the tree network connectivity of a teleoperated swarm.

This paper presents an approach for modeling new soft robotic materials which possess the ability to control directional stiffness. These materials are inspired by biological systems where movements are enabled by variable stiffness tissue and contraction of localized muscle groups. Here a low-melting-point (LMP) material lattice embedded in an elastomer serves as a rigid skeleton that may be locally melted to allow bending at selectable joint locations. The forward kinematics of the lattice has been modeled using the product of exponentials method with the incorporation of bending axis selectivity. In this paper, we develop this model to account for torques imposed by tendons, and we model the elastomer's resistance to bending as a torsional spring at the selected joints. Thus we obtain a two-way relationship between tendon forces and joint angles/axes. The concept of applying traditional robot modeling strategies to selectively compliant robotic structures could enable precise control of dexterous soft robots that satisfy stringent safety criteria.

Effective understanding of the environment and accurate trajectory prediction of surrounding dynamic obstacles are critical for intelligent systems such as autonomous vehicles and wheeled mobile robotics navigating in complex scenarios to achieve safe and high-quality decision making, motion planning and control. Due to the uncertain nature of the future, it is desired to make inference from a probability perspective instead of deterministic prediction. In this paper, we propose a conditional generative neural system (CGNS) for probabilistic trajectory prediction to approximate the data distribution, with which realistic, feasible and diverse future trajectory hypotheses can be sampled. The system combines the strengths of conditional latent space learning and variational divergence minimization, and leverages both static context and interaction information with soft attention mechanisms. We also propose a regularization method for incorporating soft constraints into deep neural networks with differentiable barrier functions, which can regulate and push the generated samples into the feasible regions. The proposed system is evaluated on several public benchmark datasets for pedestrian trajectory prediction and a roundabout naturalistic driving dataset collected by ourselves. The experimental results demonstrate that our model achieves better performance than various baseline approaches in terms of prediction accuracy.

We present the concept of concurrent flow-based localization and mapping (FLAM) for autonomous field robots navigating within background flows. Different from the classical simultaneous localization and mapping (SLAM) problem, where the robot interacts with discrete features, FLAM utilizes the continuous flow fields as navigation references for mobile robots and provides flow field mapping capability with in-situ flow velocity observations. This approach is of importance to underwater vehicles in mid-depth oceans or aerial vehicles in GPS-denied atmospheric circulations. This article introduces the formulation of FLAM as a full SLAM solution motivated by the feature-based GraphSLAM framework. The performance of FLAM was demonstrated through simulation within artificial flow fields that represent typical geophysical circulation phenomena: a steady single-gyre flow field and a double-gyre flow field with unsteady turbulent perturbations. The results indicate that FLAM provides significant improvements in the robots' localization accuracy and a consistent approximation of the background flow field. It is also shown that FLAM leads to smooth robot trajectory estimates.

In production line, the conventional industrial robots and automatic machines require machinery safety protection to guarantee the safety of human operators. A scalable and easy-to-conFigure safety system concept called “Real-time Safety Virtual Positioning” (RSVP) is proposed, which could act as potentially key enabler for agile production systems by eliminating fixed safety installation and thus increasing productivity and flexibility. The RSVP provides easy access to robots of automatic assembly lines in plants, e.g., automotive OEM, and supports virtualization and transparent to fully automatic and semi-automatic assembly line by knowing the position of persons and relevant objects (e.g., tools, finished and/or semi-finished goods, and materials). The focus of the paper will discuss the functional safety certification realization (concept approved by TÜV (Technical Inspection Association)) and validation details of indoor localization-based safety system developments. The detailed strategy of the functional safety requirements, danger diagnosis and reaction approach, communication among safety controller, robot and machine, and safe failure reaction are listed and analyzed. The physical hardware framework and software architecture of the safety system are built and developed with the 1oo2-architecture according to the Performance Levels (PL d) of ISO 13849 and the Safety Integrity Levels (SIL 2) of IEC 61508. The implicated algorithms and data process of the UWB-based (Ultra-Wide Band) indoor localization system are introduced. The safety system concept is validated and verified in an ABS (Anti-lock Braking System) production line with human-robot co-existence environment.

In this paper we consider the problem of computing an optimal set of motion primitives for a lattice planner. The objective we consider is to compute a minimal set of motion primitives that t-span a configuration space lattice. A set of motion primitives t-span a lattice if, given a real number t greater or equal to one, any configuration in the lattice can be reached via a sequence of motion primitives whose cost is no more than t times the cost of the optimal path to that configuration. Determining the smallest set of t-spanning motion primitives allows for quick traversal of a state lattice in the context of robotic motion planning, while maintaining a t-factor adherence to the theoretically optimal path. While several heuristics exist to determine a t-spanning set of motion primitives, these are presented without guarantees on the size of the set relative to optimal. This paper provides a proof that the minimal t-spanning control set problem for a lattice defined over an arbitrary robot configuration space is NP-complete, and presents a compact mixed integer linear programming formulation to compute an optimal t-spanner. We show that solutions obtained by the mixed integer linear program have significantly fewer motion primitives than state of the art heuristic algorithms, and out perform a set of standard primitives used in robotic path planning.