IROS 2019 Papers

Thermal cameras can enable autonomous flight at night without GPS. However, image-based navigation in the thermal infrared spectrum has been researched significantly less than in the visible spectrum. In this paper, we demonstrate closed-loop controlled outdoor flights at night on a quadrotor. Our state estimator can tightly couple inertial data with either thermal images at nighttime, or visual images at daytime. It is integrated in an autonomy framework for motion planning and control, which runs in real time on a standard embedded computer. We analyze thermal-inertial odometry performance extensively from sunset to sunrise, for various thermal non-uniformity levels, and compare it to visual-inertial odometry at daytime.

Robotic airborne chemical source localization has critical applications ranging from search and rescue to hazard detection to pollution assessment. Previous demonstrations on flying robots have required search times in excess of ten minutes, or required computation-intensive signal processing, largely because of the slow response of semiconductor gas sensors. To mitigate these limitations, we developed a hybrid biological/synthetic chemical sensing platform consisting of a moth antenna on an aerial robot. We demonstrate that our robot, a 9 centimeter nano drone, can repeatedly detect and reach the source of a volatile organic chemical plume in less than a minute. We also introduce wind vanes to passively aim the robot upwind, greatly simplifying control. To our knowledge this is the first odor-finding robot to use this approach, and it allows for localization using feedback only from sensors carried on-board rather than GPS, allowing indoor operation. The chemical sensor consists of a hybrid biological/synthetic integrated chemical sensor (electroantennogram) using an excised antenna of the hawkmoth Manduca sexta and associated miniaturized electrophysiology conditioning circuitry. Our robot performs an insect-inspired cast-and-surge search algorithm inspired by the odor-tracking behavior observed in Manduca sexta. These results represent a significant step toward robots that have the speed and sensitivity of biological systems.

Human supervisory control (HSC) is a widely used knowledge-based control scheme, in which human operators are in charge of planning and making high-level decisions for systems with embedded autonomy. With the variability of operators' behaviors in such systems, the stability of an operator modeling technique, i.e., that a modeling approach produces similar results across repeated applications, is critical to the extensibility and utility of such a model. Using an unmanned vehicle simulation testbed where such vehicles can be hacked, we compared two operator behavioral models from two different experiments using a hidden Markov modeling (HMM) approach. The resulting HMM models revealed operators' dominant strategies when conducting hacking detection tasks. The similarity between these two models was measured via multiple aspects, including model structure, state distribution, divergence distance, and co-emission probability distance. The similarity measure results demonstrate the stability of modeling human operators in HSC scenarios using HMM models. These results indicate that even when operators perform differently on specific tasks, such an approach can reliably detect whether strategies change across different experiments.

In practical robotic applications various types of tools are attached for manipulating objects. Besides adding gravitational load to the robot, which results in larger joint torques, such payloads influence the collision safety characteristics through changing surface curvature properties, reflected mass and effective robot speed along a desired motion direction. In this paper, we evaluate the effect a known, unactuated pay-load that is attached to the end-effector has on the intantaneous reflected inertial parameters and maximum task velocity of a robot. The proposed mass update approach relies on the analysis of the kinetic energy matrices, while the velocity maximization is tackled by formulating static optimization problems with different constraints on angular motion of the end-effector. Finally, for analyzing the validity of the introduced approach in the framework of Safety Maps, we discuss simulation results of a PUMA 560 robot that has an exemplary payload attached to its end-effector.

This paper consists of a failure analysis of two robot performance productions. Both productions included three-week rehearsal periods, and culminated in live performances including both robots and humans. To develop these productions, a theater artist collaborated with a robotics lab to develop, (1) a narrative dance performed live on stage and (2) an improvisational performance in a public space. While the interdisciplinary team did not set out to explore robot failures, failures played an ever-present role during the eighteen rehearsals and two live performances. This paper details strategies for addressing, planning for, and rehearsing responses to robot failures on stage, both technical and choreographic. In addition to scaffolding future robot theater performances, we discuss how these strategies apply to other customer- and audience-facing robots, including sponsor demos. The on-stage exploration of robot chairs and human performers also suggests that humans can conceptualize minimal robots as both characters and props, moving fluidly from one to the other. We hope these strategies ensure that future audiences will want the robot shows to go on, as well as expand ideas about the types of robots that can be cast in future human-robot productions.

Deep networks have brought significant advances in robot perception, enabling to improve the capabilities of robots in several visual tasks, ranging from object detection and recognition to pose estimation, semantic scene segmentation and many others. Still, most approaches typically address visual tasks in isolation, resulting in overspecialized models which achieve strong performances in specific applications but work poorly in other (often related) tasks. This is clearly sub-optimal for a robot which is often required to perform simultaneously multiple visual recognition tasks in order to properly act and interact with the environment. This problem is exacerbated by the limited computational and memory resources typically available onboard to a robotic platform. The problem of learning flexible models which can handle multiple tasks in a lightweight manner has recently gained attention in the computer vision community and benchmarks supporting this research have been proposed. In this work we study this problem in the robot vision context, proposing a new benchmark, the RGB-D Triathlon, and evaluating state of the art algorithms in this novel challenging scenario. We also define a new evaluation protocol, better suited to the robot vision setting. Results shed light on the strengths and weaknesses of existing approaches and on open issues, suggesting directions for future research.

The progress of obstacle detection via semantic segmentation on unmanned surface vehicles (USVs) has been significantly lagging behind the developments in the related field of autonomous cars. The reason is the lack of large curated training datasets from USV domain required for development of data-hungry deep CNNs. This paper addresses this issue by presenting MaSTr1325, a marine semantic segmentation training dataset tailored for development of obstacle detection methods in small-sized coastal USVs. The dataset contains 1325 diverse images captured over a two year span with a real USV, covering a range of realistic conditions encountered in a coastal surveillance task. The images are per-pixel semantically labeled. The dataset exceeds previous attempts in this domain in size, scene complexity and domain realism. In addition, a dataset augmentation protocol is proposed to address slight appearance differences of the images in the training set and those in deployment. The accompanying experimental evaluation provides a detailed analysis of popular deep architectures, annotation accuracy and influence of the training set size. MaSTr1325 will be released to research community to facilitate progress in obstacle detection for USVs.

Recent advances in deep learning–based object detection techniques have revolutionized their applicability in several fields. However, since these methods rely on unwieldy and large amounts of data, a common practice is to download models pre-trained on standard datasets and fine-tune them for specific application domains with a small set of domain–relevant images. In this work, we show that using synthetic datasets that are not necessarily photo-realistic can be a better alternative to simply fine-tune pre-trained networks. Specifically, our results show an impressive 25%improvement in the mAP metric over a fine-tuning baseline when only about 200 labelled images are available to train. Finally, an ablation study of our results is presented to delineate the individual contribution of different components in the randomization pipeline.

A robot can now grasp an object more effectively than ever before, but once it has the object what happens next? We show that a mild relaxation of the task and workspace constraints implicit in existing object grasping datasets can cause neural network based grasping algorithms to fail on even a simple block stacking task when executed under more realistic circumstances. To address this, we introduce the JHU CoSTAR Block Stacking Dataset (BSD), where a robot interacts with 5.1 cm colored blocks to complete an order-fulfillment style block stacking task. It contains dynamic scenes and real time-series data in a less constrained environment than comparable datasets. There are nearly 12,000 stacking attempts and over 2 million frames of real data. We discuss the ways in which this dataset provides a valuable resource for a broad range of other topics of investigation. We find that hand-designed neural networks that work on prior datasets do not generalize to this task. Thus, to establish a baseline for this dataset, we demonstrate an automated search of neural network based models using a novel multiple-input HyperTree MetaModel, and find a final model which makes reasonable 3D pose predictions for grasping and stacking on our dataset. The CoSTAR BSD, code, and instructions are available at sites.google.com/site/costardataset.

We present the ANBOT, an intelligent robotic coworker for physical human-robot collaboration. The ANBOT system assists workers performing industrial abrasive blasting, shielding them from the large forces experienced during this physically demanding task. The co-operative robotic system combines the strength and endurance of robots with the decision making of skilled workers. The inherent challenges in human-robot collaboration, combined with the difficult blasting environment required novel design decisions to be made and new solutions to be developed. These include an approach for handling kinematic singularities in a manner suitable for human-robot co-operation, estimating worker pose under poor visibility conditions, and an intuitive control scheme that adapts the robotic assistance based on the estimated strength of the worker. In this work we summarise the ANBOT system and present findings from preliminary site trials. The trials included several real industrial blasting tasks under the control of a skilled abrasive blasting worker who had no experience working alongside a robot. Results demonstrate the suitability of the ANBOT for practical industrial applications.

This paper presents an algorithm which can generate live digtial surface model (DSM) during the flight based on simultaneous localization and mapping (SLAM). We process the keyframe which is output by a monocular SLAM system to generate a local DSM, and fuse the local DSM to the global tiled DSM incrementally. During the local DSM generation, a local digital elevation model (DEM) is estimated by projecting the filtered 2D Delaunay mesh to a 3D mesh, and a local orthomosaic is obtained by projecting triangle image patches onto a 2D mesh. During the DSM fusion, both the local DEM and orthomosaic are split into tiles and fused to the global tiled DEM and orthomosaic respectively with multiband algorithm. Both the efficient DSM generation and fusion algorithms contribute to achieving a real-time reconstruction. Qualitative and quantitative experiments on a public aerial image dataset with different scenarios are performed to validate the effectiveness of the proposed method. Compared with traditional structure from motion (SfM) based approaches, the presented system is able to output both large-scale high-quality DEM and orthomosaic in real-time with low computational cost.

Deep reinforcement learning algorithms have been proven successful in a variety of simulation tasks with dense reward feedback. However, real-world RL applications, e.g. robotic manipulation, remain challenging as most of them are multi-stage and a positive reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward shaping mechanism, which provides the robot with additional hints based on a discriminative learning on past experience. The reward along with a stage-awareness network help accelerate solving a multistage task split into shorter phases in a reverse curriculum learning manner. We extensively study the advantages of TRL on the standard long-term goal-oriented robotics domains such as pick-and-place, and show that TRL performs more efficiently and robustly than prior approaches in tasks with large state space. In addition, we demonstrate that TRL can solve difficult robot manipulation challenges directly from perception.

Sending humans to Mars' surface to build habitats is, for now, prohibitively dangerous and costly. An alternative is to have humans in orbiters, teleoperating robots on Mars to construct habitats, deferring human arrival until these habitats are finished. This paper describes the Kontur-2 experiments, in which the feasibility of this scenario was tested with the International Space Station as an orbiter, a cosmonaut operating a force-feedback joystick as an input device for teleoperation, and Earth as the planet where the teleoperated robot is located. In particular, we focus on human teleoperation performance, which is known to deteriorate under conditions of spaceflight. We investigate whether the provision of force feedback at the joystick is as beneficial as under terrestrial conditions. Our results show that, to support humans operating in weightlessness, haptic assistance needs to be adjusted to the altered environmental condition.

To operate successfully in the unstructured environment of homes and small businesses, robots will be implemented by unskilled operators who cannot explicitly program their motions. Recently, imitation learning has been used to train robots that manipulate their environment based on pixel-to-action control. The robot actions are determined by camera inputs without programmed trajectories. Such training is often performed on robotic hardware which was not designed for imitation learning. This paper describes a new robot which is designed expressly to improve the training speed and ability of pixel-to-action policies. In particular, hexapod robot hardware is designed for teleoperation, thus improving correspondence and simplifying human control.

In this paper, we present a new method based on the Aerial Social Force Model (ASFM) to allow human-drone side-by-side social navigation in real environments. To tackle this problem, the present work proposes a new nonlinear-based approach using Neural Networks. To learn and test the rightness of the new approach, we built a new dataset with simulated environments and we recorded motion controls provided by a human expert tele-operating the drone. The recorded data is then used to train a neural network which maps interaction forces to acceleration commands. The system is also reinforced with a human path prediction module to improve the drone's navigation, as well as, a collision detection module to completely avoid possible impacts. Moreover, a performance metric is defined which allows us to numerically evaluate and compare the fulfillment of the different learned policies. The method was validated by a large set of simulations; we also conducted real-life experiments with an autonomous drone to verify the framework described for the navigation process. In addition, a user study has been realized to reveal the social acceptability of the method.

The musculoskeletal humanoid has many benefits that human beings have, but the modeling of its complex flexible body is difficult. Although we have developed an online acquisition method of the nonlinear relationship between joints and muscles, we could not completely match the actual robot and its self-body image. When realizing a certain task, the direct relationship between the control input and task state needs to be learned. So, we construct a neural network representing the time-series relationship between the control input and task state, and realize the intended task state by applying the network to a real-time control. In this research, we conduct accelerator pedal control experiments as one application, and verify the effectiveness of this study.

In task-oriented grasping, the robot is supposed to manipulate the objects in a task-compatible manner, which is more important but more challenging than just stably grasping. However, most of existing works perform task-oriented grasping only in single object scenes. This greatly limits their practical application in real world scenes, in which there are usually multiple stacked objects with serious overlaps and occlusions. To perform task-oriented grasping in object stacking scenes, in this paper, we firstly build a synthetic dataset named Object Stacking Grasping Dataset (OSGD) for task-oriented grasping in object stacking scenes. Secondly, a Conditional Random Field (CRF) is constructed to model the semantic contents in object regions. The modelled semantic contents can be illustrated as incompatibility of task labels and continuity of task regions. This proposed approach can greatly reduce the interference of overlaps and occlusions in object stacking scenes. To embed the CRF-based semantic model into our grasp detection network, we implement the inference process of CRFs as a RNN so that the whole model, Task-oriented Grasping CRFs (TOG-CRFs) can be trained end to end. Finally, in object stacking scenes, the constructed model can help robot achieve 69.4% success rate for task-oriented grasping.

Task-motion planning (TMP) addresses the problem of efficiently generating executable and low-cost task plans in a discrete space such that the (initially unknown) action costs are determined by motion plans in a corresponding continuous space. A task-motion plan for a mobile service robot that behaves in a highly dynamic domain can be sensitive to domain uncertainty and changes, leading to suboptimal behaviors or execution failures. In this paper, we propose a novel framework, TMP-RL, which is an integration of TMP and reinforcement learning (RL), to solve the problem of robust TMP in dynamic and uncertain domains. The robot first generates a low-cost, feasible task-motion plan by iteratively planning in the discrete space and updating relevant action costs evaluated by the motion planner in continuous space. During execution, the robot learns via model-free RL to further improve its task-motion plans. RL enables adaptability to the current domain, but can be costly with regards to experience; using TMP, which does not rely on experience, can jump-start the learning process before executing in the real world. TMP-RL is evaluated in a mobile service robot domain where the robot navigates in an office area, showing significantly improved adaptability to unseen domain dynamics over TMP and task planning (TP)-RL methods.

We propose a novel pipeline that blends encodings from natural language and 3D semantic maps obtained from visual imagery to generate local trajectories that are executed by a low-level controller. The pipeline precludes the need for a prior registered map through a local waypoint generator neural network. The waypoint generator network (WGN) maps semantics and natural language encodings (NLE) to local waypoints. A local planner then generates a trajectory from the ego location of the vehicle (an outdoor car in this case) to these locally generated waypoints while a low-level controller executes these plans faithfully. The efficacy of the pipeline is verified in the CARLA simulator environment as well as on local semantic maps built from real-world KITTI dataset. In both these environments (simulated and real-world) we show the ability of the WGN to generate waypoints accurately by mapping NLE of varying sequence lengths and levels of complexity. We compare with baseline approaches and show significant performance gain over them. And finally, we show real implementations on our electric car verifying that the pipeline lends itself to practical and tangible realizations in uncontrolled outdoor settings. In loop execution of the proposed pipeline that involves repetitive invocations of the network is critical for any such language-based navigation framework. This effort successfully accomplishes this thereby bypassing the need for prior metric maps or strategies for metric level localization during traversal.