IROS 2019 Papers

Some robots can interact with humans using natural language, and identify service requests through human-robot dialog. However, few robots are able to improve their language capabilities from this experience. In this paper, we develop a dialog agent for robots that is able to interpret user commands using a semantic parser, while asking clarification questions using a probabilistic dialog manager. This dialog agent is able to augment its knowledge base and improve its language capabilities by learning from dialog experiences, e.g., adding new entities and learning new ways of referring to existing entities. We have extensively evaluated our dialog system in simulation as well as with human participants through MTurk and real-robot platforms. We demonstrate that our dialog agent performs better in efficiency and accuracy in comparison to baseline learning agents. Demo video can be found at https://youtu.be/DFB3jbHBqYE.

The design of augmented reality interfaces for people with mobility impairments is a novel area with great potential, as well as multiple outstanding research challenges. In this paper we present an augmented reality user interface for controlling a smart wheelchair with a head-mounted display to provide assistance for mobility restricted people. Our motivation is to reduce the cognitive requirements needed to control a smart wheelchair. A key element of our platform is the ability to control the smart wheelchair using the concepts of affordances and signifiers. In addition to the technical details of our platform, we present a baseline study by evaluating our platform through user-trials of able-bodied individuals and two different affordances: 1) Door Go Through and 2) People Approach. To present these affordances to the user, we evaluated fixed symbol based signifiers versus our novel dynamic signifiers in terms of ease to understand the suggested actions and its relation with the objects. Our results show a clear preference for dynamic signifiers. In addition, we show that the task load reported by participants is lower when controlling the smart wheelchair with our augmented reality user interface compared to using the joystick, which is consistent with their qualitative answers.

We present an audio-visual dataset recorded outdoors from a quadcopter and discuss baseline results for multiple applications. The dataset includes a scenario for source localization and sound enhancement with up to two static sources, and a scenario for source localization and tracking with a moving sound source. These sensing tasks are made challenging by the strong and time-varying ego-noise generated by the rotating motors and propellers. The dataset was collected using a small circular array with 8 microphones and a camera mounted on the quadcopter. The camera view was used to facilitate the annotation of the sound-source positions and can also be used for multi-modal sensing tasks. We discuss the audio-visual calibration procedure that is needed to generate the annotation for the dataset, which we make available to the research community.

Performing safe and efficient lane changes is a crucial feature for creating fully autonomous vehicles. Recent advances have demonstrated successful lane following behavior using deep reinforcement learning, yet the interactions with other vehicles on-road for lane changes are rarely considered. In this paper, we design a hierarchical Deep Reinforcement Learning (DRL) algorithm to learn lane change behaviors in dense traffic. By breaking down overall behavior to sub-policies, faster and safer lane change actions can be learned. We also apply temporal and spatial attention to the DRL architecture, which helps the vehicle focus more on surrounding vehicles and leads to smoother lane change behavior. We conduct our experiments in the TORCS simulator and the results outperform the state-of-the-art deep reinforcement learning algorithm in various lane change scenarios.

Scanning Electron Microscopy (SEM) is an ideal observation tool for small scales robotics. It has the potential to achieve automated nano-robotic tasks such as nano-handling and nano-assembly. Path following control of nano-robot end effectors using SEM vision feedback is a key for an intuitive programming of elementary robotic tasks sequences. It requires the ability to track end effectors under various SEM scan speeds. SEM suffers however from tricky issues that limits robotic tracking capabilities. This paper focuses on one specific issue related to the compromise between the scan speed and the image quality. This restriction seriously limits the performance of conventional vision tracking algorithms when used with electron images. At high scan speed, the image quality is very noisy making very difficult to differentiate the robot end effector from the background, hence limiting the tracking capabilities. The work related in this paper explores for the first time the potential value of Convolutional Neural Networks (ConvNet) in the context of nano-robotic vision tracking inside SEM. The aim is to localize an end-effector, AFM cantilever in the case of the study, from SEM images for any scan speed configuration and despite of low images quality. For that purpose, a data set of AFM tip images is build up from SEM images for the learning algorithm. Network performances are estimated under different SEM scan speeds. Thanks to the learning algorithm, experimental results show robust AFM tip tracking capabilities inside the SEM under various scan speed conditions.

Complex aircraft systems are becoming a target for automation. For successful operation, they require both efficient and readable mission execution system (MES). Flight control computer (FCC) units, as well as all important subsystems, are often duplicated. Discrete nature of MES does not allow small differences in data flow among redundant FCCs which are acceptable for continuous control algorithms. Therefore, mission state consistency has to be specifically maintained. We present a novel MES which includes FCC state synchronization. To achieve this result we developed the new concept of Asynchronous Behavior Tree with Memory (ABTM) and proposed a state synchronization algorithm. The implemented system was tested and proven to work in a real-time simulation of High Altitude Pseudo Satellite (HAPS) mission.

Lateral line system (LLS) is a sensory organ system which serves functions in varieties of flow-relative fish behaviors. Inspired by excellent performances of LLS in fish behaviors, multiple artificial lateral line systems (ALLSs) have been designed and applied to promote underwater robot technology. In this article, we focus on using ALLS for longitudinal separation sensing between two adjacent swimming robotic fish whose tails oscillate, and meanwhile the two fish are towed with a precisely controlled speed which equals to the rectilinear speed of freely-swimming robotic fish with the same oscillating parameters. Flow visualizations based on dye injection technique, hydrogen bubble technique, and computational fluid dynamics simulation are conducted to study the distribution characteristics of the vortices caused by the robotic fish. In addition, towing tank experiments are conducted for investigating the qualitative relationships between the longitudinal separations and the ALLS-measured HPVs of the two robotic fish. This work provide a guidance for the application of ALLS in relative states sensing of underwater robot group composed of two or more individuals, which has been rarely explored.

While monocular and stereo camera based motion estimation has reached a level of maturity that enables industrial use, the community keeps exploring novel multi-sensor solutions to meet the high robustness and accuracy requirements of certain applications such as autonomous vehicles. The present paper focuses on motion estimation with multi-perspective camera systems. In particular, we look into the intricate case in which the cameras are distributed over an articulated body, a scenario that occurs in truck motion estimation where additional cameras are installed on the trailer. The resulting articulated multi-perspective camera is analyzed in theory and practice, and we show that- by taking the non-holonomic constraints of the vehicle into account- a single point correspondence measured from the trailer is sufficient to render the additional unknown parameters given by the internal joint configuration before and after a relative displacement fully observable. optimizing over all parameters enhances the accuracy of the motion estimation of the entire system with respect to using the cameras on each rigid part alone. Results are confirmed on both simulated and real data.

Autonomous robots are already being used, for example, as tour guides, receptionists, or office-assistants. The proximity to humans and the possibility to physically interact with them highlights the importance of developing secure robot applications. It is crucial to consider security implications to be an important part of the robot application's development process. Adding security later in the application's life-cycle usually leads to high costs, or is not possible due to earlier design decisions. In this work, we present the Robot Application Security Process (RASP) as a lightweight process that enables the development of secure robot applications. Together with RASP we introduce the role of a Security Engineer (SecEng) as an important stakeholder in any robot application development process. RASP enables the SecEng to verify the completeness of his work and allows him to argue about the application's security with other stakeholders. Furthermore, we demonstrate how the RASP supports the SecEng and also other developers in their daily work.

Human has an ability to adjust utterance depending on the state of interlocutor. In this study, we explore the verbal behaviors of human through interaction with two smart speakers that have different level of task competence. We analyzed (1) linguistic behaviors appeared in user’s utterance, (2) length of the uttered speech, and (3) required pragmatics skills to understand the user’s intent. As a result, there were no significant difference in linguistic behaviors and length of the speech while user interacts with speakers with different task competence. In addition, various pragmatics elements were equally utilized and especially, implied intentions were frequently observed in user’s short utterance even under simple interaction scenarios.

The increasing use of autonomous mobile robots in different parts of society, and not restricted only to industrial environments, makes it important to propose techniques that will allow them to behave in the most socially acceptable way as possible. In most real-world scenarios, individuals in the environment are interacting with each other and are arranged into groups. Therefore, it is paramount the proposition of techniques to efficiently and correctly identify and represent such groups. This information can be useful in different tasks such as approaching and initiating an interaction, escorting, and the navigation itself. In this work, we propose a novel graph-based approach to evaluate the possible association of individuals in the environment based on their position and body orientation. Next, based on this association, we propose a representation of the combined social space of individuals in the same group. The methodology was evaluated using synthetic and real-world datasets, showing that it achieves results comparable to or better than the state-of-the-art.

We present a new way of constructing sparse roadmaps using point clouds that approximates and measures the underlying topology of the Cfree space. The main advantage of the constructed roadmap is its homotopy equivalence to the η-offset of the Cfree space. Though only used to plan paths as a regular roadmap in this work, because the roadmap preserves the topology of the underlying sampled space, the information can be used to plan paths beyond the simple connection of graph vertices. To construct the roadmap, we first sample the configuration space so that the resulting graph is a n-skeleton graph that constructs a Vietoris-Rips (VR) complex. Then, we perform a series of topological collapses to remove vertices from the graph while still preserving its topological properties. The resulting roadmaps are used to plan paths for different robots and the experimental results show that the proposed topological approach is faster and more feasible in complex high-dimensional spaces.

To maintain a high level of motivation is a vital issue during rehabilitation because rehabilitation involves much pain, depression, and discomfort. To study the mental state of patients is necessary. A new assistive approach using emotion evaluation combined with an assistive walking device to maintain motivation during the rehabilitation is proposed. To realize it, the walking-emotion relationship within personal walking limitation (PWL) was explored at first. The results showed that when people walked by following the personal preferred walking ratio (PPWR), they were evoked the arousal emotion and kept the current valence emotion simultaneously; however, when people were not following PPWR to walk, the negative emotion was elicited. After that, subject was required to wear the assistive walking device to conduct the test, which aims to observe the emotion variation when subject walked at specific walking gait with an assistive device. We then are according to current emotion state to tune device. The findings show if people mistakenly use the device, unpleasant feeling would be elicited. When we properly tuning assistive device conforms to current gait, it can validly alleviate the negative emotion further becoming the positive emotion. From these results, we conclude that emotion can be affected by walking. By using walking-emotion evaluation to an assistive device, we can be according to users' emotion to control device; also, we can use an assistive device to inspire users' emotion to further maintain motivation. Finally, this assistive approach provides a useful treatment way for serving rehabilitation.

Superior operation skills for hydraulic excavators are learned over time. Therefore, it is difficult to improve the skills of non-expert operators quickly, and their work needs to be supported. This paper presents a control system, which realizes smooth digging (similar to that performed by expert operators), based on the Center of Mass velocity of the attachment as an index. In addition, corresponding to the system switching caused by the digging reaction force, a controller gain tuning method that uses the Fictitious Exogenous Signal is proposed. The proposed method is applied to a hydraulic excavator, and its effect is verified.

In this paper, we investigate whether applying ankle torques during mid-stance can be a more effective way to reduce energetic cost of locomotion than actuating leg length alone. Ankles are useful in human gaits for many reasons including static balancing. In this work, we specifically avoid the heel-strike and toe-off benefits to investigate whether the progression of the center of pressure from heel-to-toe during mid-stance, or some other approach, is beneficial in and of itself. We use an Ankle Actuated Spring Loaded Inverted Pendulum model to simulate the shifting center of pressure dynamics, and trajectory optimization is applied to find limit cycles that minimize cost of transport. The results show that, for the vast majority of gaits, ankle torques do not affect cost of transport. Ankles reduce the cost of transport during a narrow band of gaits at the transition from grounded running to aerial running. This suggests that applying ankle torque during mid-stance of a steady gait is not a directly beneficial strategy, but is most likely a path between beneficial heel-strikes and toe-offs.

In recent years, ultra-wideband radio technology has become increasingly popular as a space-and cost-effective solution to the problem of indoor localization. This paper demonstrates how measurements of the channel impulse response can be used to estimate a signal's angle of arrival at a receiving antenna. This novel method requires no additional hardware, uses only a single antenna, and works with unsynchronized clocks and one-way communication. We evaluate our method on a real-dataset, and experimentally demonstrate how a mobile robot can localize itself by measuring angles to multiple ultra-wideband anchors.

Existing work to estimate the weight of a liquid poured into a target container often require predefined source weights or visual data. We present novel audio-based and audio-augmented techniques, in the form of multimodal convolutional neural networks (CNNs), to estimate poured weight, perform overflow detection, and classify liquid and target container. Our audio-based neural network uses the sound from a pouring sequence-a liquid being poured into a target container. Audio inputs consist of converting raw audio into mel-scaled spectrograms. Our audio-augmented network fuses this audio with its corresponding visual data based on video images. Only a microphone and camera are required, which can be found in any modern smartphone or Microsoft Kinect. Our approach improves classification accuracy for different environments, containers, and contents of the robot pouring task. Our Pouring Sequence Neural Networks (PSNN) are trained and tested using the Rethink Robotics Baxter Research Robot. To the best of our knowledge, this is the first use of audio-visual neural networks to analyze liquid pouring sequences by classifying their weight, liquid, and receiving container.

Behavior Trees (BTs) are becoming a popular tool to model the behaviors of autonomous agents in the computer game and the robotics industry. One of the key advantages of BTs lies in their composability, where complex behaviors can be built by composing simpler ones. The parallel composition is the one with the highest potential since the complexity of composing pre-existing behaviors in parallel is much lower than the one needed using classical control architectures as finite state machines. However, the parallel composition is rarely used due to the underlying concurrency problems that are similar to the ones faced in concurrent programming. In this paper, we define two synchronization techniques to tackle the concurrency problems in BTs compositions and we show how to exploit them to improve behavior predictability. Also, we introduce measures to assess execution performance and we show how design choices can affect them. To illustrate the proposed framework, we provide a set of experiments using the R1 robot and we gather statistically-significant data.

We present a concept of constrained collaborative mobile agents (CCMA) system, which consists of multiple wheeled mobile agents constrained by a passive kinematic chain. This mobile robotic system is modular in nature, the passive kinematic chain can be easily replaced with different designs and morphologies for different functions and task adaptability. Depending solely on the actuation of the mobile agents, this mobile robotic system can manipulate or position an end-effector. However, the complexity of the system due to presence of several mobile agents, passivity of the kinematic chain and the nature of the constrained collaborative manipulation requires development of an optimization framework. We therefore present an optimization framework for forward simulation and kinematic control of this system. With this optimization framework, the number of deployed mobile agents, actuation schemes, the design and morphology of the passive kinematic chain can be easily changed, which reinforces the modularity and collaborative aspects of the mobile robotic system. We present results, in simulation, for spatial 4-DOF to 6-DOF CCMA system examples. Finally, we present experimental quantitative results for two different fabricated 4-DOF prototypes, which demonstrate different actuation schemes, control and collaborative manipulation of an end-effector.