IROS 2018 论文列表

We address the problem of audio-visual gaze control in the specific context of human-robot interaction, namely how controlled robot motions are combined with visual and acoustic observations in order to direct the robot head towards targets of interest. The paper has the following contributions: (i) a novel audio-visual fusion framework that is well suited for controlling the gaze of a robotic head; (ii) a reinforcement learning (RL) formulation for the gaze control problem, using a reward function based on the available temporal sequence of camera and microphone observations; and (iii) several deep architectures that allow to experiment with early and late fusion of audio and visual data. We introduce a simulated environment that enables us to learn the proposed deep RL model without the need of spending hours of tedious interaction. By thoroughly experimenting on a publicly available dataset and on a real robot, we provide empirical evidence that our method achieves state-of-the-art performance.

We propose a reinforcement learning approach for automatically building dry stacked (i.e. no mortar) structures with irregular objects. Stacking irregular objects is a challenging problem since each assembly action can be drawn from a continuous space of poses for an object, and several local geometric and physical considerations strongly affect the stability. To tackle this challenge, we concentrate on a simplified 2D version of the problem. We present a reinforcement learning algorithm based on deep Q-learning, where the learned Q-function, which maps state-action pairs into expected long-term rewards, is represented by a deep neural network. As the action space is continuous the Q-network is trained by sampling a finite number of actions that consider both geometric and physical constraints to approximate the target Q-values, Experiments show that the proposed method outperforms previous heuristics-based planning, leading to super construction with objects containing a significant amount of variations. We validate the generated stacking plans by executing them using a robot arm and manufactured, irregular objects.

We present a novel approach for interactive auditory object analysis with a humanoid robot. The robot elicits sensory information by physically shaking visually indistinguishable plastic capsules. It gathers the resulting audio signals from microphones that are embedded into the robotic ears. A neural network architecture learns from these signals to analyze properties of the contents of the containers. Specifically, we evaluate the material classification and weight prediction accuracy and demonstrate that the framework is fairly robust to acoustic real-world noise.

Reliable and accurate lane detection has been a long-standing problem in the field of autonomous driving. In recent years, many approaches have been developed that use images (or videos) as input and reason in image space. In this paper we argue that accurate image estimates do not translate to precise 3D lane boundaries, which are the input required by modern motion planning algorithms. To address this issue, we propose a novel deep neural network that takes advantage of both LiDAR and camera sensors and produces very accurate estimates directly in 3D space. We demonstrate the performance of our approach on both highways and in cities, and show very accurate estimates in complex scenarios such as heavy traffic (which produces occlusion), fork, merges and intersections.

This paper discusses deep learning for solving static and dynamic search and recovery tasks - such as the retrieval of all instances of actively moving targets - based on partial-view Unmanned Aerial Vehicle (UAV)-like sensing. In particular, we demonstrate that abstracted tactic and strategic explorational agency can be implemented effectively via a single deep network that optimises in unity: the mapping of sensory inputs and positional history towards navigational actions. We propose a dual-stream classification paradigm that integrates one Convolutional Neural Network (CNN) for sensory processing with a second one for interpreting an evolving longterm map memory. In order to learn effective search behaviours given agent location and agent-centric sensory inputs, we train this design against 400k+ optimal navigational decision samples from each set of static and dynamic evolutions for different multi-target behaviour classes. We quantify recovery performance across an extensive range of scenarios; including probabilistic placement and dynamics, as well as fully random target walks and herd-inspired behaviours. Detailed results comparisons show that our design can outperform naive, independent stream and off-the-shelf DRQN solutions. We conclude that the proposed dual-stream architecture can provide a unified, rationally motivated and effective architecture for solving online search tasks in dynamic, multi-target environments. With this paper we publish3 key source code and associated models.

A provably stable decentralized control scheme is proposed to allow multiple conventional quadcopters carry a cable-suspended payload. The method exploits a fundamental energetic passivity property of the combined drones, cables, and payload system to stably move the payload from its origin to destination. This is achieved without making any assumption about the status of the cables tension during the flight, and any measurement from the payload. The controller is decentralized in the sense that inter-drone communication of feedback measurements is not required. Motion stability is demonstrated via a Lyapunov analysis. The proposed controller is successfully implemented on a three-drone payload transport system in an indoor environment, using measurement from an optical tracking systems and the drones on-board IMUs.

Highly accurate and robust real-time localization is an essential technique for various autonomous driving applications. Numerous localization methods have been proposed that combine various types of sensors, including an environmental sensor, IMU and GPS. However, the usage of a single environmental sensor is rather fragile. Although the use of multi-environment sensors is a better alternative, fusion methods from previous studies have not adequately compensated for shortcomings in dissimilar sensors or have not considered errors in the pre-built map. In this paper, we propose a decentralized localization framework using heterogeneous map-matching sources. Decentralized localization performs two independent map-matchings and integrates them with a stochastic situational analysis model. By applying a stochastic model, the reliability of the two map matchings is collected and system stability is verified. A number of experiments with autonomous vehicles within the actual driving environment have shown that combining multiple map-matching sources ensures more robust results than the use of a single environmental sensor.

We present a decentralized and scalable approach for deployment of a robot swarm. Our approach tackles scenarios in which the swarm must reach multiple spatially distributed targets, and enforce the constraint that the robot network cannot be split. The basic idea behind our work is to construct a logical tree topology over the physical network formed by the robots. The logical tree acts as a backbone used by robots to enforce connectivity constraints. We study and compare two algorithms to form the logical tree: outwards and inwards. These algorithms differ in the order in which the robots join the tree: the outwards algorithm starts at the tree root and grows towards the targets, while the inwards algorithm proceeds in the opposite manner. Both algorithms perform periodic reconfiguration, to prevent suboptimal topologies from halting the growth of the tree. Our contributions are (i) The formulation of the two algorithms; (ii) A comparison of the algorithms in extensive physics-based simulations; (iii) A validation of our findings through real-robot experiments.

We consider a multi-robot variant of the mission monitoring problem. This problem arises in tasks where a robot observes the progress of another robot that is stochastically following a known trajectory, among other applications. We formulate and solve a variant where multiple tracker robots must monitor a single target robot, which is important because it enables the use of multi-robot systems to improve task performance in practice, such as in marine robotics missions. Our algorithm coordinates the behaviour of the trackers by computing optimal single-robot paths given a probabilistic representation of the other robots' paths. We employ a decentralised scheme that optimises over probability distributions of plans and has useful analytical properties. The planned trajectories collectively maximise the probability of observing the target throughout the mission with respect to probabilistic motion and observation models. We report simulation results for up to 8 robots that support our analysis and indicate that our algorithm is a feasible solution for improving the performance of mission monitoring systems.

Motion planning for fast locomotion of compliantly actuated robotic legs is generally considered to be a challenging issue, posing considerable real-time problems. This is at least the case if time-continuous trajectories need to be generated online. In this paper we take advantage of a simple controller structure, which reduces the motion planning to a discrete-time planning problem, in which only a small set of input parameters need to be determined for each step. We show that for a planar leg with serial elastic actuation, hopping on a ground with stairs of irregular length and height can be planned online, based on a parameter mapping which has been learned in a data-driven manner by performing hopping trials with an adaptive exploration algorithm to evenly sample the parameter space. Experiments on a planar hopping leg prototype validate the approach.

Simultaneous Localization and Mapping (SLAM) is considered to be a fundamental capability for intelligent mobile robots. Over the past decades, many impressed SLAM systems have been developed and achieved good performance under certain circumstances. However, some problems are still not well solved, for example, how to tackle the moving objects in the dynamic environments, how to make the robots truly understand the surroundings and accomplish advanced tasks. In this paper, a robust semantic visual SLAM towards dynamic environments named DS-SLAM is proposed. Five threads run in parallel in DS-SLAM: tracking, semantic segmentation, local mapping, loop closing and dense semantic map creation. DS-SLAM combines semantic segmentation network with moving consistency check method to reduce the impact of dynamic objects, and thus the localization accuracy is highly improved in dynamic environments. Meanwhile, a dense semantic octo-tree map is produced, which could be employed for high-level tasks. We conduct experiments both on TUM RGB-D dataset and in real-world environment. The results demonstrate the absolute trajectory accuracy in DS-SLAM can be improved one order of magnitude compared with ORB-SLAM2. It is one of the state-of-the-art SLAM systems in high-dynamic environments.

Wire detection, depth estimation and avoidance is one of the hardest challenges towards the ubiquitous presence of robust autonomous aerial vehicles. We present an approach and a system which tackles these three challenges along with generic obstacle avoidance as well. First, we perform monocular wire detection using a convolutional neural network under the semantic segmentation paradigm, and obtain a confidence map of wire pixels. Along with this, we also use a binocular stereo pair to detect other generic obstacles. We represent wires and generic obstacles using a disparity space representation and do a C-space expansion by using a non-linear sensor model we develop. Occupancy inference for collision checking is performed by maintaining a pose graph over multiple disparity images. For avoidance of wire and generic obstacles, we use a precomputed trajectory library, which is evaluated in an online fashion in accordance to a cost function over proximity to the goal. We follow this trajectory with a path tracking controller. Finally, we demonstrate the effectiveness of our proposed method in simulation for wire mapping, and on hardware by multiple runs for both wire and generic obstacle avoidance.

This paper introduces DREGON, a novel publicly-available dataset that aims at pushing research in sound source localization using a microphone array embedded in an unmanned aerial vehicle (UAV). The dataset contains both clean and noisy in-flight audio recordings continuously annotated with the 3D position of the target sound source using an accurate motion capture system. In addition, various signals of interests are available such as the rotational speed of individual rotors and inertial measurements at all time. Besides introducing the dataset, this paper sheds light on the specific properties, challenges and opportunities brought by the emerging task of UAV-embedded sound source localization. Several baseline methods are evaluated and compared on the dataset, with real-time applicability in mind. Very promising results are obtained for the localization of a broad-band source in loud noise conditions, while speech localization remains a challenge under extreme noise levels.

In this paper, a deep neural network (DNN) based integrated background noise suppression and acoustic modeling for speech recognition proposed in which on/off state of the motor for the head shaking robot is employed as the relevant auxiliary information of the DNN input. Since the motor sound being generated when the robot is moving or shaking its head severely degrades the performance of the speech recognition accuracy, we propose to use the motor on/off state as additional information when designing the DNN-based recognition system. Our speech recognition algorithm consists of two parts including the feature mapping model for feature enhancement and the acoustic model for phoneme recognition. As for the feature mapping, the stacked DNN is designed for the precise feature enhancement such that the lower DNN and upper DNN are trained separately and combined after which the motor state is plugged into both the lower DNN and upper DNN in addition to the input noisy speech. Then, the acoustic model is trained upon the feature enhancement model in which the motor state is again used as the augmented feature. The proposed technique to suppress the acoustic and motor noises was evaluated in term of the phoneme error rate (PER) and showed a significant improvement over the conventional system.

We develop a Deep Learning-based Wearable Vision-system with Vibrotactile-feedback (DLWV2)to guide Blind and Visually Impaired (BVI)people to reach objects. The system achieves high accuracy in object detection and tracking in 3-D using an extended deep learning-based 2.5-D detector and a 3-D object tracker with the ability to track 3-D object locations even outside the camera field-of-view. We train our detector with a large number of images with 2.5-D object ground-truth (i.e., 2-D object bounding boxes and distance from the camera to objects). A novel combination of HTC Vive Tracker with our system enables us to automatically obtain the ground-truth labels for training while requiring very little human effort to set up the system. Moreover, our system processes frames in real-time through a client-server computing platform such that BVI people can receive realtime vibrotactile guidance. We conduct a thorough user study on 12 BVI people in new environments with object instances which are unseen during training. Our system outperforms the non-assistive guiding strategy with statistic significance in both time and the number of contacting irrelevant objects. Finally, the interview with BVI users confirms that our system with distance-based vibrotactile feedback is mostly preferred, especially for objects requiring gentle manipulation such as a bottle with water inside.

Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images.

The way in which humans behave, speak and interact is deeply influenced by their culture. For example, greeting is done differently in France, in Sweden or in Japan; and the average interpersonal distance changes from one cultural group to the other. In order to successfully coexist with humans, robots should also adapt their behavior to the culture, customs and manners of the persons they interact with. In this paper, we deal with an important ingredient of cultural adaptation: how to generate robot plans that respect given cultural preferences, and how to execute them in a way that is sensitive to those preferences. We present initial results in this direction in the context of the CARESSES project, a joint EU-Japan effort to build culturally competent assistive robots.

Task autonomy is an important consideration for the development of future surgical robots. For robot-assisted anastomosis, suture thread detection is a prerequisite for subsequent robot manipulation. Previous works on automatic thread detection are focused on the learning of the models with specific surgical settings that are poorly generalisable to generic settings. In this paper, we propose a joint feature learning framework that caters for the foreground and background adaptation for surgical suture thread detection. The proposed method is developed in the context of semi-supervised and unsupervised domain adaptation, leveraging the labelled training data from the source domain to learn the detection model for unlabelled or partially labelled target domain, which can also be from different types of threads or organs. Based on adversarial learning, we further preserve the semantic identity and introduce curriculum adaptation to generate synthetic data. Experiments on four domain adaptation tasks for suture thread detection demonstrate the strength of the proposed method being able to generate good quality synthetic data and transfer between specific domains with limited or even no labelled data of the target domain.

In this paper, we introduce a coverage path planning algorithm with adaptive viewpoint sampling to construct accurate 3D models of complex large structures using Unmanned Aerial Vehicle (UAV). The developed algorithm, Adaptive Search Space Coverage Path Planner (ASSCPP), utilizes an existing 3D reference model of the complex structure and the onboard sensors' noise models to generate paths that are evaluated based on the traveling distance and the quality of the model. The algorithm generates a set of viewpoints by performing adaptive sampling that directs the search towards areas with low accuracy and low coverage. The algorithm predicts the coverage percentage obtained by following the generated coverage path using the reference model. A set of experiments were conducted in real and simulated environments with structures of different complexities to test the validity of the proposed algorithm.