IROS 2019 Papers

WiFi technology has been used pervasively in fine-grained indoor localization, gesture recognition, and adaptive communication. Achieving better performance in these tasks generally boils down to differentiating Line-Of-Sight (LOS) from Non-Line-Of-Sight (NLOS) signal propagation reliably which generally requires expensive/specialized hardware due to the complex nature of indoor environments. Hence, the development of low-cost accurate positioning systems that exploit available infrastructure is not entirely solved. In this paper, we develop a framework for indoor localization and tracking of ubiquitous mobile devices such as smartphones using on-board sensors. We present a novel deep LOS/NLOS classifier which uses the Received Signal Strength Indicator (RSSI), and can classify the input signal with an accuracy of 85%. The proposed algorithm can globally localize and track a smartphone (or robot) with a priori unknown location, and with a semi-accurate prior map (error within 0.8m) of the WiFi Access Points (AP). Through simultaneously solving for the trajectory and the map of access points, we recover a trajectory of the device and corrected locations for the access points. Experimental evaluations of the framework show that localization accuracy is increased by using the trained deep network; furthermore, the system becomes robust to any error in the map of APs.

Synthesis of a feedback controller for nonlinear dynamical systems like a quadrotor requires to deal with the trade-off between performance and online computation requirement of the controller. Model predictive controllers (MPC) provide excellent control performance, but at the cost of high online computation. In this paper, we present our experience in approximating the behavior of an MPC for a quadrotor with a feed-forward neural network. To facilitate the collection of training data, we create a faithful model of the quadrotor and use Gazebo simulator to collect sufficient training data. The deep neural network (DNN) controller learned from the training data has been tested on various trajectories to compare its performance with a model-predictive controller. Our experimental results show that our DNN controller can provide almost similar trajectory tracking performance at a lower control computation cost, which helps in increasing the flight time of the quadrotor. Moreover, the hardware requirement for our DNN controller is significantly less than that for the MPC controller. Thus, the use of DNN based controller also helps in reducing the overall price of a quadrotor.

An essential feature for navigating socially with a mobile robot is the upper body orientation of persons in its vicinity. For example, in a supermarket orientation indicates whether a person is looking at goods on the shelves or where a person is likely to go. However, given limited computing and battery capabilities, it is not possible to rely on high-performance graphics cards to run large, computationally expensive deep neural networks for orientation estimation in real time. Nevertheless, deep learning performs quite well for regression problems. Therefore, we tackle the problem of upper body orientation estimation with small yet efficient deep neural networks on a mobile robot in this paper. We employ a fast person detection approach as preprocessing that outputs fixed size person images before the actual estimation of the orientation is done. The combination with lightweight networks allows us to estimate a continuous angle in real time, even using a CPU only. We experimentally evaluate the performance of our system on a new, self-recorded data set consisting of more than 100,000 RGB-D samples from 37 persons, which is made publicly available. We also do an extensive comparison of different network architectures and output encodings for their applicability in estimating orientations. Furthermore, we show that depth images are more suitable for the task of orientation estimation than RGB images or the combination of both.

Place recognition as one of the most significant requirements for long-term simultaneous localization and mapping (SLAM) has been developed rapidly in recent years. Also, deep learning is proved to be more capable than traditional methods to extract features under some complex environments. However, in real-world environments, there are many challenging problems such as viewpoint changes and illumination changes. The existing deep learning-based place recognition in extracting feature phases and matching process is both time-consuming. Moreover, features extracted from convolution neural network (CNN) are floating-point type with high dimension. In this paper, we propose deep supervised hashing for place recognition, where we design a similar hierarchy loss function to learn a model. The model can distinguish the similar images more accurately which is well suitable to place recognition. Besides the model can learn high quality hash codes by maximizing the likelihood of triplet labels. Experiments on several benchmark datasets for place recognition show that our approach is robust to viewpoints, illuminations and season changes with high accuracy. Furthermore, the trained model can extract features and match in real time on CPU with less memory consumption.

Cameras and 2D laser scanners, in combination, are able to provide low-cost, light-weight and accurate solutions, which make their fusion well-suited for many robot navigation tasks. However, correct data fusion depends on precise calibration of the rigid body transform between the sensors. In this paper we present the first framework that makes use of Convolutional Neural Networks (CNNs) for odometry estimation fusing 2D laser scanners and mono-cameras. The use of CNNs provides the tools to not only extract the features from the two sensors, but also to fuse and match them without needing a calibration between the sensors. We transform the odometry estimation into an ordinal classification problem in order to find accurate rotation and translation values between consecutive frames. Results on a real road dataset show that the fusion network runs in real-time and is able to improve the odometry estimation of a single sensor alone by learning how to fuse two different types of data information.

In this paper, a novel robotic grasping system is established to automatically pick up objects in cluttered scenes. A composite robotic hand composed of a suction cup and a gripper is designed for grasping the object stably. The suction cup is used for lifting the object from the clutter first and the gripper for grasping the object accordingly. We utilize the affordance map to provide pixel-wise lifting point candidates for the suction cup. To obtain a good affordance map, the active exploration mechanism is introduced to the system. An effective metric is designed to calculate the reward for the current affordance map, and a deep Q-Network (DQN) is employed to guide the robotic hand to actively explore the environment until the generated affordance map is suitable for grasping. Experimental results have demonstrated that the proposed robotic grasping system is able to greatly increase the success rate of the robotic grasping in cluttered scenes.

Autonomous driving is a challenging problem because the autonomous vehicle must understand complex and dynamic environment. This understanding consists of predicting future behavior of nearby vehicles and recognizing predefined rules. It is observed that not all rules have equivalent values, and the priority of the rules may change depending on the situation or the driver's driving style. In this work, we jointly reason both a future trajectories of vehicles and degree of satisfaction of each rule in the deep learning framework. Joint reasoning allows modeling interactions between vehicles, and leads to better prediction results. A rule is represented as a signal temporal logic (STL) formula, and a robustness slackness, a margin to the satisfaction of the rule, is predicted for the both autonomous and other vehicle, in addition to future trajectories. Learned robustness slackness decides which rule should be prioritized for the given situation for the autonomous vehicle, and filter out non-valid predicted trajectories for surrounding vehicles. The predicted information from the deep learning framework is used in model predictive control (MPC), which allows the autonomous vehicle navigate efficiently and safely. We test the feasibility of our approach in publicly available NGSIM datasets. Proposed method shows a driving style similar to the human one and considers the safety related to the rules through the future prediction of the surrounding vehicles.

This paper presents a novel metric inspection robot system using a deep neural network to detect and measure surface flaws (i.e., crack and spalling) on concrete structures performed by a wall-climbing robot. The system consists of four modules: robotics data collection module to obtain RGB-D images and IMU measurement, visual-inertial SLAM module to generate pose coupled key-frames with depth information, InspectionNet module to classify each pixel into three classes (back-ground, crack and spalling), and 3D registration and map fusion module to register the flaw patch into registered 3D model overlaid and highlighted with detected flaws for spatial-contextual visualization. The system enables the metric model of each surface flaw patch with pixel-level accuracy and determines its location in 3D space that is significant for structural health assessment and monitoring. The InspectionNet achieves an average accuracy of 87.64% for crack and spalling inspection. We also demonstrate our InspectionNet is robust to view angle, scale and illumination variation. Finally, we design a metric voxel volume map to highlight the flaw in 3D model and provide location and metric information.

Recently, a whole-body tactile sensing have emerged in robotics for safe human-robot interaction. A key issue in the whole-body tactile sensing is ensuring large-area manufacturability and high durability. To fulfill these requirements, a reconstruction method called electrical impedance tomography (EIT) was adopted in large-area tactile sensing. This method maps voltage measurements to conductivity distribution using only a few number of measurement electrodes. A common approach for the mapping is using a linearized model derived from the Maxwell's equation. This linearized model shows fast computation time and moderate robustness against measurement noise but reconstruction accuracy is limited. In this paper, we propose a novel nonlinear EIT algorithm through Deep Neural Network (DNN) approach to improve the reconstruction accuracy of EIT-based tactile sensors. The neural network architecture with rectified linear unit (ReLU) function ensured extremely low computational time (0.002 seconds) and nonlinear network structure which provides superior measurement accuracy. The DNN model was trained with dataset synthesized in simulation environment. To achieve the robustness against measurement noise, the training proceeded with additive Gaussian noise that estimated through actual measurement noise. For real sensor application, the trained DNN model was transferred to a conductive fabric-based soft tactile sensor. For validation, the reconstruction error and noise robustness were mainly compared using conventional linearized model and proposed approach in simulation environment. As a demonstration, the tactile sensor equipped with the trained DNN model is presented for a contact force estimation.

Corner cases are the main bottlenecks when applying Artificial Intelligence (AI) systems to safety-critical applications. An AI system should be intelligent enough to detect such situations so that system developers can prepare for subsequent planning. In this paper, we propose semi-supervised anomaly detection considering the imbalance of normal situations: In particular, driving data consists of multiple normal situations (e.g., right turn, going straight), some of which (e.g., U-turn) could be as rare as anomalous ones. Existing machine learning based anomaly detection approaches do not fare sufficiently well when applied to such imbalanced data. In this paper, we present a novel multi-task learning (LSTM autoencoder and predictor) based approach that leverages domain-knowledge (maneuver labels) for anomaly detection in driving data. We evaluate the proposed approach both quantitatively and qualitatively on 150 hours of real-world driving data and show improved performance over baseline/existing approaches.

Localization for mobile robots in dynamic, large-scale environments is a challenging task, especially when relying solely on odometry and 2D LIDAR data. When operating in fleets, mutual detection and the exchange of localization information can be highly valuable. Detecting and classifying different robot types in a heterogeneous fleet, however, is nontrivial with 2D LIDAR data due to the sparse observation information. In this paper a novel approach for mutual robot detection, classification and relative pose estimation based on a combination of convolutional and ConvLSTM layers is presented in order to solve this issue. The algorithm learns an end-to-end classification and pose estimation of robot shapes using 2D LIDAR information transformed into a grid-map. Subsequently a mixture model representing the probability distribution of the pose measurement for each robot type is extracted out of the heatmap output of the network. The output is then used in a cloud-based collaborative localization system in order to increase the localization of the individual robots. The effectiveness of our approach is demonstrated in both, simulation and real-world experiments. The results of our evaluation show that the classification network is able to achieve a precision of 90% on real-world data with an average position estimation error of 14 cm. Moreover, the collaborative localization system is able to increase the localization accuracy of a robot equipped with low-cost sensors by 63%.

Many robotic tasks require fast and accurate force sensing capabilities to ensure adaptive behavior execution. While dedicated force-torque (FT) sensors are a common option, such devices induce extra costs, need additional power supply, and add weight to otherwise light-weight robotic systems. This paper presents a machine learning approach for estimating external forces acting on a robot based on common internal sensors only. In the training phase, a behavior-specific proprioceptive model is learned as compact representation of the expected proprioceptive feedback during task execution. First, the proprioceptive sensors relevant for the given behavior are identified using information-theoretic measures. Then, the proprioceptive model is learned using deep learning techniques. During behavior execution, the proprioceptive model is applied to actual sensor readings for estimation of external forces. Experiments performed with the UR5 robot demonstrate the ability for fast and accurate force estimation even in situations where a dedicated commercial FT sensor is not applicable.

Applying Deep Learning to control has a lot of potential for enabling the intelligent design of robot control laws. Unfortunately common deep learning approaches to control, such as deep reinforcement learning, require an unrealistic amount of interaction with the real system, do not yield any performance guarantees, and do not make good use of extensive insights from control theory. In particular, common black-box approaches - that abandon all insight from control - are not suitable for complex robot systems. We propose a deep control approach as a bridge between the solid theoretical foundations of energy-based control and the flexibility of deep learning. To accomplish this goal, we extend Deep Lagrangian Networks (DeLaN) to not only adhere to Lagrangian Mechanics but also ensure conservation of energy and passivity of the learned representation. This novel extension is embedded within a energy control law to control under-actuated systems. The resulting DeLaN for energy control (DeLaN 4EC) is the first model learning approach using generic function approximation that is capable of learning energy control because existing approaches cannot learn the system energies directly. DeLaN 4EC exhibits excellent real-time control on the physical Furuta pendulum and learns to swing-up the pendulum while the control law using system identification does not.

The decision and planning system for autonomous driving in urban environments is hard to design. Most current methods manually design the driving policy, which can be expensive to develop and maintain at scale. Instead, with imitation learning we only need to collect data and the computer will learn and improve the driving policy automatically. However, existing imitation learning methods for autonomous driving are hardly performing well for complex urban scenarios. Moreover, the safety is not guaranteed when we use a deep neural network policy. In this paper, we proposed a framework to learn the driving policy in urban scenarios efficiently given offline connected driving data, with a safety controller incorporated to guarantee safety at test time. The experiments show that our method can achieve high performance in realistic simulations of urban driving scenarios.

Building models capable of generating structured output is a key challenge for AI and robotics. While generative models have been explored on many types of data, little work has been done on synthesizing lidar scans, which play a key role in robot mapping and localization. In this work, we show that one can adapt deep generative models for this task by unravelling lidar scans into a 2D point map. Our approach can generate high quality samples, while simultaneously learning a meaningful latent representation of the data. We demonstrate significant improvements against state-of-the-art point cloud generation methods. Furthermore, we propose a novel data representation that augments the 2D signal with absolute positional information. We show that this helps robustness to noisy and imputed input; the learned model can recover the underlying lidar scan from seemingly uninformative data.

One of the advancements in humanization of robots is its ability to recognize human emotions. Facial expression plays a key role in identifying human emotions relative to other cues. In this research, an intelligent network capable of real-time emotion recognition from multiple faces using deep learning technique is presented. The proposed network is based on Convolution Neural Network (CNN) in which three blocks of Convolution layers for feature extraction and two blocks of Dense layers for classification are used. The novelty of this method lies in recognizing emotions from multiple faces simultaneously in real time and its invariance to head pose, illumination and age factor. Most of reported work in literature for multiple faces is for frontal face without illumination variation. The proposed emotion recognition system is deployed on Raspberry Pi3 B+ for human robot interaction applications and achieved an average accuracy of 95.8% in real time.

In this paper, we present a novel approach to shared augmented reality (AR) for mobile devices operating in the same area that does not rely on cloud computing. In particular, each user's device processes the visual and inertial data received from its sensors and almost immediately broadcasts a partial feature map of its surroundings. In parallel, every device localizes against the broadcasted maps by employing feature observations to determine its 4-DOF transformation to each of the gravity-aligned maps. By doing so, virtual content placed by any of the users is quickly and accurately displayed in all other users' views. Furthermore, to reduce the effect of inconsistency introduced by relocalizing against incrementally created and updated maps, their transformations w.r.t. the device are modeled as random processes driven by white noise instead of constant, unknown parameters. Lastly, we assess the accuracy of our approach for the case of two users in a room-scale environment against VICON ground-truth.

Modular reconfigurable robots are touted for their flexibility, as their bodies can assume a wide range of shapes. A particular challenge is to make them move efficiently in 3D without compromising the scalability of the system. This paper proposes decentralized and fully reactive controllers for pose control of 3D modular reconfigurable robots. The robots operate in liquid environments, and move by routing fluid through themselves. Each module uses only two bits of sensory information per face. Additionally, the modules can use up to five bits of information that are exchanged via shared power lines. We prove that robots of convex shape are guaranteed to reach a goal object with a preferred orientation. Using computer simulations of Modular Hydraulic Propulsion robots, all controllers are assessed for different environments, system sizes and noise, and their performances compared against a centralized controller. Given the simplicity of the solutions, modules could be realized at scales below a millimeter-cube, where robots of high spatial resolution could perform accurate movements in 3D liquid environments.

An optional robotic arm unit, like animals' neck, allows quadruped robot to achieve multifunctional tasks such as manipulation and patrol. However, increases in body weight due to the neck unit will make energetic issues in legged robots' locomotion more serious. This study presents a minimal decentralized coordination mechanism between a neck and limbs, especially focusing on quadruped mammals' nodding behaviors for efficient walking. The results of two dimensional (2D) simulation suggest that bilateral sensory feedback mechanism between neck and limbs plays essential roles for the quadruped robot to coordinate neck and limbs in response to the physical situation of the body parts, achieving efficient quadrupedal walking.