IROS 2020 Papers

In this paper, we consider a problem of foothold selection for the quadrupedal robots equipped with compliant adaptive feet. Starting from a model of the foot we compute the quality of the potential footholds considering also kinematic constraints and collisions during evaluation. Since terrain assessment and constraints checking are computationally expensive we applied a Convolutional Neural Network (CNN) to evaluate the potential footholds on the elevation map. We propose an efficient strategy for data clustering and segmentation with CNN. The data for training the neural network is collected off-line but the inference works on-line when the robot walks on rough terrains and allows for efficient adaptation to the terrain and exploitation of the properties of the soft adaptive feet.

We present a new measure, CMetric, to classify driver behaviors using centrality functions. Our formulation combines concepts from computational graph theory and social traffic psychology to quantify and classify the behavior of human drivers. CMetric is used to compute the probability of a vehicle executing a driving style, as well as the intensity used to execute the style. Our approach is designed for realtime autonomous driving applications, where the trajectory of each vehicle or road-agent is extracted from a video. We compute a dynamic geometric graph (DGG) based on the positions and proximity of the road-agents and centrality functions corresponding to closeness and degree. These functions are used to compute the CMetric based on style likelihood and style intensity estimates. Our approach is general and makes no assumption about traffic density, heterogeneity, or how driving behaviors change over time. We present an algorithm to compute CMetric and demonstrate its performance on real-world traffic datasets. To test the accuracy of CMetric, we introduce a new evaluation protocol (called "Time Deviation Error") that measures the difference between human prediction and the prediction made by CMetric.

There have been significant advances in neural networks for both 3D object detection using LiDAR and 2D object detection using video. However, it has been surprisingly difficult to train networks to effectively use both modalities in a way that demonstrates gain over single-modality networks. In this paper, we propose a novel Camera-LiDAR Object Candidates (CLOCs) fusion network. CLOCs fusion provides a low-complexity multi-modal fusion framework that significantly improves the performance of single-modality detectors. CLOCs operates on the combined output candidates before Non-Maximum Suppression (NMS) of any 2D and any 3D detector, and is trained to leverage their geometric and semantic consistencies to produce more accurate final 3D and 2D detection results. Our experimental evaluation on the challenging KITTI object detection benchmark, including 3D and bird's eye view metrics, shows significant improvements, especially at long distance, over the state-of-the-art fusion based methods. At time of submission, CLOCs ranks the highest among all the fusion-based methods in the official KITTI leaderboard. We will release our code upon acceptance.

Learning accurate models of the physical world is required for a lot of robotic manipulation tasks. However, during manipulation, robots are expected to interact with un-known workpieces so that building predictive models which can generalize over a number of these objects is highly desirable. In this paper, we study the problem of designing deep learning agents which can generalize their models of the physical world by building context-aware learning models. The purpose of these agents is to quickly adapt and/or generalize their notion of physics of interaction in the real world based on certain features about the interacting objects that provide different contexts to the predictive models. With this motivation, we present context-aware zero shot learning (CAZSL, pronounced as casual) models, an approach utilizing a Siamese network architecture, embedding space masking and regularization based on context variables which allows us to learn a model that can generalize to different parameters or features of the interacting objects. We test our proposed learning algorithm on the recently released Omnipush datatset that allows testing of meta-learning capabilities using low-dimensional data. Codes for CAZSL are available at https://www.merl.com/research/license/CAZSL.

We propose Action-Oriented Semantic Maps (AOSMs), a representation that enables a robot to acquire object manipulation behaviors and semantic information about the environment from a human teacher with a Mixed Reality Head-Mounted Display (MR-HMD). AOSMs are a representation that captures both: a) high-level object manipulation actions in an object class's local frame, and b) semantic representations of objects in the robot's global map that are grounded for navigation. Humans can use a MR-HMD to teach the agent the information necessary for planning object manipulation and navigation actions by interacting with virtual 3D meshes overlaid on the physical workspace. We demonstrate that our system enables users to quickly and accurately teach a robot the knowledge required to autonomously plan and execute three household tasks: picking up a bottle and throwing it in the trash, closing a sink faucet, and flipping a light switch off.

As a strategy to address the difficulties encountered when modeling and controlling a musculoskeletal system, we present a straightforward implementation of an autonomous decentralized motion control system in this paper; the system is inspired by the spinal reflex system of animals. We developed an artificial receptor, a muscle, and a neuron to mechanically implement the reflex mechanisms of animals. Among the reflex mechanisms, this paper presents a reflex system with reciprocal innervation for a musculoskeletal quasi-quadruped robot, including antagonist muscles. In the experiments, the robot autonomously generated a leg trajectory and a gait pattern with smooth alternating motions of the antagonist muscles through the interaction between the body, the ground, and the artificial reflex systems. To evaluate the reciprocal innervation, we compared the developed robot with one that does not include antagonist muscles. The reciprocal innervation allows for twice as many muscle implementations as those offered by the robot without antagonist muscles. Moreover, it improves the running speed by 5% on average and the flexion and extension velocities of all joints by 28% on average at around touchdowns and liftoffs of the foot. This successful result lead to implement more advanced nervous systems by solely mechanical devices.

Multi-robot path planning (MRPP) is a task of planning collision free paths for a group of robots in a graph. Each robot starts in its individual starting vertex and its task is to reach a given goal vertex. Existing techniques for solving MRPP optimally under various objectives include search-based and compilation-based approaches. Often however finding an optimal solution is too difficult hence sub-optimal algorithms that trade-off the quality of solutions and the runtime have been devised. We suggest eSMT-CBS, a new bounded sub-optimal algorithm built on top of recent compilation-based method for optimal MRPP based on satisfiability modulo theories (SMT). We compare eSMT-CBS with ECBS, a major representative of bounded sub-optimal search-based algorithms. The experimental evaluation shows significant advantage of eSMT-CBS across variety of scenarios.

Although picking up objects a few centimeters in size is a common task, achieving such ability in a robot manipulator remains challenging. We take a step toward solving this problem by focusing on the task of picking a 1.0-cm screw from a bulk bin using only tactile information to achieve the task. Inspired by how humans pick up small objects from a bin, we propose a "grasp-separate" strategy for robotic picking, which involves grasping many objects first and then separating a single object through manipulation in the fingers, for robotic picking. Based on this strategy, we developed a tactile-based screw bin-picking system. We trained a convolution neural network to estimate the number of screws in the fingers first and built a controller that generates manipulation behaviors to separate a screw using reinforcement learning. To compensate for the low resolution of off-the-shelf tactile sensor arrays, we adopted active sensing, which uses observations obtained during a predefined simple movement. We show that this approach enhances the estimation accuracy and manipulation performance. Furthermore, to enable flexible finger motion, such as between the thumb and the index finger in a human hand, we propose a soft robot finger structure that leverages compliant materials. A soft actor-critic algorithm successfully found dexterous screw separation behaviors in compliant soft robotic fingers. In the evaluation, the system obtained an average success rate of 80%, which was difficult to achieve without the grasp-separate manipulation technique.

In this research, we have taken a biomimetic approach to the control of musculoskeletal humanoids. A controller was designed based on the motor directional tuning phenomenon seen in the motor cortex of primates. Despite the simple implementation of the control scheme, complex coordinated movements such as reaching for target objects with its upper body was achieved, and is demonstrated in the accompanying video. The controller does not require an internal model, and instead constantly observes its body in relation to the external world to update motor commands. We claim that such an embodied approach to the control of musculoskeletal robots will be able to effectively take advantage of their complex bodies to achieve motion.

Designing a swarm of robots to address different tasks and adapt to various environments through self-assembly is one of the most challenging topics in the field of robotics research. Here, we present an assembly robotic system with inchworm robots as modules. The system is called BioARS (Bionic Assembly Robotic System). It can either work as a swarm of individual untethered inchworm robots or be assembled into a quadruped robot. The inchworm robots are connected by magnets using a "shoulder-to-shoulder" connecting method, which helps strengthen the magnetic connection. Central pattern generators are used to control the trot gait of the quadruped robot. Our experiments demonstrate that the bionic assembly system is adaptive in that it can pass through confined spaces in the form of inchworms and walk on rough terrain in the form of a quadruped robot. The proposed BioARS, therefore, combines the flexibility of inchworms and the adaptability of quadruped animals, which is promising for application in planetary exploration, earthquake search and rescue operations. We also provide several examples of directions for future research regarding our system.

Landing upside down on a ceiling is challenging as it requires a flier to invert its body and land against the gravity, a process that demands a stringent spatiotemporal coordination of body translational and rotational motion. Although such an aerobatic feat is routinely performed by biological fliers such as flies, it is not yet achieved in aerial robots using onboard sensors. This work describes the development of a bio-inspired inverted landing strategy using computationally efficient Relative Retinal Expansion Velocity (RREV) as a visual cue. This landing strategy consists of a sequence of two motions, i.e. an upward acceleration and a rapid angular maneuver. A policy search algorithm is applied to optimize the landing strategy and improve its robustness by learning the transition timing between the two motions and the magnitude of the target body angular velocity. Simulation results show that the aerial robot is able to achieve robust inverted landing, and it tends to exploit its maximal maneuverability. In addition to the computational aspects of the landing strategy, the robustness of landing is also significantly dependent on the mechanical design of the landing gear, the upward velocity at the start of body rotation, and timing of rotor shutdown.

The ability to measure multi-axis contact forces and contact surface normals in real time is critical to allow robots to improve their dexterous manipulation and locomotion abilities. This paper presents a new fingertip sensor for 3-axis contact force and contact location detection, as well as improvements on an existing footpad sensor through use of a new artificial neural network estimator. The fingertip sensor is intended for use in manipulation, while the footpad sensor is intended for high force use in locomotion. Both sensors consist of pressure sensing elements embedded within a rubber hemisphere, and utilize an artificial neural network to estimate the applied forces (fx, fy, and fz), and contact angles (θ and φ) from the individual sensor element readings. The sensors are inherently robust, and the hemispherical shape allows for easy integration into point feet and fingertips. Both the fingertip and footpad sensors demonstrate the ability to track forces and angles accurately over the surface of the hemisphere (θ=±45° and φ=±45°) and can experience up to 25N and 450N normal force, respectively, without saturating. The performance of the sensor is demonstrated with experimental results of dynamic control of a robotic arm with real-time sensor feedback.

Many modern simultaneous localization and mapping (SLAM) techniques rely on sparse landmark-based maps due to their real-time performance. However, these techniques frequently assert that these landmarks are fixed in position over time, known as the static-world assumption. This is rarely, if ever, the case in most real-world environments. Even worse, over long deployments, robots are bound to observe traditionally static landmarks change, for example when an autonomous vehicle encounters a construction zone. This work addresses this challenge, accounting for changes in complex three-dimensional environments with the creation of a probabilistic filter that operates on the features that give rise to landmarks. To accomplish this, landmarks are clustered into cliques and a filter is developed to estimate their persistence jointly among observations of the landmarks in a clique. This filter uses estimated spatial-temporal priors of geometric objects, allowing for dynamic and semi-static objects to be removed from a formally static map. The proposed algorithm is validated in a 3D simulated environment.

Traffic simulators are important tools in autonomous driving development. While continuous progress has been made to provide developers more options for modeling various traffic participants, tuning these models to increase their behavioral diversity while maintaining quality is often very challenging. This paper introduces an easily-tunable policy generation algorithm for autonomous driving agents. The proposed algorithm balances diversity and driving skills by leveraging the representation and exploration abilities of deep reinforcement learning via a distinct policy set selector. Moreover, we present an algorithm utilizing intrinsic rewards to widen behavioral differences in the training. To provide quantitative assessments, we develop two trajectory-based evaluation metrics which measure the differences among policies and behavioral coverage. We experimentally show the effectiveness of our methods on several challenging intersection scenes.

We consider the problem of designing synthetic cells to achieve a complex goal (e.g., mimicking the immune system by seeking invaders) in a complex environment (e.g., the circulatory system), where they might have to change their control policy, communicate with each other, and deal with stochasticity including false positives and negatives-all with minimal capabilities and only a few bits of memory. We simulate the immune response in cyclic, maze-like environments and use targets at unknown locations to represent invading cells. Using only a few bits of memory, the synthetic cells are programmed to perform a physically-feasible algorithm with which they update their control policy based on randomized encounters with other cells. As the synthetic cells work together to find the target, their interactions as an ensemble function as a physical implementation of a Bayesian update. That is, the particles act as a particle filter. This result provides formal properties about the behavior of the synthetic cell ensemble that can be used to ensure robustness and safety. This method of self-organization is evaluated in simulations, and applied to an actual model of the human circulatory system.

This paper presents a method for target localization and tracking in clutter using Bayesian fusion of vision and Radio Frequency (RF) sensors used aboard a small Unmanned Aircraft System (sUAS). Sensor fusion is used to ensure tracking robustness and reliability in case of camera occlusion or RF signal interference. Camera data is processed using an off-the-shelf algorithm that detects possible objects of interest in a given image frame, and the true RF emitting target needs to be identified from among these if it is present. These data sources, as well as the unknown motion of the target, lead to a heavily non-linear non-Gaussian target state uncertainties, which are not amenable to typical data association methods for tracking. A probabilistic model is thus first rigorously developed to relate conditional dependencies between target movements, RF data and visual object detections. A modified particle filter is then developed to simultaneously reason over target states and RF emitter association hypothesis labels for visual object detections. Truth model simulations are presented to compare and validate the effectiveness of the RF + visual data fusion filter.

To improve the accuracy of the grasping detection, this paper proposes a novel detector with batch normalization masked evaluation model. It is designed with a two-layer sparse autoencoder, and a Batch Normalization based mask is incorporated into the second layer of the model to effectively reduce the features with weak correlation. The extracted features from such model are more distinctive, which guarantees the higher accuracy of the grasping detection. Extensive experiments show that the proposed evaluation model outperforms the state-of- the-art, and the recognition accuracy can reach 95.51% for robotic grasping detection.

This paper describes basic implementation of an embedded controller board based on a hybrid architecture equipped with an Intel FPGA SoC and an NVIDIA GPU SoC. Embedded distributed network involving motor-drivers or other embedded boards is constructed with low-latency optical transmission link. The central controller for high-level motion planning is connected via Gigabit Ethernet. The controller board with the hybrid architecture provides lower-latency feedback control performance. Computing performance of the FPGA SoC, the GPU SoC, and the central controller is evaluated by computation time of matrix multiplication. Then, the total feedback latency is estimated to show the performance of the hybrid architecture.

We present our second generation tactile sensor for the Shadow Dexterous Hand's palm. We were able to significantly improve the tactile sensor characteristics by utilizing our latest barometer-based tactile sensing technology with linear (R2 ≥ 0.9996) sensor output and no noticeable hysteresis. The sensitivity threshold of the tactile cells and the spatial density were both dramatically increased. We demonstrate the benefits of the new sensor by re-running an experiment to estimate the stiffness of different objects that we originally used to test our first generation palm sensor. The results underline a considerable performance boost in estimation accuracy, just due to the improved tactile skin. We also propose a revised neural network architecture that even further improves the average classification accuracy to 96% in a 5-fold cross-validation.