IROS 2018 Papers

This paper presents an approach to learn and generalize robotic skills from a demonstration using deep reinforcement learning (deep RL). Dynamic Movement Primitives (DMPs) formulate a nonlinear differential equation and produce the observed movement from a demonstration. However, it is hard to generate new behaviors from using DMPs. Thus, we apply DMPs framework into deep RL as an initial setting for learning the robotic skills. First, we build a network to represent this differential equation, and learn and generalize the movements by optimizing the shape of DMPs with respect to the rewards up to the end of each sequence of movement primitives. In order to do this, we consider a deterministic actor-critic algorithm for deep RL and we also apply a hierarchical strategy. This drastically reduces the search space for a robot by decomposing the task, which allows to solve the sparse reward problem from a complex task. In order to integrate DMPs with hierarchical deep RL, the differential equation is considered as temporal abstraction of option. The overall structure is mainly composed of two controllers: meta-controller and sub-controller. The meta-controller learns a policy over intrinsic goals and a sub-controller learns a policy over actions to accomplish the given goals. We demonstrate our approach on a 6 degree-of-freedom (DOF) arm with a I-DOF gripper and evaluate our approach through a pick-and-place task.

Robust data association is necessary for virtually every SLAM system and finding corresponding points is typically a preprocessing step for scan alignment algorithms. Traditionally, handcrafted feature descriptors were used for these problems but recently learned descriptors have been shown to perform more robustly. In this work, we propose a local feature descriptor for 3D LiDAR scans. The descriptor is learned using a Convolutional Neural Network (CNN). Our proposed architecture consists of a Siamese network for learning a feature descriptor and a metric learning network for matching the descriptors. We also present a method for estimating local surface patches and obtaining ground-truth correspondences. In extensive experiments, we compare our learned feature descriptor with existing 3D local descriptors and report highly competitive results for multiple experiments in terms of matching accuracy and computation time.

Nonlinear optimal control problems are challenging to solve efficiently due to non-convexity. This paper introduces a trajectory optimization approach that achieves realtime performance by combining machine learning to predict optimal trajectories with refinement by quadratic optimization. First, a library of optimal trajectories is calculated offline and used to train a neural network. Online, the neural network predicts a trajectory for a novel initial state and cost function, and this prediction is further optimized by a sparse quadratic programming solver. We apply this approach to a fly-to-target movement problem for an indoor quadrotor. Experiments demonstrate that the technique calculates near-optimal trajectories in a few milliseconds, and generates agile movement that can be tracked more accurately than existing methods.

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end-effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors even amid challenging cases of tightly packed clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu/

A critical capability required for generally intelligent robot behavior is the ability to sequence motor skills to reach a goal. This requires a (typically abstract) representation that supports goal-directed planning, which raises the question of how to construct such a representation. Previous work has addressed this question in the context of simple black-box motor skills, which are insufficiently flexible to support the wide range of behavior required of intelligent robots. We now extend that work to include parametrized motor skills, where a robot must both select an action to execute and also decide how to parametrize it. We show how to construct a representation suitable for planning with parametrized motor skills, and specify conditions which are sufficient to separate the selection of motor skills from the parametrization of those skills. Our method results in a simple discrete abstract representation for planning followed by a parameter selection process that operates on a fixed plan. We first demonstrate learning this representation in a virtual domain based on Angry Birds and then learn an abstract symbolic representation for a robot manipulation task.

In this paper, we propose a novel architecture and a self-supervised policy gradient algorithm, which employs unsupervised auxiliary tasks to enable a mobile robot to learn how to navigate to a given goal. The dependency on the global information is eliminated by providing only sparse range-finder measurements to the robot. The partially observable planning problem is addressed by splitting it into a hierarchical process. We use convolutional networks to plan locally, and a differentiable memory to provide information about past time steps in the trajectory. These modules, combined in our network architecture, produce globally consistent plans. The sparse reward problem is mitigated by our modified policy gradient algorithm. We model the robots uncertainty with unsupervised tasks to force exploration. The novel architecture we propose with the modified version of the policy gradient algorithm allows our robot to reach the goal in a sample efficient manner, which is orders of magnitude faster than the current state of the art policy gradient algorithm. Simulation and experimental results are provided to validate the proposed approach.

Given the description of an object, s physical attributes, humans can determine a proper strategy and grasp an object. This paper proposes an approach to determine grasping strategy for an anthropomorphic robotic hand simply based on natural-language descriptions of an object. A learning-based approach is proposed to help a robotic hand learn suitable grasp poses starting from the natural language description of the object. Object features are parsed from natural-language descriptions by using a customized natural-language processing technique. The most likely grasp type for the given object is learned from the human grasping taxonomy based on the parsed features. The grasping strategy generated by the proposed approach is evaluated both by simulation study and execution of the grasps on an AR10 robotic hand.

This paper presents a methodology to design mechanosensor feedback to oscillator-based controller for worm-like soft-bodied robots. A reinforcement learning technique, i.e., PEPG, is employed to embed appropriate mechanosensor feedback to harness global entrainment among the controller, the body dynamics, and the environment without explicitly designing the interaction between the oscillators. Another reinforcement learning, actor-critic, was applied to train the controller for the simulation models to analyze the effectiveness of PEPG in the system. Furthermore, the gait controller was trained under different body dynamics, i.e., the physical model of a caterpillar and an earthworm. We found that PEPG is suitable for the system probably because it does not add exploration noise to actions and it conducts episode based parameter updates. The simulation results show the proposed method can acquire distinct behavior, i.e., caterpillars' crawling, inching and earthworms' crawling, under different body dynamics. The outcome implies, that by utilizing appropriate learning method, desired functionality can be achieved in soft-bodied robots without explicitly designing their behavior.

This paper introduces a fully deep learning approach to monocular SLAM, which can perform simultaneous localization using a neural network for learning visual odometry (L-VO) and dense 3D mapping. Dense 2D flow and a depth image are generated from monocular images by sub-networks, which are then used by a 3D flow associated layer in the L-VO network to generate dense 3D flow. Given this 3D flow, the dual-stream L-VO network can then predict the 6DOF relative pose and furthermore reconstruct the vehicle trajectory. In order to learn the correlation between motion directions, the Bivariate Gaussian modeling is employed in the loss function. The L-VO network achieves an overall performance of 2.68 % for average translational error and 0.0143°/m for average rotational error on the KITTI odometry benchmark. Moreover, the learned depth is leveraged to generate a dense 3D map. As a result, an entire visual SLAM system, that is, learning monocular odometry combined with dense 3D mapping, is achieved.

Sampling-based motion planners have experienced much success due to their ability to efficiently and evenly explore the state space. However, for many tasks, it may be more efficient to not uniformly explore the state space, especially when there is prior information about its structure. Previous methods have attempted to modify the sampling distribution using hand selected heuristics that can work well for specific environments but not universally. In this paper, a policy-search based method is presented as an adaptive way to learn implicit sampling distributions for different environments. It utilizes information from past searches in similar environments to generate better distributions in novel environments, thus reducing overall computational cost. Our method can be incorporated with a variety of sampling-based planners to improve performance. Our approach is validated on a number of tasks, including a 7DOF robot arm, showing marked improvement in number of collision checks as well as number of nodes expanded compared with baseline methods.

Millirobots are a promising robotic platform for many applications due to their small size and low manufacturing costs. Legged millirobots, in particular, can provide increased mobility in complex environments and improved scaling of obstacles. However, controlling these small, highly dynamic, and underactuated legged systems is difficult. Hand-engineered controllers can sometimes control these legged millirobots, but they have difficulties with dynamic maneuvers and complex terrains. We present an approach for controlling a real-world legged millirobot that is based on learned neural network models. Using less than 17 minutes of data, our method can learn a predictive model of the robot's dynamics that can enable effective gaits to be synthesized on the fly for following user-specified waypoints on a given terrain. Furthermore, by leveraging expressive, high-capacity neural network models, our approach allows for these predictions to be directly conditioned on camera images, endowing the robot with the ability to predict how different terrains might affect its dynamics. This enables sample-efficient and effective learning for locomotion of a dynamic legged millirobot on various terrains, including gravel, turf, carpet, and styrofoam. Videos and further details can be found at https://sites.google.com/view/imageconddyn.

Humans and mobile robots will be increasingly cohabiting in the same environments, which has lead to an increase in studies on human robot interaction (HRI). One important topic in these studies is the development of robot navigation algorithms that are socially compliant to humans navigating in the same space. In this paper, we present a method to learn human navigation behaviors using maximum entropy deep inverse reinforcement learning (MEDIRL). We use a large open dataset of pedestrian trajectories collected in an uncontrolled environment as the expert demonstrations. Human navigation behaviors are captured by a nonlinear reward function through deep neural network (DNN) approximation. The developed MEDIRL algorithm takes feature inputs including social affinity map (SAM) that are extracted from human motion trajectories. We perform simulation experiments using the learned reward function, and the performance is evaluated comparing it with the real measured pedestrian trajectories in the dataset. The evaluation results show that the proposed method has acceptable prediction accuracy compared to other state-of-the-art methods, and it can generate pedestrian trajectories similar to real human trajectories with natural social navigation behaviors such as collision avoidance, leader-follower, and split-and-rejoin.

The hardware compatibility of legged locomotion is often illustrated by Zero Moment Point (ZMP) that has been extensively studied for decades. One of the most popular models for computing the ZMP is the linear inverted pendulum (LIP) model that expresses ZMP as a linear function of the center of mass(COM) and its acceleration. In the real world, however, it may not accurately predict the true ZMP of hardware due to various reasons such as unmodeled dynamics and differences between simulation model and hardware. In this paper, we aim to improve the theoretical ZMP model by learning the real hardware dynamics from experimental data. We first optimize the motion plan using the theoretical ZMP model and collect COP data by executing the motion on a force plate. We then train a new ZMP model that maps the motion plan variable to the actual ZMP and use the learned model for finding a new hardware-compatible motion plan. Through various locomotion tasks of a quadruped, we demonstrate that motions planned for the learned ZMP model are compatible on hardware when those for the theoretical ZMP model are not. Furthermore, experiments using ZMP models with different complexities reveal that overly complex models may suffer from over-fitting even though they can potentially represent more complex, unmodeled dynamics.

Learning from Demonstration (LfD) is a popular approach to endowing robots with skills without having to program them by hand. Typically, LfD relies on human demonstrations in clutter-free environments. This prevents the demonstrations from being affected by irrelevant objects, whose influence can obfuscate the true intention of the human or the constraints of the desired skill. However, it is unrealistic to assume that the robot's environment can always be restructured to remove clutter when capturing human demonstrations. To contend with this problem, we develop an importance weighted batch and incremental skill learning approach, building on a recent inference-based technique for skill representation and reproduction. Our approach reduces unwanted environmental influences on the learned skill, while still capturing the salient human behavior. We provide both batch and incremental versions of our approach and validate our algorithms on a 7-DOF JACO2 manipulator with reaching and placing skills.

When learning forward and inverse kinematics maps of manipulators, usually little attention is paid to data-efficiency, i.e., the accuracy gained per action-outcome sample. This paper examines properties of popular (online) learning techniques and demonstrates that - regardless of the employed exploration strategy - the structure of kinematics mappings does not allow for a practically viable trade-off between the number of samples and the resulting approximation error for manipulators with more than a few DoFs - unless tailored parametric models are employed. We discuss suitable choices for these parametric models for both rigid and elastic discretely-actuated robots and compare their data -efficiency to that of popular exploratory learning approaches relying on non-parametric models. Our theoretical considerations are confirmed by various experimental results for inverse kinematics mappings of rigid and omnielastic manipulators.

High-fidelity computational human models provide a safe and cost-efficient method for studying driver experience in vehicle maneuvers and for validation of vehicle design. Compared to passive human models, active human models capable of reproducing the decision-making, as well as vehicle maneuver motion planning and control, will be able to support realistic simulation of human-vehicle interaction. In this paper, we propose an integrated human-vehicle interaction simulation framework which learns vehicle maneuver motion primitives from human drivers, and uses them to compose natural and contextual driving motions. Specifically, we recruited six experienced drivers and recorded their vehicle maneuver motions on a fixed-base driving simulation testbed. We further segmented and classified the collected data based on their similarity in joint coordination. Using a combination of imitation learning methods, we extracted the regularity and variability of vehicle maneuver motions across subjects, and learned the dynamic motion primitives to be used for motion reproduction in simulation. We present an implementation of the framework on lower-extremity joint coordination in pedal activation for longitudinal vehicle control. Our research efforts lead to a motion primitive library which enables planning natural driver motions, and will be integrated with the driving decision-making, motion control, and vehicle dynamics in the proposed framework for simulating human-vehicle interaction.

In this work we explore a new approach for robots to teach themselves about the world simply by observing it. In particular we investigate the effectiveness of learning task-agnostic representations for continuous control tasks. We extend Time-Contrastive Networks (TCN) that learn from visual observations by embedding multiple frames jointly in the embedding space as opposed to a single frame. We show that by doing so, we are now able to encode both position and velocity attributes significantly more accurately. We test the usefulness of this self-supervised approach in a reinforcement learning setting. We show that the representations learned by agents observing themselves take random actions, or other agents perform tasks successfully, can enable the learning of continuous control policies using algorithms like Proximal Policy Optimization (PPO) using only the learned embeddings as input. We also demonstrate significant improvements on the real-world Pouring dataset with a relative error reduction of 39.4% for motion attributes and 11.1% for static attributes compared to the single-frame baseline. Video results are available at https://sites.google.com/view/actionablerepresentations.

Hand gestures are a natural component of human-human communication. Simple hand gestures are intuitive and can exhibit great lexical variety. It stands to reason that such a user input mechanism can have many benefits, including seamless interaction, intuitive control and robustness to physical constraints and ambient electrical, light and sound interference. However, while semantic and logical information encoded via hand gestures is readily decoded by humans, leveraging this communication channel in human-machine interfaces remains a challenge. Recent data-driven deep learning approaches are promising towards uncovering abstract and complex relationships that manual and direct rule-based classification schemes fail to discover. Such an approach is amenable towards hand gesture recognition, but requires myriad data which can be collected physically via user experiments. This process, however, is onerous and tedious. A streamlined approach with less overhead is sought. To that end, this work presents a novel method of synthetic hand gesture dataset generation that leverages modern gaming engines. Furthermore, preliminary results indicate that the dataset, despite being synthetic and requiring no physical data collection, is both accurate and rich enough to train a real-world hand gesture classifier that operates in real-time.

We propose a lightweight and ground-optimized lidar odometry and mapping method, LeGO-LOAM, for realtime six degree-of-freedom pose estimation with ground vehicles. LeGO-LOAM is lightweight, as it can achieve realtime pose estimation on a low-power embedded system. LeGO-LOAM is ground-optimized, as it leverages the presence of a ground plane in its segmentation and optimization steps. We first apply point cloud segmentation to filter out noise, and feature extraction to obtain distinctive planar and edge features. A two-step Levenberg-Marquardt optimization method then uses the planar and edge features to solve different components of the six degree-of-freedom transformation across consecutive scans. We compare the performance of LeGO-LOAM with a state-of-the-art method, LOAM, using datasets gathered from variable-terrain environments with ground vehicles, and show that LeGO-LOAM achieves similar or better accuracy with reduced computational expense. We also integrate LeGO-LOAM into a SLAM framework to eliminate the pose estimation error caused by drift, which is tested using the KITTI dataset.