RSS 2021 Papers

Cooperatively avoiding collision is a critical functionality for robots navigating in dense human crowds; failure of which could lead to either overaggressive or overcautious behavior. A necessary condition for cooperative collision avoidance is to couple the prediction of the agents' trajectories with the planning of the robot's trajectory. However; it is unclear that trajectory based cooperative collision avoidance captures the correct agent attributes. In this work we migrate from trajectory based coupling to a formalism that couples agent preference distributions. In particular; we show that preference distributions (probability density functions representing agents' intentions) can capture higher order statistics of agent behaviors; such as willingness to cooperate. Thus; coupling in distribution space exploits more information about inter-agent cooperation than coupling in trajectory space. We thus introduce a general objective for coupled prediction and planning in distribution space; and propose an iterative best response optimization method based on variational analysis with guaranteed sufficient decrease. Based on this analysis; we develop a sampling-based motion planning framework called DistNav that runs in real time on a laptop CPU. We evaluate our approach on challenging scenarios from both real world datasets and simulation environments; and benchmark against a wide variety of model based and machine learning based approaches. The safety and efficiency statistics of our approach outperform all other models. Finally; we find that DistNav is competitive with human safety and efficiency performance.

Robots deployed in human-populated spaces often need human help to effectively complete their tasks. Yet; a robot that asks for help too frequently or at the wrong times may cause annoyance; and a robot that asks too infrequently may be unable to complete its tasks. In this paper; we present a model of humans' helpfulness towards a robot in an office environment; learnt from online user study data. Our key insight is that effectively planning for a task that involves bystander help requires disaggregating individual and contextual factors and explicitly reasoning about uncertainty over individual factors. Our model incorporates the individual factor of latent helpfulness and the contextual factors of human busyness and robot frequency of asking. We integrate the model into a Bayes-Adaptive Markov Decision Process (BAMDP) framework and run a user study that compares it to baseline models that do not incorporate individual or contextual factors. The results show that our model significantly outperforms baseline models by a factor of 1.5X; and it does so by asking for help more effectively: asking 1.2X times less while still receiving more human help on average.

Imitation learning is an effective tool for robotic learning tasks where specifying a reinforcement learning (RL) reward is not feasible or where the exploration problem is particularly difficult. Imitation; typically behavior cloning or inverse RL; derive a policy from a collection of first-person action-state trajectories. This is contrary to how humans and other animals imitate: we observe a behavior; even from other species; understand its perceived effect on the state of the environment; and figure out what actions our body can perform to reach a similar outcome. In this work; we explore the possibility of third-person visual imitation of manipulation trajectories; only from vision and without access to actions; demonstrated by embodiments different to the ones of our imitating agent. Specifically; we investigate what would be an appropriate representation method with which an RL agent can visually track trajectories of complex manipulation behavior —non-planar with multiple-object interactions— demonstrated by experts with different embodiments. We present a way to train manipulator-independent representations (MIR) that primarily focus on the change in the environment and have all the characteristics that make them suitable for cross-embodiment visual imitation with RL: domain-invariant; temporally smooth; and actionable. We show that with our proposed method our agents are able to imitate; with complex robot control; trajectories from a variety of embodiments and with significant visual and dynamics differences; e.g. simulation-to-reality gap.

In this paper; we propose a novel task; Manipulation Question Answering (MQA); where the robot performs manipulation actions to change the environment in order to answer a given question. To solve this problem; a framework consisting of a QA module and a manipulation module is proposed. For the QA module; we adopt the method for the Visual Question Answering (VQA) task. For the manipulation module; a Deep Q Network (DQN) model is designed to generate manipulation actions for the robot to interact with the environment. We consider the situation where the robot continuously manipulating objects inside a bin until the answer to the question is found. Besides; a novel dataset that contains a variety of object models; scenarios and corresponding question-answer pairs is established in a simulation environment. Extensive experiments have been conducted to validate the effectiveness of the proposed framework.

The partially observable Markov decision process (POMDP) is a principled general framework for robot decision making under uncertainty; but POMDP planning suffers from high computational complexity; when long-term planning is required. While temporally-extended macro-actions help to cut down the effective planning horizon and significantly improve computational efficiency; how do we acquire good macro-actions? This paper proposes Macro-Action Generator-Critic (MAGIC); which performs offline learning of macro-actions optimized for online POMDP planning. Specifically; MAGIC learns a macro-action generator end-to-end; using an online planner's performance as the feedback. During online planning; the generator generates on the fly situation-aware macro-actions conditioned on the robot's belief and the environment context. We evaluated MAGIC on several long-horizon planning tasks both in simulation and on a real robot. The experimental results show that the learned macro-actions offer significant benefits in online planning performance; compared with primitive actions and handcrafted macro-actions.

Deep learning has had a far reaching impact in robotics. Specifically; deep reinforcement learning algorithms have been highly effective in synthesizing neural-network controllers for a wide range of tasks. However; despite this empirical success; these controllers still lack theoretical guarantees on their performance; such as Lyapunov stability (i.e.; all trajectories of the closed-loop system are guaranteed to converge to a goal state under the control policy). This is in stark contrast to traditional model-based controller design; where principled approaches (like LQR) can synthesize stable controllers with provable guarantees. To address this gap; we propose a generic method to synthesize a Lyapunov-stable neural-network controller; together with a neural-network Lyapunov function to simultaneously certify its stability. Our approach formulates the Lyapunov condition verification as a mixed-integer linear program (MIP). Our MIP verifier either certifies the Lyapunov condition; or generates counter examples that can help improve the candidate controller and the Lyapunov function. We also present an optimization program to compute an inner approximation of the region of attraction for the closed-loop system. We apply our approach to robots including an inverted pendulum; a 2D and a 3D quadrotor; and showcase that our neural-network controller outperforms a baseline LQR controller. The code is open sourced at

For robots to work alongside humans and perform in unstructured environments; they must learn new motion skills and adapt them to unseen situations on the fly. This demands learning models that capture relevant motion patterns; while offering enough flexibility to adapt the encoded skills to new requirements; such as dynamic obstacle avoidance. We introduce a Riemannian manifold perspective on this problem; and propose to learn a Riemannian manifold from human demonstrations on which geodesics are natural motion skills. We realize this with a variational autoencoder (VAE) over the space of position and orientations of the robot end-effector. Geodesic motion skills let a robot plan movements from and to arbitrary points on the data manifold. They also provide a straightforward method to avoid obstacles by redefining the ambient metric in an online fashion.Moreover; geodesics naturally exploit the manifold resulting from multiple-solution settings to design motions that were not demonstrated previously. We test our learning framework usinga7-DoF robotic manipulator; where the robot satisfactorily learns and reproduces realistic skills featuring elaborated motion patterns; avoids previously–unseen obstacles; and generates novel movements in multiple-solution settings.

We present a learning-based approach to prove infeasibility of kinematic motion planning problems. Sampling-based motion planners are effective in high-dimensional spaces but are only probabilistically complete. Consequently; these planners cannot provide a definite answer if no plan exists; which is important for high-level scenarios; such as task-motion planning. We propose a combination of bidirectional sampling-based planning (such as RRT-connect) and machine learning to construct an infeasibility proof alongside the two search trees. An infeasibility proof is a closed manifold in the obstacle region of the configuration space that separates the start and goal into disconnected components of the free configuration space. We train the manifold using common machine learning techniques and then triangulate the manifold into a polytope to prove containment in the obstacle region. Under assumptions about learning hyper-parameters and robustness of configuration space optimization; the output is either an infeasibility proof or a motion plan. We demonstrate proof construction for 3-DOF and 4-DOF manipulators and show improvement over previous algorithms.

Robots operating in everyday environments need to effectively perceive; model; and infer semantic properties of objects. Existing knowledge reasoning frameworks only model binary relations between an object's class label and its semantic properties; unable to collectively reason about object properties detected by different perception algorithms and grounded in diverse sensory modalities. We bridge the gap between multimodal perception and knowledge reasoning by introducing an n-ary representation that models complex; inter-related object properties. To tackle the problem of collecting n-ary semantic knowledge at scale; we propose a transformer neural network that directly generalizes knowledge from observations of object instances. The learned model can reason at different levels of abstraction; effectively predicting unknown properties of objects in different environmental contexts given different amounts of observed information. We quantitatively validate our approach against five prior methods on LINK; a unique dataset we contribute that contains 1457 situated object instances with 15 multimodal properties types and 200 total properties. Compared to the prior state of the art Markov Logic Network; our model obtains a 10% improvement in predicting unknown properties of novel object instances while reducing training and inference time by 150 times. Additionally; we apply our work to a mobile manipulation robot; demonstrating its ability to leverage n-ary reasoning to retrieve objects and actively detect object properties.

We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward; which is necessary for reinforcement learning; planning; or knowing when to ask for help. For a general-purpose robot operating in the real world; this reward function must also be able to generalize broadly across environments; tasks; and objects; while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language; collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast; “in-the-wild” videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work; we propose a simple approach; Domain-agnostic Video Discriminator (DVD); that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task; and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets; this reward function (a) can generalize zero shot to unseen environments; (b) generalize zero shot to unseen tasks; and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.

This paper describes a system for visually guided autonomous navigation of under-canopy farm robots. Low-cost under-canopy robots can drive between crop rows under the plant canopy and accomplish tasks that are infeasible for over-the-canopy drones or larger agricultural equipment. However; autonomously navigating them under the canopy presents a number of challenges: unreliable GPS and LiDAR; high cost of sensing; challenging farm terrain; clutter due to leaves and weeds; and large variability in appearance over the season and across crop types. We address these challenges by building a modular system that leverages machine learning for robust and generalizable perception from monocular RGB images from low-cost cameras; and model predictive control for accurate control in challenging terrain. Our system; CropFollow; is able to autonomously drive 485 meters per intervention on average; outperforming a state-of-the-art LiDAR based system (286 meters per intervention) in extensive field testing spanning over 25 km.

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot. Prior work in imitation learning typically requires each task be specified with a task id or goal image—something that is often impractical in open-world environments. On the other hand; previous approaches in instruction following allow agent behavior to be guided by language; but typically assume structure in the observations; actuators; or language that limit their applicability to complex settings like robotics. In this work; we present a method for incorporating free-form natural language conditioning into imitation learning. Our approach learns perception from pixels; natural language understanding; and multitask continuous control end-to-end as a single neural network. Unlike prior work in imitation learning; our method is able to incorporate unlabeled and unstructured demonstration data (i.e. no task or language labels). We show this dramatically improves language conditioned performance; while reducing the cost of language annotation to less than 1% of total data. At test time; a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment; specified only with natural language descriptions of each task (e.g. "open the drawer...now pick up the block...now press the green button...") (see video). To scale up the number of instructions an agent can follow; we propose combining text conditioned policies with large pretrained neural language models. We find this allows a policy to be robust to many out-of-distribution synonym instructions; without requiring new demonstrations. See videos of a human typing live text commands to our agent at

We present Ruckig; an algorithm for Online Trajectory Generation (OTG) respecting third-order constraints and complete kinematic target states. Given any initial state of a system with multiple Degrees of Freedom (DoFs); Ruckig calculates a time-optimal trajectory to an arbitrary target state defined by its position; velocity; and acceleration limited by velocity; acceleration; and jerk constraints. The proposed algorithm and implementation allows three contributions: (1) To the best of our knowledge; we derive the first time-optimal OTG algorithm for arbitrary; multi-dimensional target states; in particular including non-zero target acceleration. (2) This is the first open-source prototype of time-optimal OTG with limited jerk and complete time synchronization for multiple DoFs. (3) Ruckig allows for directional velocity and acceleration limits; enabling robots to better use their dynamical resources. We evaluate the robustness and real-time capability of the proposed algorithm on a test suite with over 1 000 000 000 random trajectories as well as in real-world applications.

Robots and autonomous systems must interact with one another and their environment to provide high-quality services to their users. Dynamic game theory provides an expressive theoretical framework for modeling scenarios involving multiple agents with differing objectives interacting over time. A core challenge when formulating a dynamic game is designing objectives for each agent that capture desired behavior. In this paper; we propose a method for inferring parametric objective models of multiple agents based on observed interactions. Our inverse game solver jointly optimizes player objectives and continuous-state estimates by coupling them through Nash equilibrium constraints. Hence; our method is able to directly maximize the observation likelihood rather than other non-probabilistic surrogate criteria. Our method does not require full observations of game states or player strategies to identify player objectives. Instead; it robustly recovers this information from noisy; partial state observations. As a byproduct of estimating player objectives; our method computes a Nash equilibrium trajectory corresponding to those objectives. Thus; it is suitable for downstream trajectory forecasting tasks. We demonstrate our method in several simulated traffic scenarios. Results show that it reliably estimates player objectives from a short sequence of noise-corrupted partial state observations. Furthermore; using the estimated objectives; our method makes accurate predictions of each player's trajectory.

Loops are pervasive in robotics problems; appearing in mapping and localization; where one is interested in finding loop closure constraints to better approximate robot poses or other estimated quantities; as well as planning and prediction; where one is interested in the homotopy classes of the space through which a robot is moving. We generalize the standard topological definition of a loop to cases where a trajectory passes close to itself; but doesn't necessarily touch; giving a definition that is more practical for real robotics problems. This relaxation leads to new and useful properties of inexact loops; such as their ability to be partitioned into topologically connected sets closely matching the concept of a "loop closure"; and the existence of simple and nonsimple loops. Building from these ideas; we introduce several ways to measure properties and quantities of inexact loops on a trajectory; such as the trajectory's "loop area" and "loop density'"; and use them to compare strategies for sampling representative inexact loops to build constraints in mapping and localization problems.

This paper presents INVIGORATE; a robot system that interacts with humans through natural language and grasps a specified object in clutter. The objects may occlude; obstruct; or even stack on top of one another. INVIGORATE embodies several challenges: (i) infer the target object among other occluding objects; from input language expressions and RGB images; (ii) infer object blocking relationships (OBRs) from the images; and (iii) synthesize a multi-step plan to ask questions that disambiguate the target object and to grasp it successfully. We train separate neural networks for object detection; for visual grounding; for question generation; and for OBR detection and grasping. They allow for unrestricted object categories and language expressions; subject to the training datasets. However; errors in visual perception and ambiguity in human languages are inevitable and negatively impact the robot’s performance. To overcome these uncertainties; we build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules. Through approximate POMDP planning; the robot tracks the history of observations and asks disambiguation questions in order to achieve a near-optimal sequence of actions that identify and grasp the target object. INVIGORATE combines the benefits of model-based POMDP planning and data-driven deep learning. Preliminary experiments with INVIGORATE on a Fetch robot show significant benefits of this integrated approach to object grasping in clutter with natural language interactions. A demonstration video is available online:

We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input. The family of nonlinear dynamical system-based methods have successfully demonstrated dynamic robot behaviors but have difficulty in generalizing to unseen configurations as well as learning from image inputs. Recent works approach this issue by using deep network policies and reparameterize actions to embed the structure of dynamical systems but still struggle in domains with diverse configurations of image goals; and hence; find it difficult to generalize. In this paper; we address this dichotomy by leveraging embedding the structure of dynamical systems in a hierarchical deep policy learning framework; called Hierarchical Neural Dynamical Policies (H-NDPs). Instead of fitting deep dynamical systems to diverse data directly; H-NDPs form a curriculum by learning local dynamical system-based policies on small regions in state-space and then distill them into a global dynamical system-based policy that operates only from high-dimensional images. H-NDPs additionally provide smooth trajectories; a strong safety benefit in the real world. We perform extensive experiments on dynamic tasks both in the real world (digit writing; scooping; and pouring) and simulation (catching; throwing; picking). We show that H-NDPs are easily integrated with both imitation as well as reinforcement learning setups and achieve state-of-the-art results. Video results at

Accurate models of robot dynamics are critical for safe and stable control and generalization to novel operational conditions. Hand-designed models; however; may be insufficiently accurate; even after careful parameter tuning. This motivates the use of machine learning techniques to approximate the robot dynamics over a training set of state-control trajectories. The dynamics of many robots; including ground; aerial; and underwater vehicles; are described in terms of their SE(3) pose and generalized velocity; and satisfy conservation of energy principles. This paper proposes a Hamiltonian formulation over the SE(3) manifold of the structure of a neural ordinary differential equation (ODE) network to approximate the dynamics of a rigid body. In contrast to a black-box ODE network; our formulation guarantees total energy conservation by construction. We develop energy shaping and damping injection control for the learned; potentially under-actuated SE(3) Hamiltonian dynamics to enable a unified approach for stabilization and trajectory tracking with various platforms; including pendulum; rigid-body; and quadrotor systems.

In this work we present a planning and control method for a quadrotor in an autonomous drone race. Our method combines the advantages of both model-based optimal control and model-free deep reinforcement learning. We consider a single drone racing on a track marked by a series of gates; through which it must maneuver in minimum time. Firstly we solve the discretized Hamilton-Jacobi-Bellman (HJB) equation to produce a closed-loop policy for a simplified; reduced order model of the drone. Next; we train a deep network policy in a supervised fashion to mimic the HJB policy. Finally; we further train this network using policy gradient reinforcement learning on the full drone dynamics model with a low-level feedback controller in the loop. This gives a deep network policy for controlling the drone to pass through a single gate. In a race course; this policy is applied successively to each new oncoming gate to guide the drone through the course. The resulting policy completes a high-fidelity AirSim drone race with 12 gates in 34.89s (on average); outracing a model-based HJB policy by 33.20s; a supervised learning policy by 1.24s; and a trajectory planning policy by 12.99s; while a model-free RL policy was never able to complete the race.