RSS 2021 论文列表

We address the problem of calculating complex Jacobian matrices that can arise from optimization problems. An example is the inverse optimal control in human motion analysis which has a cost function that depends on the second order time-derivative of torque ̈τ. Thus; its gradient decomposed to; among other; the Jacobian δ ̈τ/δq. We propose a new concept called N-order Comprehensive Motion Transformation Matrix (N-CMTM) to provide an exact analytical solution of several Jacobians. The computational complexity of the basic Jacobian and its N-order time-derivatives computed from the N-CMTM is experimentally shown to be linear to the number of joints Nj. The N-CMTM is based on well-known spatial algebra which makes it available for any type of robots. Moreover; it can be used along classical algorithms. The computational complexity of the construction of the N-CMTM itself is experimentally shown to be N².

Mapping and localization using surface features is prone to failure due to environment changes such as inclement weather. Recently; Localizing Ground Penetrating Radar (LGPR) has been proposed as an alternative means of localizing using underground features that are stable over time and less affected by surface conditions. However; due to the lack of commercially available LGPR sensors; the wider research community has been largely unable to replicate this work or build new and innovative solutions. We present GROUNDED an open dataset of LGPR scans collected in a variety of environments and weather conditions. By labeling this data with ground truth localization from an RTK-GPS / Inertial Navigation System; and carefully calibrating and time synchronizing the radar scans with ground truth positions; camera imagery; and Lidar data; we enable researchers to build novel localization solutions that are resilient to changing surface conditions. We include 108 individual runs totalling 450 km of driving with LGPR; GPS; Odometry; Camera; and Lidar measurements. We also present two new evaluation benchmarks for 1) Localizing in Weather and 2) Multi-lane Mapping; to enable comparisons of future work supported by the dataset. The dataset can be accessed at

Tool use requires reasoning about the fit between an object’s affordances and the demands of a task. Visual affordance learning can benefit from goal-directed interaction experience; but current techniques rely on human labels or expert demonstrations to generate this data. In this paper; we describe a method that grounds affordances in physical interactions instead; thus removing the need for human labels or expert policies. We use an efficient sampling-based method to generate successful trajectories that provide contact data; which are then used to reveal affordance representations. Our framework; GIFT; operates in two phases: first; we discover visual affordances from goal-directed interaction with a set of procedurally generated tools; second; we train a model to predict new instances of the discovered affordances on novel tools in a self-supervised fashion. In our experiments; we show that GIFT can leverage a sparse keypoint representation to predict grasp and interaction points to accommodate multiple tasks; such as hooking; reaching; and hammering. GIFT outperforms baselines on all tasks and matches a human oracle on two of three tasks using novel tools.

Accurate system modeling and identification gain importance as tasks executed by autonomously acting unmanned aerial vehicles (UAVs) get more complex and demanding. This paper presents a Bayesian filter approach to online and continuously identify the system parameters; sensor suite calibration states; and vehicle navigation states in a holistic framework. Previous work only tackles subsets of the overall state vector during dedicated phases (e.g.; motionless; online during flight; post-processing). These works often introduce the artificial so-called body frame forcing assumptions on system states; such as the inertia matrix’s principal axes orientation. Our approach estimates the entire state vector in the (usually not precisely known) center of mass; eliminating several assumptions caused by the artificially introduced body frame in other work. Since our approach also estimates geometric states such as the rotor and sensor placements; no hand-made measures to the unknown center of mass are required – the system is fully self-calibrating. A detailed discussion on the system’s observability reveals additionally required (different) measurements for a theoretical and a real N-arm multicopter. We show that easy and precise hand-measurable quantities in real applications can provide the required information. Statistically relevant simulations in Gazebo/RotorS providing ground truth for all states yet having realistic physics validate all our findings.

In this paper; we first prove an interesting result for point feature based SLAM.

Visual place recognition is the task of finding same places in a set of database images for a given set of query images. This task becomes particularly challenging if the environmental condition changes between database and query; for example from day to night. In this paper; we build upon our recent work on graph optimization for place recognition; where a graph was used to model additional structural knowledge like sequences. A subsequent non-linear least squares optimization (NLSQ) improved the place recognition performance. While this approach achieves very high performance; it is quite slow and memory inefficient. This paper addresses the long runtime and the high memory usage in order to obtain the same or better place recognition performance faster on larger problems. We propose a novel graph optimization procedure that is based on Iterated Conditional Modes (ICM). In addition; we investigate a new cost function for an edge in the graph. Our novel ICM-based approach achieves 9.1msec maximum runtime per query; which is 260x faster than the minimum runtime with NLSQ. Moreover; with ICM we can optimize problems that are not feasible with NLSQ on a full graph due to memory limitations. To demonstrate the superior performance of our ICM-based method; we provide extensive experimental evaluations with the essence of 987 precision-recall curves: Our proposed ICM-based method is compared to the NLSQ-based method as well as to six sequence-based approaches from the literature on 21 sequence combinations from five datasets with four different image descriptors. Our experiments show that our ICM-based method with sequence-exploitation not only improves the NLSQ-based performance by 10% on average while being 385x faster and using more than 60x less memory. It also significantly outperforms all six sequence-based methods from the literature by at least 32% on average with the NetVLAD descriptor while using comparable runtime and memory. Code is available online.

We present a fast and feature-complete differentiable physics engine; Nimble (

This paper explores a system for assembling structures by dropping block components into place. During and after assembly; the blocks are held together by geometric interlock; so that fasteners or mortar are only needed to bind the final block to one of its neighbors. Drop assembly is a promising strategy for assembly by swimming or flying robots; as it may allow structures to be built without requiring close contact with the existing structure. The current paper explores a mathematical model of interlock; and presents a particular block design that allows interlock to be achieved using only gravity. Proof-of-concept demonstrations of the system are presented using a low-cost and relatively low-precision robot arm. The paper finally analyzes some of the potential limitations of the approach; particularly including flexing of the structure due to manufacturing tolerance limitations.

Simulation provides a safe and efficient way to generate useful data for learning complex robotic tasks. However; matching simulation and real-world dynamics can be quite challenging; especially for systems that have a large number of unobserved or unmeasurable parameters; which may lie in the robot dynamics itself or in the environment with which the robot interacts. We introduce a novel approach to tackle such a sim-to-real problem by developing policies capable of adapting to new environments; in a zero-shot manner. Key to our approach is an error-aware policy (EAP) that is explicitly made aware of the effect of unobservable factors during training. An EAP takes as input the predicted future state error in the target environment; which is provided by an error-prediction function; simultaneously trained with the EAP. We validate our approach on an assistive walking device trained to help the human user recover from external pushes. We show that a trained EAP for a hip-torque assistive device can be transferred to different human agents with unseen biomechanical characteristics. In addition; we show that our method can be applied to other standard RL control tasks.

High level declarative constraints provide a powerful (and popular) way to define and construct control policies; however; most synthesis algorithms do not support specifying the degree of randomness (unpredictability) of the resulting controller. In many contexts; e.g.; patrolling; testing; behavior prediction; and planning on idealized models; predictable or biased controllers are undesirable. To address these concerns; we introduce the Entropic Reactive Control Improvisation (ERCI) framework and algorithm that supports synthesizing control policies for stochastic games that are declaratively specified by (i) a hard constraint specifying what must occur (ii) a soft constraint specifying what typically occurs; and (iii) a randomization constraint specifying the unpredictability and variety of the controller; as quantified using causal entropy. This framework; which extends the state-of-the-art by supporting arbitrary combinations of adversarial and probabilistic uncertainty in the environment; enables a flexible modeling formalism which we argue; theoretically and empirically; remains tractable.

The rapid rise of accessibility of unmanned aerial vehicles or drones pose a threat to general security and confidentiality. Most of the commercially available or custom-built drones are multi-rotors and are comprised of multiple propellers. Since these propellers rotate at a high-speed; they are generally the fastest moving parts of an image and cannot be directly "seen'' by a classical camera without severe motion blur. We utilize a class of sensors that are particularly suitable for such scenarios called event cameras; which have a high temporal resolution; low-latency; and high dynamic range. In this paper; we model the geometry of a propeller and use it to generate simulated events which are used to train a deep neural network called EVPropNet to detect propellers from the data of an event camera. EVPropNet directly transfers to the real world without any fine-tuning or retraining. We present two applications of our network: (a) tracking and following an unmarked drone and (b) landing on a near-hover drone. We successfully evaluate and demonstrate the proposed approach in many real-world experiments with different propeller shapes and sizes. Our network can detect propellers at a rate of 85.1% even when 60% of the propeller is occluded and can run at upto 35Hz on a 2W power budget. To our knowledge; this is the first deep learning-based solution for detecting propellers (to detect drones). Finally; our applications also show an impressive success rate of 92% and 90% for the tracking and landing tasks respectively.

Model predictive control (MPC) schemes have a proven track record for delivering aggressive and robust performance in many challenging control tasks; coping with nonlinear system dynamics; constraints; and observational noise. Despite their success; these methods often rely on simple control distributions; which can limit their performance in highly uncertain and complex environments. MPC frameworks must be able to accommodate changing distributions over system parameters; based on the most recent measurements. In this paper; we devise an implicit variational inference algorithm able to estimate distributions over model parameters and control inputs on-the-fly. The method incorporates Stein Variational gradient descent to approximate the target distributions as a collection of particles; and performs updates based on a Bayesian formulation. This enables the approximation of complex multi-modal posterior distributions; typically occurring in challenging and realistic robot navigation tasks. We demonstrate our approach on both simulated and real-world experiments requiring real-time execution in the face of dynamically changing environments.

This paper presents a first low-cost autonomous robotic system for underwater assembly of mortarless structures. The long-term goal is to enable the construction of large-scale underwater structures; such as retaining walls and artificial reefs. The approach follows the principle of co-design; the 2-DOF manipulator and blocks are designed to complement the localization and control strategies. The blocks and gripper are designed with a connector geometry that removes error during pickup of blocks and drop assembly. This error correction feature allows a simplification of localization and control; which are based on fiducial markers on custom platforms. We developed the proposed system on a low-cost heavily modified BlueROV2 autonomous vehicle — which we call Droplet — with a two-degree of freedom hand that can open and close a gripper and rotate over the yaw. We performed extensive experiments in the pool to evaluate each component and the system as a whole. Results showed a 100% success rate in dropping blocks in the presence of some localization and control errors and the assembly of several different 3D structures composed of up to eight blocks.

In this paper; we address the problem of steering a team of agents under stochastic linear dynamics to prescribed final state means and covariances. The agents operate in a common environment where inter-agent constraints may also be present. In order for our method to be scalable to large-scale systems and computationally efficient; we approach the problem in a distributed control framework using the Alternating Direction Method of Multipliers (ADMM). Each agent solves its own covariance steering problem in parallel; while additional copy variables for its closest neighbors are introduced to ensure that the inter-agent constraints will be satisfied. The inclusion of these additional variables creates a requirement for consensus between original and copy variables that involve the same agent. For this reason; we employ a variation of ADMM for consensus optimization. Simulation results on multi-vehicle systems under uncertainty with collision avoidance constraints illustrate the effectiveness of our algorithm. The substantially improved scalability of our distributed approach with respect to the number of agents is also demonstrated; in comparison with an equivalent centralized scheme.

The learning efficiency of an intelligent agent can be greatly improved by utilizing a useful set of skills. However; the design of robot skills can often be intractable in real-world applications due to the prohibitive amount of effort and expertise that it requires. In this work; we introduce Skill Learning In Diversified Environments (SLIDE); a method to discover generalizable skills via automated generation of a diverse set of tasks. As opposed to prior work on unsupervised discovery of skills which incentivizes the skills to produce different outcomes in the same environment; our method pairs each skill with a unique task produced by a trainable task generator. To encourage generalizable skills to emerge; our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks. A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective. The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks. We demonstrate that the proposed method can effectively learn a variety of robot skills in two tabletop manipulation domains. Our results suggest that the learned skills can effectively improve the robot's performance in various unseen target tasks compared to existing reinforcement learning and skill learning methods.

The locomotion for many modern robotic systems is optimized for a single target domain - aerial; surface or underwater. In this work; we address the challenge of developing a robotic system capable of controlled motion in air and underwater. Further; we explore the particular challenge of dynamic transitions between air and water. We propose Dipper; an aerial-aquatic hybrid vehicle. Dipper is a light-weight fixed-wing UAV with actively swept wings. The bio-inspired system is not only capable of maneuvering efficiently during flight and underwater; but can also perform dynamic aerial-aquatic transitions. We describe the design; construction and testing of the Dipper prototype; and demonstrate repeatability and robustness especially during the transition phases.

Robotic cutting of soft materials is critical for applications such as food processing; household automation; and surgical manipulation. As in other areas of robotics; simulators can facilitate controller verification; policy learning; and dataset generation. Moreover; differentiable simulators can enable gradient-based optimization; which is invaluable for calibrating simulation parameters and optimizing controllers. In this work; we present DiSECt: the first differentiable simulator for cutting soft materials. The simulator augments the finite element method (FEM) with a continuous contact model based on signed distance fields (SDF); as well as a continuous damage model that inserts springs on opposite sides of the cutting plane and allows them to weaken until zero stiffness; enabling crack formation. Through various experiments; we evaluate the performance of the simulator. We first show that the simulator can be calibrated to match resultant forces and deformation fields from a state-of-the-art commercial solver and real-world cutting datasets; with generality across cutting velocities and object instances. We then show that Bayesian inference can be performed efficiently by leveraging the differentiability of the simulator; estimating posteriors over hundreds of parameters in a fraction of the time of derivative-free methods. Finally; we illustrate that control parameters in the simulation can be optimized to minimize cutting forces via lateral slicing motions. We publish videos and additional results on our project website at

In this paper; we address the trajectory planning problem in uncertain nonconvex static and dynamic environments that contain obstacles with probabilistic location; size; and geometry. To address this problem; we provide a risk bounded trajectory planning method that looks for continuous-time trajectories with guaranteed bounded risk over the planning time horizon. Risk is defined as the probability of collision with uncertain obstacles. Existing approaches to address risk bounded trajectory planning problems either are limited to Gaussian uncertainties and convex obstacles or rely on sampling-based methods that need uncertainty samples and time discretization. To address the risk bounded trajectory planning problem; we leverage the notion of risk contours to transform the risk bounded planning problem into a deterministic optimization problem. Risk contours are the set of all points in the uncertain environment with guaranteed bounded risk. The obtained deterministic optimization is; in general; nonlinear and nonconvex time-varying optimization. We provide convex methods based on sum-of-squares optimization to efficiently solve the obtained nonconvex time-varying optimization problem and obtain the continuous-time risk bounded trajectories without time discretization. The provided approach deals with arbitrary probabilistic uncertainties; nonconvex and nonlinear; static and dynamic obstacles; and is suitable for online trajectory planning problems.

This paper presents a novel method for continuous integration over the 3D rotation group SO(3). The key idea is to model the system's dynamics with Gaussian Processes (GPs) and learn the GP training data to match the system's instantaneous angular velocity measurements. This is formulated as the minimisation of a simple cost function that leverages the application of linear operators over GP kernels. The proposed integration method over SO(3) is applied to the preintegration of inertial data provided by a 6-DoF-Inertial Measurement Unit (IMU). Unlike standard integration that requires the recomputation of the integrals every time the estimate changes; preintegration combines the IMU readings into pseudo-measurements that are independent from the pose estimate and allows for efficient multi-modal sensor-fusion. The pseudo-measurements generated by the proposed method are named Unified Gaussian Preintegrated Measurements (UGPMs). UGPMs rely on GP regression and linear operators to analytically integrate the acceleration data. Moreover; a mechanism for IMU bias and time-shift correction is introduced to allow for seamless multi-modal state estimation. Over quantitative experiments; we show that the UGPMs outperform the current state-of-the-art preintegration methods.