ICRA 2019 Papers

We present a Deep Learning based system for the twin tasks of localization and obstacle avoidance essential to any mobile robot. Our system learns from conventional geometric SLAM, and outputs, using a single camera, the topological pose of the camera in an environment, and the depth map of obstacles around it. We use a CNN to localize in a topological map, and a conditional VAE to output depth for a camera image, conditional on this topological location estimation. We demonstrate the effectiveness of our monocular localization and depth estimation system on simulated and real datasets.

In the last decade, supervised deep learning approaches have been extensively employed in visual odometry (VO) applications, which is not feasible in environments where labelled data is not abundant. On the other hand, unsupervised deep learning approaches for localization and mapping in unknown environments from unlabelled data have received comparatively less attention in VO research. In this study, we propose a generative unsupervised learning framework that predicts 6-DoF pose camera motion and monocular depth map of the scene from unlabelled RGB image sequences, using deep convolutional Generative Adversarial Networks (GANs). We create a supervisory signal by warping view sequences and assigning the re-projection minimization to the objective loss function that is adopted in multi-view pose estimation and single-view depth generation network. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI [1] and Cityscapes [2] datasets show that the proposed method outperforms both existing traditional and unsupervised deep VO methods providing better results for both pose estimation and depth recovery.

Localization is one of the main challenges to be addressed to develop autonomous vehicles able to perform complex maneuvers on roads opened to public traffic. Having an accurate dead-reckoning system is an essential step to reach this objective. This paper presents a dead-reckoning model for car-like vehicles that performs the data fusion of complementary and redundant sensors: wheel encoders, yaw rate gyro and steering wheel measurements. In order to get an accurate dead-reckoning system with a drift reduced to the minimum, the parameters have to be well calibrated and the procedure has to be simple and efficient. We present a method able to accurately calibrate the parameters without knowing the ground truth by using a Rauch-Tung-Striebel smoothing scheme which enables to obtain state estimates as close to the ground truth as possible. The smoothed estimates are then used within a optimization process to calibrate the model parameters. The method has been tested using data recorded from an experimental vehicle on public roads. The results show a significant diminution of the dead-reckoning drift compared to a commonly used calibration method. We evaluate finally the average distance a vehicle can navigate without exteroceptive sensors by using the proposed four-wheeled dead reckoning system.

Humans and robots team together to perform tasks in various domains. Some tasks are easier to perform than others, but little work focuses on discovering the underlying mechanisms that affect perceived difficulty and task performance. To fill this gap, we propose a formalized approach to task characterization for human-robot teams using Taguchi design of experiments and conjoint analysis. With this, we conduct a 20 person study where participants operate a 6 degree of freedom robotic arm to perform manipulations defined by 6 kinematic features. We find that rotational features of a task contribute significantly more to decreased performance and increased difficulty than translational features. The participants also perform the activities with autonomy assistance. The data shows a reduction in the effect of these features on performance and difficulty when assistance is active. Furthermore, we examine when to trigger assistance based on thresholds set from outlier detection. The analysis indicates that rotational features and features leading to kinematic singularities are useful for triggering assistance.

We consider the problem of learning structured, closed-loop policies (feedback laws) from demonstrations in order to control under-actuated robotic systems, so that formal behavioral specifications such as reaching a target set of states are satisfied. Our approach uses a “counterexample-guided” iterative loop that involves the interaction between a policy learner, a demonstrator and a verifier. The learner is responsible for querying the demonstrator in order to obtain the training data to guide the construction of a policy candidate. This candidate is analyzed by the verifier and either accepted as correct, or rejected with a counterexample. In the latter case, the counterexample is used to update the training data and further refine the policy.The approach is instantiated using receding horizon model-predictive controllers (MPCs) as demonstrators. Rather than using regression to fit a policy to the demonstrator actions, we extend the MPC formulation with the gradient of the cost-to-go function evaluated at sample states in order to constrain the set of policies compatible with the behavior of the demonstrator. We demonstrate the successful application of the resulting policy learning schemes on two case studies and we show how simple, formally-verified policies can be inferred starting from a complex and unverified nonlinear MPC implementations. As a further benefit, the policies are many orders of magnitude faster to implement when compared to the original MPCs.

This paper introduces a force-controllable quadruped robot system consists of twelve Actuator Modules embedded a novel Capacitive-type Joint Torque Sensor (CJTS) which is accurate (0.05 Nm), robust to impact, and easy to manufacture at low cost. The Actuator Module with CJTS shows accurate joint torque controllability in range of ±70 Nm (90 % settling time 0.04 s). The leg made by the three Actuator Modules is capable of generating forces in the z-axis up to 350 N and shows force control performances with zero-force control and lifting weights in three-dimensional space. To reduce the reflected limb inertia, all the Actuator Modules are located on the body frame and light-weight limbs made of the carbon pipe (3.6 % of total body weight). The introduced robot performed the motion on various terrains with walking/trot gaits.

Recent failures in real-world self-driving tests have suggested a paradigm shift from directly learning in real-world roads to building a high-fidelity driving simulator as an alternative, effective, and safe tool to handle intricate traffic environments in urban areas. To date, traffic simulation can construct virtual urban environments with various weather conditions, day and night, and traffic control for autonomous vehicle testing. However, mutual interactions between autonomous vehicles and pedestrians are rarely modeled in existing simulators. Besides vehicles and pedestrians, the usage of personal mobility devices is increasing in congested cities as an alternative to the traditional transport system. A simulator that considers all potential road-users in a realistic urban environment is urgently desired. In this work, we propose a novel, extensible, and microscopic method to build heterogenous traffic simulation using the force-based concept. This force-based approach can accurately replicate the sophisticated behaviors of various road users and their interactions through a simple and unified way. Furthermore, we validate our approach through simulation experiments and comparisons to the popular simulators currently used for research and development of autonomous vehicles.

This paper presents Flying STAR (FSTAR) a reconfigurable hybrid flying quadcopter robot. FSTAR is the latest in the family of the STAR robots fitted with a sprawling mechanism and propellers allowing it to both run and fly using the same motors. The combined capabilities of running and flying allows FSTAR to fly over obstacles or run underneath them and move inside pipes. The robot can reduce its width to crawl in confined spaces or underneath obstacles while touching the ground. We first describe the design of the robot and the configuration of the wheels and propellers in the flying and running modes. Then we present the 3D printed prototype of the FSTAR robot which we used for our experiments. We evaluate the energy requirements of the robot and the forces it can generate. The experimental robot can fly like an ordinary quadcopter but can also run on the ground at a speed of up to 2.6 m/s to save energy (see video).

With increasing autonomy, robots like drones are increasingly accessible to untrained users. Most users control drones using a low-level interface, such as a radio-controlled (RC) controller. For a wider adoption of these technologies by the public, a much higher-level interface, such as natural language or mixed reality (MR), allows the automation of the control of the agent in a goal-oriented setting. We present an interface that uses natural language grounding within an MR environment to solve high-level task and navigational instructions given to an autonomous drone. To the best of our knowledge, this is the first work to perform fully autonomous language grounding in an MR setting for a robot. Given a map, our interface first grounds natural language commands to reward specifications within a Markov Decision Process (MDP) framework. Then, it passes the reward specification to an MDP solver. Finally, the drone performs the desired operations in the real world while planning and localizing itself. Our approach uses MR to provide a set of known virtual landmarks, enabling the drone to understand commands referring to objects without being equipped with object detectors for multiple novel objects or a predefined environment model. We conducted an exploratory user study to assess users' experience of our MR interface with and without natural language, as compared to a web interface. We found that users were able to command the drone more quickly via both MR interfaces as compared to the web interface, with roughly equal system usability scores across all three interfaces.

A method was previously developed by this author to optimise the flight path of a fixed wing UAV performing aerial surveys of complex concave agricultural fields. This relies heavily on a flight time in wind prediction model as its cost function. This paper aims to validate this model by comparing flight test results with the model prediction. There are a number of assumptions that this model relies on. The major assumption is that wind is steady and uniform over the small area and time scales involved in a survey. To show that this is reasonable, wind fields measurements will be taken from a multi rotor UAV with an ultrasonic windspeed sensor.

We propose a combined method for the collaborative transportation of a suspended payload by a team of rotorcraft. A recent distance-based formation-motion control algorithm based on assigning distance disagreements among robots generates the acceleration signals to be tracked by the vehicles. In particular, the proposed method does not need global positions nor tracking prescribed trajectories for the motion of the members of the team. The acceleration signals are followed accurately by an Incremental Nonlinear Dynamic Inversion controller designed for rotorcraft that measures and resists the tensions from the payload. Our approach allows us to analyze the involved accelerations and forces in the system so that we can calculate the worst case conditions explicitly to guarantee a nominal performance, provided that the payload starts at rest in the 2D centroid of the formation, and it is not under significant disturbances. For example, we can calculate the maximum safe deformation of the team with respect to its desired shape. We demonstrate our method with a team of four rotorcraft carrying a suspended object two times heavier than the maximum payload for an individual. Last but not least, our proposed algorithm is available for the community in the open-source autopilot Paparazzi.

Insects and hummingbirds exhibit extraordinary flight performance and can simultaneously master seemingly conflicting goals: stable hovering and aggressive maneuvering, which are unmatched by conventional small scale man-made vehicles. Flapping Wing Micro Air Vehicles (FWMAVs) hold great promise for closing this performance gap. However, design and control of such systems remain challenging. Here, we present an open source high fidelity dynamic simulation for FWMAVs. The simulator serves as a testbed for the design, optimization and flight control of FWMAVs. To validate the simulation, we recreated the at-scale hummingbird robot developed in our lab in the simulation. System identification was performed to obtain the model parameters. Force generation and dynamic response of open-loop and closed loop systems between simulated and experimental flights were compared. The unsteady aerodynamics and the highly nonlinear flight dynamics present challenging control problems for conventional and learning control algorithms such as Reinforcement Learning. The interface of the simulation is fully compatible with OpenAI Gym environment. As a benchmark study, we present a linear controller for hovering stabilization and a Deep Reinforcement Learning control policy for goal-directed maneuvering. Finally, we demonstrate direct simulation-to-real transfer of both control policies onto the physical robot, further demonstrating the fidelity of the simulation.

Robot-diver communication underwater is complicated by the attenuation of RF signals, the complexities of the environment in terms of deploying interaction devices, and issues related to the cognitive loading of human operators. Humans operating underwater have developed a simple yet effective strategy for diver-diver communication based on the visual recognition of gestures. Can a similar approach be effective for diver-robot communication? Here we present experiments with SCUBANet, an underwater detection dataset of body parts associated with diver-robot communication. Given the nature of standard diver gestures, here we concentrate on diver recognition and in particular on diver body-head-hand localization and examine the feasibility of using a CNN-based approach to address this problem. Such data-driven approaches typically require an appropriately annotated dataset. The SCUBANet dataset contains images of object classes commonly encountered during human-robot communication underwater. Object classes are labeled using per-instance bounding boxes. Annotations were created through crowd sourcing via a web-based interface to ease deployment. We provide baseline performance on diver and diver component recognition and localization using transfer learning on three widely available pre-trained models.

We present a sampling-based framework for feedback motion planning of legged robots. Our framework is based on switching between limit cycles at a fixed instance of motion, the Poincaré section (e.g., apex or touchdown), by finding overlaps between the regions of attraction (ROA) of two limit cycles. First, we assume a candidate orbital Lyapunov function (OLF) and define a ROA at the Poincaré section. Next, we solve multiple trajectory optimization problems, one for each sampled initial condition on the ROA to minimize an energy metric and subject to the exponential convergence of the OLF between two steps. The result is a table of control actions and the corresponding initial conditions at the Poincaré section. Then we develop a control policy for each control action as a function of the initial condition using deep learning neural networks. The control policy is validated by testing on initial conditions sampled on ROA of randomly chosen limit cycles. Finally, the rapidly-exploring random tree algorithm is adopted to plan transitions between the limit cycles using the ROAs. The approach is demonstrated on a hopper model to achieve velocity and height transitions between steps.

This paper presents 2D feedback control and open loop 3D trajectories of heterogeneous chemically catalyzing Janus particles. Self-actuated particles have enormous implications for both in vivo and in vitro environments, which make them a diverse resource for a variety of medical and assembly applications. Janus particles, consisting of cobalt and platinum hemispheres, can self-propel in hydrogen peroxide solutions due to platinum's catalyzation properties. These particles are directionally controlled using static magnetic fields produced from a triaxial approximate Helmholtz coil system. Since the magnetization direction of Janus particles is often heterogeneous, and thereby not consistent with the propulsion direction, this creates a unique opportunity to explore the motion effects of these particles under 2D feedback control and open loop 3D control. Using a modified closed loop controller, Janus particles with magnetization both closely aligned and greatly misaligned to the propulsion vectors, were instructed to perform complex trajectories. These trajectories were then compared between trials to measure both consistency and accuracy. The effects of increasing offset between the magnetization and propulsion vectors were also analyzed. The effects this heterogeneity had on 3D motion is also briefly discussed. It is our hope going forward to develop a 3D closed loop control system that can retroactively account for variations in the magnetization vector.

We consider the problem of feasible coordination control for multiple homogeneous or heterogeneous mobile vehicles subject to various constraints (nonholonomic motion constraints, holonomic coordination constraints, equality/inequality constraints etc). We develop a general framework involving differential-algebraic equations and viability theory to describe and determine coordination feasibility for a coordinated motion control under heterogeneous vehicle dynamics and various constraints. A heuristic algorithm is proposed for generating feasible trajectories for each individual vehicle. We show several application examples and simulation experiments on multi-vehicle coordination under various constraints to validate the theory and the effectiveness of the proposed algorithm and control schemes.

In this paper, we present the design of a steerable needle with proximal notch patterns for compliance and an embedded rotational tip joint for articulation. The device is fabricated by laser machining NiTi tube so that an inner working channel exists (to enable delivery of fluids, drugs or microtools) and no assembly is required for the joints. We formulate its model based on the classical Cosserat Rod theory. This is extended with incremental state prediction and a simple spring model for tissue reaction to integrate into a planning algorithm based on Dynamic Region RRT which efficiently explores the needle's state space. The planner was initialized with a target zone and arbitrary anatomical obstacles before running simulations which propagated incremental state changes at every step while adhering to constraints based on the physical system. Finally, we demonstrate the steering capability of the needle through insertion tests into a phantom.

This paper deals with motion control for an 8-degree-of-freedom (DOF) high performance driving simulator. We formulate a constrained optimal control that defines the dynamical behavior of the system. Furthermore, the paper brings together various methodologies for addressing feasibility issues arising in implicit model predictive control-based motion cueing algorithms. The implementation of different techniques is described and discussed subsequently. Several simulations are carried out in the simulator platform. It is observed that the only technique that can provide ensured closed-loop stability by assuring feasibility over all prediction horizons is a braking law that basically saturates the control inputs in the constrained form.

Compared to other vertical take-off and landing (VTOL) systems, a tailsitter minimizes the number of actuators and moving parts necessary. The downside of having a minimalistic actuation is its inherent low fault-tolerance. The failure of an actuator usually results in a loss of controllability, resulting in a crash. In this paper we analyze possible actuator failures and the constraints they pose on the capabilities of the system. We further present light-weight adaptations to the nominal flight controller to make it fault-tolerant. The fault-tolerant controller is implemented on a small tailsitter VTOL aircraft and adjusted to the system by means of extensive experimental studies. Finally, the capabilities and performance under failures are demonstrated and analyzed.