ICRA 2019 Papers

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

We present a novel approach of multi-modal deep generative models and apply this to coordinated heterogeneous multi-agent active sensing. A major approach to achieve this objective is to train a multi-modal variational Auto Encoder (M2VAE) that integrates the information of different sensor modalities into a joint latent representation. Furthermore, we derive an objective from the M2VAE that enables the maximization of the evidence lower bound via selection of sensor modalities. Using this approach as a direct reward signal to a multi-modal and multi-agent deep reinforcement learning setup leads intuitively to an epistemic active sensing behavior that coordinately resolves the ambiguity of observations.

This paper presents a general multi-camera visual-inertial navigation system (mc-VINS) with online instrinsic and extrinsic calibration, which is able to utilize all the information from an arbitrary number of asynchronous cameras. In particular, within the standard multi-state constraint Kalman Filter (MSCKF) framework, we only clone the IMU poses related to a single “base camera” (rather than all cameras) in the state vector, while the IMU poses corresponding to all other camera images are represented via an interpolation of the poses bounding the measuring time. By doing so, we can fuse all observations from all cameras with inertial measurements while allowing for efficient, tightly-coupled state estimation through parallelization and asynchrony. Moreover, we perform online sensor calibration of each camera's intrinsics as well as the spatial (transformation) and temporal (time offset) extrinsic parameters between all involved sensors (cameras and IMU), thus enabling high-fidelity localization. We validate the proposed mc-VINS algorithm in various real-world experiments with different sensor configurations, showing the ability to offer real-time high-precision localization and calibration results.

Approximate dynamic programming platforms are employed to solve dynamic graphical games, where the agents interact among each other using communication graphs in order to achieve synchronization. Although the action dependent dual heuristic dynamic programming schemes provide fast solution platforms for several control problems, their capabilities degrade for systems with unknown or uncertain dynamical models. An online model-free adaptive learning solution based on action dependent dual heuristic dynamic programming is proposed to solve the dynamic graphical games. It employs distributed actor-critic neural networks to approximate the optimal value function and the associated model-free control strategy for each agent. This is done using a policy iteration process where it does not employ any extensive computational effort, as traditionally observed. The duality between the model-free coupled Bellman optimality equation and the underlying coupled Riccati equation is highlighted. This is followed by a graph simulation scenario to test the usefulness of the proposed policy iteration process.

A lot of motion planning research in robotics focuses on efficient means to find trajectories between individual start and goal regions, but it remains challenging to specify and plan robotic manipulation actions which consist of multiple interdependent subtasks. The Task Constructor framework we present in this work provides a flexible and transparent way to define and plan such actions, enhancing the capabilities of the popular robotic manipulation framework MoveIt!.11The Task Constructor framework is publicly available at https://github.com/ros-planning/moveit_task_constructor Subproblems are solved in isolation in black-box planning stages and a common interface is used to pass solution hypotheses between stages. The framework enables the hierarchical organization of basic stages using containers, allowing for sequential as well as parallel compositions. The flexibility of the framework is illustrated in multiple scenarios performed on various robot platforms, including bimanual ones.

Robotic teleoperation brings great potential for advances within the field of surgery. The ability of a surgeon to reach patient remotely opens exciting opportunities. Early experience with telerobotic surgery has been interesting, but the clinical feasibility remains out of reach, largely due to the deleterious effects of communication delays. Teleoperation tasks are significantly impacted by unavoidable signal latency, which directly results in slower operations, less precision in movements, and increased human errors. Introducing significant changes to the surgical workflow, for example by introducing semi-automation or self-correction, present too significant a technological and ethical burden for commercial surgical robotic systems to adopt. In this paper, we present three simple and intuitive motion scaling solutions to combat teleoperated robotic systems under delay and help improve operator accuracy. Motion scaling offers potentially improved user performance and reduction in errors with minimal change to the underlying teleoperation architecture. To validate the use of motion scaling as a performance enhancer in telesurgery, we conducted a user study with 17 participants, and our results show that the proposed solutions do indeed reduce the error rate when operating under high delay.

Motion Planning Templates (MPT) is a C++ template-based library that uses compile-time polymorphism to generate robot-specific motion planning code and is geared towards eking out as much performance as possible when running on the low-power CPU of a battery-powered small robot. To use MPT, developers of robot software write or leverage code specific to their robot platform and motion planning problem, and then have MPT generate a robot-specific motion planner and its associated data-structures. The resulting motion planner implementation is faster and uses less memory than general motion planning implementations based upon runtime polymorphism. While MPT loses runtime flexibility, it gains advantages associated with compile-time polymorphism- including the ability to change scalar precision, generate tightly-packed data structures, and store robot-specific data in the motion planning graph. MPT also uses compile-time algorithms to resolve the algorithm implementation, and select the best nearest neighbor algorithm to integrate into it. We demonstrate MPT's performance, lower memory footprint, and ability to adapt to varying robots in motion planning scenarios on a small humanoid robot and on 3D rigid-body motions.

Fast and efficient motion planning algorithms are crucial for many state-of-the-art robotics applications such as self-driving cars. Existing motion planning methods become ineffective as their computational complexity increases exponentially with the dimensionality of the motion planning problem. To address this issue, we present Motion Planning Networks (MPNet), a neural network-based novel planning algorithm. The proposed method encodes the given workspaces directly from a point cloud measurement and generates the end-to-end collision-free paths for the given start and goal configurations. We evaluate MPNet on various 2D and 3D environments including the planning of a 7 DOF Baxter robot manipulator. The results show that MPNet is not only consistently computationally efficient in all environments but also generalizes to completely unseen environments. The results also show that the computation time of MPNet consistently remains less than 1 second in all presented experiments, which is significantly lower than existing state-of-the-art motion planning algorithms.

Distributed tactile sensors on multi-fingered hands can provide high-dimensional information for grasping objects, but it is not clear how to optimally process such abundant tactile information. The current paper explores the possibility of using a morphology-specific convolutional neural network (MS-CNN). uSkin tactile sensors are mounted on an Allegro Hand, which provides 720 force measurements (15 patches of uSkin modules with 16 triaxial force sensors each) in addition to 16 joint angle measurements. Consecutive layers in the CNN get input from parts of one finger segment, one finger, and the whole hand. Since the sensors give 3D (x, y, z) vector tactile information, inputs with 3 channels (x, y and z) are used in the first layer, based on the idea of such inputs for RGB images from cameras. Overall, the layers are combined, resulting in the building of a tactile map based on the relative position of the tactile sensors on the hand. Seven different combination variations were evaluated, and an over-95% object recognition rate with 20 objects was achieved, even though only one random time instance from a repeated squeezing motion of an object in an unknown pose within the hand was used as input.

The state estimation and control of a ball-balancing robot under high yaw rate is a challenging problem due to its highly nonlinear 3D dynamic. The small size and low-cost components in our Micro Ball-Balancing Robot makes the system inherently very noisy which further increases the complexity of the problem. In order to drive the robot more aggressively such as translating and spinning at the same time, a good state estimator which works well under high yaw rates is required. This paper presents the derivation of a high yaw-rate Ball-Balancing Robot dynamic model and the implementation of said model in an Extended Kalman Filter (EKF) using raw on-board sensor measurements. The EKF using the new model is then compared to a Kalman Filter which uses a linearized dynamic model. The accuracy of the attitude estimates and the controller performance under high yaw rates were verified using a motion capture system.

In this paper we propose a new model for sequential manipulation tasks that also considers robot dynamics and time-variant environments. From this model we automatically derive constraint-based controllers and use them as steering functions in a kinodynamic manipulation planner. The resulting plan is not a trajectory but a sequence of controllers that react online to disturbances. We validated our approach in simulation and on a real robot. In the experiments our approach plans and executes dual-robot manipulation tasks with online collision avoidance and reactions to estimates of object poses.

This paper presents the modeling and control of a passively-coupled tilt-rotor vertical takeoff and landing aircraft. The aircraft consists of a quadrotor frame attached to a fixed-wing aircraft by an unactuated hinged mechanism. The platform is capable of smooth transitions from hover to forward flight without the use of tilting actuators. The transition from hover to forward flight is made possible by differential thrust between the fore and aft propellers of the quadrotor frame. In this paper, the coupled dynamics between the quadrotor frame and the aircraft frame are modeled as a constrained multi-body system. The equations of motion are established using a constrained Lagrangian approach and the model developed is used to build a realistic simulation environment for control design purpose. A cascaded control architecture based on P/PID controllers is proposed to achieve inner-loop attitude, height and forward velocity control. Simulated and experimental results are obtained with a close match for hover, transitions, forward flight, and banked turn maneuvers.

Developing a reliable model to identify the sea state is significant for the autonomous ship. This paper introduces a novel deep neural network model (SeaStateNet) to estimate the sea state based on the ship motion data from dynamically positioned vessels. The SeaStateNet mainly consists of three components: an Long-Short-Term Memory (LSTM) recurrent neural network to capture the long dependency in the ship motion data; a convolutional neural network (CNN) to extract time-invariant features; and a Fast Fourier Transform (FFT) block to extract frequency features. A feature fusion layer is designed to learn the degree affected by each component. The proposed model is applied directly to the raw time series data, without needing of any hand-engineered features. A sensitivity analysis (SA) method is applied to assess the influence of data preprocessing. Through benchmark test and experiment on ship motion dataset, SeaStateNet is verified effective for sea state estimation. The investigation on real-time test further shows the practicality of the proposed model.

In this paper, we propose and investigate a new approach to modeling variable curvature continuum robot sections, based on Euler spirals. Euler spirals, also termed Clothoids, or Cornu spirals, are those curves in which the curvature increases linearly with their arc length. In this work, Euler spirals are applied to the kinematic modeling of continuum robots for the first time. The approach was evaluated using the sections of numerous continuum robots, including two novel parallel continuum robots. Each robot consists of three parallel sections, each with three thin, long McKibben actuators. These sections are poorly modeled by the widely used constant curvature kinematic model. The constant curvature and Euler spiral models were compared and the Euler spiral method was seen to be a significantly better match for a wide range of configurations of the robot hardware.

In this work we introduce an approach that allows a quadcopter to find the velocity which maximizes its flight time (endurance) or flight distance (range) while moving along a given path, using on-board power measurement. The proposed strategy is based on Extremum Seeking control and (a) does not require any model of the power consumption of the system, (b) can be executed on-line, and (c) guarantees adaptation to unknown disturbances. We show experimentally that hovering is not the most energy-efficient loitering strategy, and we demonstrate the proposed method's ability to adapt to different aerodynamic disturbances, such as payloads. The method may be especially useful in applications where a quadcopter carries an unknown payload, allowing it to adapt for improved range.

We propose a novel model-free optimal estimation and sensor placement framework for a high-DOF (degree-of-freedom) EKC (elastic kinematic chain) with only a limited number of IMU (inertial measurement unit) sensors based on POD (proper orthogonal decomposition) and MAP (maximum a posteriori) estimation. First, we (off-line) excite the system richly enough, collect the data and perform the POD to extract dominant and non-dominant modes. We then decide the minimum number of IMUs according to the dominant modes, and construct the prior distribution of the output (i.e., top-end position of EKC) based on the singular value of each POD mode. We also formulate the MAP estimation given the prior distribution and different placements of the IMUs and choose the optimal IMU placement to maximize the posterior probability. This optimal placement is then used for real-time output estimation of the EKC. Experiments are also performed to verify the theory.

Flexibility commonly exists in the joints of many industrial robots due to the elasticity of the lightweight strain-wave type transmissions being used. This leads to a dynamic time-varying displacement between the position of the drive actuator and that of the driven link. Furthermore, the joint flexibility changes with time due to the material slowly being worn off at the gear meshing. Knowing the stiffness of the robot joints is of great value, e.g. in the design of new model-based feedforward and feedback controllers, and for predictive maintenance in the case of gearing unit failure. In this paper, we address on-line estimation of robot joint stiffness using a recursive least squares strategy based on a parametric model taking into account the elastic torques' nonlinear dependency on transmission deformation. Robustness is achieved in the presence of measurement noise and in poor excitation conditions. The method can be easily extended to general classes of serial-link multi-degree-of-freedom robots. The estimation technique uses only feedback signals that are readily available on Universal Robots' e-Series manipulators. Experiments on the new UR5e manipulator demonstrate the effectiveness of the proposed method.

Fruitful developments on continuum robots have been witnessed in recent years due to their movements and manipulation capabilities in confined spaces. Due to the nature that a continuum robot has an infinite number of DoFs (Degrees of Freedom), majority of the existing systems deployed abundant actuators such that the robot can be controlled in separately modeled and actuated segments with constant or variable curvature. As the shape of a continuum robot is always jointly determined by its actuation and the interactions from the environment, it is hence worth exploring the opposite approach that how a task can be accomplished with a minimal number of actuators. This paper presents the first step of such an investigation where a slim 3-actuator continuum robot is actuated to reach different spatial locations under gravity. As the gravity greatly affects the robot's shape, a monocular camera, together with two UKFs (Unscented Kalman Filters), was used to concurrently estimate the robot's shape and the feature depth. Then the estimated shape can be used in updating the kinematics model of the robot to achieve motion control. Experiments were conducted to validate the effectiveness of the proposed shape estimation, which promises the motion control implementation in the near future work.

The inverted pendulum is by nature a dynamically unstable system and may be subjected to severe disturbances due to its environmental or loading conditions. This paper formulates a design for a nonlinear controller to balance a two-wheeled mobile robot (TWMR) based on Model Reference Adaptive Control. The proposed solution overcomes the limitations of control systems that rely on fixed parameter controllers. Given the nonlinear single-input multi-output (SIMO) nature of the TWMR platform, the proposed adaptive controller can handle non-linearities without the need for linearization, and inherently dealing with SIMO systems. By studying the influence that hidden dynamic effects can cause, we show the preference of the proposed controller over other designs. Simulation results demonstrate the applicability and efficiency of our proposed design, and experimental results validate the effectiveness of the proposed scheme in guaranteeing asymptotic output tracking, even in the presence of unknown disturbances.