ICRA 2019 Papers

This paper presents a methodology for synthesizing a motion plan for a mobile robot to ensure that the robot never gets depleted with battery charge while carrying out its mission successfully. The specification of the robot is provided in the form of an LTL (Linear Temporal Logic) formula. A trajectory satisfying an LTL formula may contain a loop whose repetitive execution causes the depletion of battery charge in the robot. The motion plan generated by our methodology ensures that the robot visits the charging station periodically in such a way that it never gets depleted with battery charge while carrying out its mission optimally. Given a set of potential charging station locations and an LTL specification, our algorithm also finds the best location for the charging station along with the optimal trajectory for the robot. We encode the motion planning problem as an SMT (Satisfiability Modulo Theory) solving problem and use the off-the-shelf SMT solver Z3 to solve the constraints to find the location of the charging station and generate an optimal trajectory for the robot. We apply our methodology to synthesize energy-aware trajectories for robots with different dynamics in various workspaces and for various LTL specifications.

The possibility of adapting online the way a robot interacts with the environment is becoming more and more important. In this paper we introduce the tank based admittance controller. We show that all the admittance controllers can be modeled as an energy optimization problem and then we introduce a novel admittance control strategy that allows to change online the interactive behavior while preserving a stable interaction with the environment. The effectiveness of the proposed architecture is experimentally validated.

In this work, we show how the interactive behavior of an aerial robot can be modeled and controlled effectively and elegantly in the port-Hamiltonian framework. We present an observer-based wrench/impedance controller for a fully-actuated hexarotor. The analysis and control are performed in a geometrically consistent manner on the configuration manifold of the special Euclidean group SE (3) such that the UAV's nonlinear geometric structure is exploited. The controller uses a wrench observer to estimate the interaction wrench without the use of a force/torque sensor. Moreover, the concept of energy tanks is used to guarantee the system's overall contact stability to arbitrary passive environments. The reliability and robustness of the proposed approach is validated through simulation and experiment.

This paper presents a novel actuation model and control allocation strategy for a redundantly-actuated multirotor unmanned aerial vehicle (UAV), referred to as the omnicopter. With an unconventional configuration, the omnicopter's eight propellers are able to produce all the six components of net force/torque, with two degrees of actuation redundancy. This enables the vehicle to execute motion trajectories unattainable with conventional underactuated multi-rotors. A new inverse actuator model is proposed that accounts for the significant interactions between propeller airflows by relating their output thrust forces to their input motor commands. Actuation redundancy is resolved by solving a convex constrained optimization problem. Its solution yields the most power efficient set of propellers thrusts that would produce a required net force/torque, while respecting the propeller thrust limits. When the required force/torque is infeasible due to the thrust limits, the solution would minimize the norm of the error between the desired and actual net force/torque vectors. Experimental results demonstrate the effectiveness of the proposed model and control allocation strategy.

In this work, we present a within-hand manipulation approach that leverages a simple energy model based on caging grasps made by underactuated hands. Instead of explicitly modeling the contacts and dynamics in manipulation, we can calculate a map to describe the energy states of different hand-object configurations under an actuation input. Since the system intrinsically steers towards low energy states, the object's movement is uniquely described by the gradient of the energy map if the corresponding actuation is applied. Such maps are pre-calculated for a range of actuation inputs to represent the system's energy profile. We discretize the workspace into a grid and construct an energy gradient-based graph by locally exploring the gradients of the stored energy profile. Given a goal configuration of a simple cylindrical object, a sequence of actuation inputs can be calculated to manipulate it towards the goal by exploiting the connectivity in the graph. The proposed approach is experimentally implemented on a Yale T42 hand. Our evaluation results show that parts of the graph are well connected, explaining our ability to successfully plan and execute trajectories within the gripper's workspace.

Energy-efficient navigation is an important technology for mobile robots because of its potential to increase the operation time of the robot. In particular, when coupled with a dynamic legged quadruped, the need for energy savings is made more apparent as payloads are limited. Due to the complexity in modeling motion and power models of these robots, a new approach is necessary to effectively motion plan for these complex robots. We accomplish this by using Sampling-Based Model Predictive optimization (SBMPO) which was extended for use on the LLAMA quadrupedal platform in simulation. SBMPO allows for direct generation of trajectories while using a heuristic-based search to speed up computations. This approach is shown to effectively motion plan while optimizing for energy consumption and maintaining the natural dynamics of the robot in a simulated environment.

Passivity is a necessary condition for a system's stability, meaning that an energy generating system may readily become unstable. Energy-aware actuation can enforce passivity by monitoring the amount of energy that is exchanged with a system, while using an allocated energy budget to execute a task. Careful communication of the energy budgets is important to prevent accidental generation of energy. Therefore, this paper proposes an energy transaction protocol to communicate energy budgets in a distributed robotic system to guarantee that passivity is kept. Simulations are performed with a model of the protocol that is applied to a simulated unreliable communication channel. It is verified that the proposed protocol keeps passivity in the system, while a naive communication strategy either violates passivity or is unnecessarily dissipative.

Endoscopic surgery is an increasingly popular alternative to laparoscopic techniques for many conditions, as the operation site can be reached without skin wounds. In many tasks, sufficient force generation is desired. As endoscopes must be highly flexible and slim, however, the force generation and sensing capabilities associated with these tools is limited due their compliance, significantly hindering the adoption rate of endoscopic surgeries. This paper proposes a technique, termed `environmental scaffolding', to stabilize an actuated, flexible segment in the intestine such that larger forces can be applied. Through the measurement of actuation forces, a method for intrinsically sensing multiple contact forces when in this configuration is presented. Experimental results show that with the environmental scaffolding technique, the tip force generated can be increased by over 50% on average compared to using the device in a purely cantilevered configuration, and the tip force estimation is accurate to within 2.97%.

Mixed Reality (MR) is a promising interface for robot programming because it can project an immersive 3D visualization of a robot's intended movement onto the real world. MR can also support hand gestures, which provide an intuitive way for users to construct and modify robot motions. We present a Mixed Reality Head-Mounted Display (MRHMD) interface that enables end-users to easily create and edit robot motions using waypoints. We describe a user study where 20 participants were asked to program a robot arm using 2D and MR interfaces to perform two pick-and-place tasks. In the primitive task, participants created typical pickand-place programs. In the adapted task, participants adapted their primitive programs to address a more complex pickand-place scenario, which included obstacles and conditional reasoning. Compared to the 2D interface, a higher number of users were able to complete both tasks in significantly less time, and reported experiencing lower cognitive workload, higher usability, and higher naturalness with the MR-HMD interface.

This paper introduces an end-to-end learning approach based on Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) for a multi-layered spiking neural network (SNN). As a case study, a snake-like robot is used as an agent to perform target tracking tasks on the basis of our proposed approach. Since the key of R-STDP is to use rewards to modulate synapse strengthens, we first propose a general way to propagate the reward back through a multi-layered SNN. Upon the proposed approach, we build up an SNN controller that drives a snake-like robot for performing target tracking tasks. We demonstrate the practicability and advantage of our approach in terms of lateral tracking accuracy by comparing it to other state-of-the-art learning algorithms for SNNs based on R-STDP.

Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. During surgery the tool needs to be inserted into the eyeball while maintaining constant contact with the sclera. Any unexpected manipulation could cause extreme tool-sclera contact force (scleral force) thus damage the sclera. The introduction of robotic assistance could enhance and expand the surgeon's manipulation capabilities during surgery. However, the potential intra-operative danger from surgeon's mis-operations remains difficult to detect and prevent by existing robotic systems. Therefore, we propose a method to predict imminent unsafe manipulation in robot-assisted retinal surgery and generate feedback to the surgeon via auditory substitution. The surgeon could then react to the possible unsafe events in advance. This work specifically focuses on minimizing sclera damage using a force-sensing tool calibrated to measure small scleral forces. A recurrent neural network is designed and trained to predict the force safety status up to 500 milliseconds in the future. The system is implemented using an existing “steady hand” eye robot. A vessel following manipulation task is designed and performed on a dry eye phantom to emulate the retinal surgery and to analyze the proposed method. Finally, preliminary validation experiments are performed by five users, the results of which indicate that the proposed early warning system could help to reduce the number of unsafe manipulation events.

Person identification and tracking (PIT) is an essential issue in computer vision and robotic applications. It has long been studied and achieved by technologies such as RFID or face/fingerprint/iris recognition. These approaches, however, have their limitations due to environmental constraints (such as lighting and obstacles) or require close contact to specific devices. Therefore, their recognition accuracy highly depends on use scenarios. In this work, we propose RCU (Robot Catch yoU), an accompanyist robot system that provides follow-me or guide-me services. Such robots are capable of distinguishing users' profiles in front of them and keep tracking a specific target person. We study a more challenging scenario where the target person may be under occlusion from time to time. To enable robust PIT, we develop a data fusion technique that integrates two types of sensors, an RGB-D camera and wearable inertial sensors. Since the data generated by these sensors share common features, we are able to fuse them to achieve identity-aware tracking. Practical issues, such as time synchronization and coordinate calibration, are also addressed. We implement our design on a robotic platform and show that it can track a target person even when no biological feature is captured by the RGB-D camera. Our experimental evaluation shows a recognition rate of 95% and a following rate of 88%.

In this paper we present an end-to-end deep learning framework to turn images that show dynamic content, such as vehicles or pedestrians, into realistic static frames. This objective encounters two main challenges: detecting all the dynamic objects, and inpainting the static occluded background with plausible imagery. The former challenge is addressed by the use of a convolutional network that learns a multiclass semantic segmentation of the image. The second problem is approached with a conditional generative adversarial model that, taking as input the original dynamic image and its dynamic/static binary mask, is capable of generating the final static image. These generated images can be used for applications such as augmented reality or vision-based robot localization purposes. To validate our approach, we show both qualitative and quantitative comparisons against other state-of-the-art inpainting methods by removing the dynamic objects and hallucinating the static structure behind them. Furthermore, to demonstrate the potential of our results, we carry out pilot experiments that show the benefits of our proposal for visual place recognition.

Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixel-level observations for future vehicle localization. We show that incorporating dense optical flow improves prediction results significantly since it captures information about motion as well as appearance change. We also find that explicitly modeling future motion of the ego-vehicle improves the prediction accuracy, which could be especially beneficial in intelligent and automated vehicles that have motion planning capability. To evaluate the performance of our approach, we present a new dataset of first-person videos collected from a variety of scenarios at road intersections, which are particularly challenging moments for prediction because vehicle trajectories are diverse and dynamic. Code and dataset have been made available at: https://usa.honda-ri.com/hevi.

Robotic exoskeletons open up promising interventions during post-stroke rehabilitation by assisting individuals with sensorimotor impairments to complete therapy tasks. These devices have the ability to provide variable assistance tailored to individual-specific needs and, additionally, can measure several parameters associated with the movement execution. Metrics representative of movement quality are important to guide individualized treatment. While robots can provide data with high resolution, robustness, and consistency, the delineation of the human contribution in the presence of the kinematic guidance introduced by the robotic assistance is a significant challenge. In this paper, we propose a method for assessing voluntary effort from an individual fitted in an upper-body exoskeleton called Harmony. The method separates the active torques generated by the wearer from the effects caused by unmodeled dynamics and passive neuromuscular properties and involuntary forces. Preliminary results show that the effort estimated using the proposed method is consistent with the effort associated with muscle activity and is also sensitive to different levels, indicating that it can reliably evaluate user's contribution to movement. This method has the potential to serve as a high resolution assessment tool to monitor progress of movement quality throughout the treatment and evaluate motor recovery.

There has been considerable recent work in motion planning for UAVs to enable aggressive, highly dynamic flight in known environments with motion capture systems. However, these existing planners have not been shown to enable the same kind of flight in unknown, outdoor environments. In this paper we present a receding horizon planning architecture that enables the fast replanning necessary for reactive obstacle avoidance by combining three techniques. First, we show how previous work in computationally efficient, closed-form trajectory generation method can be coupled with spatial partitioning data structures to reason about the geometry of the environment in real-time. Second, we show how to maintain safety margins during fast flight in unknown environments by planning velocities according to obstacle density. Third, our receding-horizon, sampling-based motion planner uses minimum-jerk trajectories and closed-loop tracking to enable smooth, robust, high-speed flight with the low angular rates necessary for accurate visual-inertial navigation. We compare against two state-of-the-art, reactive motion planners in simulation and benchmark solution quality against an offline global planner. Finally, we demonstrate our planner over 80 flights with a combined distance of 22km of autonomous quadrotor flights in an urban environment at speeds up to 9.4ms ^{-1}^{-1}.

When humans and robots perform complex tasks together, the robot must have a strategy to choose its actions based on observed human behavior. One well-studied approach for finding such strategies is reactive synthesis. Existing approaches for finite-horizon tasks have used an explicit state approach, which incurs high runtime. In this work, we present a compositional approach to perform synthesis for finite-horizon tasks based on binary decision diagrams. We show that for pick-and-place tasks, the compositional approach achieves orders-of-magnitude speed-ups compared to previous approaches. We demonstrate the synthesized strategy on a UR5 robot.

We present an algorithm that produces a plan for relocating obstacles in order to grasp a target in clutter by a robotic manipulator without collisions. We consider configurations where objects are densely populated in a constrained and confined space. Thus, there exists no collision-free path for the manipulator without relocating obstacles. Since the problem of planning for object rearrangement has shown to be NP-hard, it is difficult to perform manipulation tasks efficiently which could frequently happen in service domains (e.g., taking out a target from a shelf or a fridge). Our proposed planner employs a collision avoidance scheme which has been widely used in mobile robot navigation. The planner determines an obstacle to be removed quickly in real time. It also can deal with dynamic changes in the configuration (e.g., changes in object poses). Our method is shown to be complete and runs in polynomial time. Experimental results in a realistic simulated environment show that our method improves up to 31% of the execution time compared to other competitors.

Coupling the endface of an optical fiber to an integrated optical component is currently a low-throughput and costly manual process in the fabrication of the optical devices. In order to meet the high-volume demand for commercial optoelectronic devices, coupling must be automated. This paper presents a robotic positioning system and corresponding path planning strategy based on both the position and light intensity feedback. In this work, a micro-robotic positioning system with 3 degrees of freedoms (DOFs) is developed and integrated with an optical microscopy. Then the fuzzy controller is developed to design the trajectory. Lastly, simulation and experimental results demonstrate the accuracy and efficiency of the proposed system. Compared with the traditional manual method, the robotic positioning system can realize the coupling within 40 seconds. This method will have a significant impact on the automatic process of the micro manufacture field.