IROS 2020 Papers

A motion taxonomy can encode manipulations as a binary-encoded representation, which we refer to as motion codes. These motion codes innately represent a manipulation action in an embedded space that describes the motion’s mechanical features, including contact and trajectory type. The key advantage of using motion codes for embedding is that motions can be more appropriately defined with robotic-relevant features, and their distances can be more reasonably measured using these motion features. In this paper, we develop a deep learning pipeline to extract motion codes from demonstration videos in an unsupervised manner so that knowledge from these videos can be properly represented and used for robots. Our evaluations show that motion codes can be extracted from demonstrations of action in the EPIC-KITCHENS dataset.

Estimating the inertial properties of an object can make robotic manipulations more efficient, especially in extreme environments. This paper presents a novel method of estimating the 2D inertial parameters of an object, by having a robot applying a push on it. We draw inspiration from previous analyses on quasi-static pushing mechanics, and introduce a data-driven model that can accurately represent these mechanics and provide a prediction for the object's inertial parameters. We evaluate the model with two datasets. For the first dataset, we set up a V-REP simulation of seven robots pushing objects with large range of inertial parameters, acquiring 48000 pushes in total. For the second dataset, we use the object pushes from the MIT M-Cube lab pushing dataset. We extract features from force, moment and velocity measurements of the pushes, and train a Multi-Output Regression Random Forest. The experimental results show that we can accurately predict the 2D inertial parameters from a single push, and that our method retains this robust performance under various surface types.

This paper deals with the problem of planning grasp strategies on constrained and cluttered scenarios. The planner sequences the objects for grasping by considering multiple factors: (i) possible environmental constraints that can be exploited to grasp an object, (ii) object neighborhood, (iii) capability of the arm, and (iv) confidence score of the vision algorithm. To successfully exploit the environmental constraints, this work uses the CLASH hand, a compliant hand that can vary its passive stiffness. The hand can be softened such that it can comply with the object shape, or it can be stiffened to pierce between the objects in clutter. A stiffness decision tree is introduced to choose the best stiffness setting for each particular scenario. In highly cluttered scenarios, a finger position planner is used to find a suitable orientation for the hand such that the fingers can slide in the free regions around the object. Thus, the grasp strategy planner predicts not only the sequence in which the objects can be grasped, but also the required stiffness of the end effector, and the appropriate positions for the fingers around the object. Different experiments are carried out in the context of grocery handling to test the performance of the planner in scenarios that require different grasping strategies.

Haptic simulators can help medical students to train and improve their skills before practicing with a real patient. However, the vast majority of needle insertion haptic simulators are based on sophisticated models that are accurate but highly demanding in computing resources. Most of them do not provide haptic feedback and/or are not suitable for haptic control due to their computing time. In this paper, we presented a new low computing consuming method that aims to provide a realistic needle insertion experience to the student. A description of the proposed solution is provided, and it is illustrated by experimental results to highlight its performance.

Simulation data can be utilized to extend real-world driving data in order to cover edge cases, such as vehicle accidents. The importance of handling edge cases can be observed in the high societal costs in handling car accidents, as well as potential dangers to human drivers. In order to cover a wide and diverse range of all edge cases, we systemically parameterize and simulate the most common accident scenarios. By applying this data to autonomous driving models, we show that transfer learning on simulated data sets provide better generalization and collision avoidance, as compared to random initialization methods. Our results illustrate that information from a model trained on simulated data can be inferred to a model trained on real-world data, indicating the potential influence of simulation data in real world models and advancements in handling of anomalous driving scenarios.

Hybrid locomotion, which combines multiple modalities of locomotion within a single robot, enables robots to carry out complex tasks in diverse environments. This paper presents a novel method for planning multi-modal locomotion trajectories using approximate dynamic programming. We formulate this problem as a shortest-path search through a state-space graph, where the edge cost is assigned as optimal transport cost along each segment. This cost is approximated from batches of offline trajectory optimizations, which allows the complex effects of vehicle under-actuation and dynamic constraints to be approximately captured in a tractable way. Our method is illustrated on a hybrid double-integrator, an amphibious robot, and a flying-driving drone, showing the practicality of the approach.

One of the key factors for extended autonomy and resilience of multi-robot systems, especially when robots operate on batteries, is their ability to maintain energy sufficiency by recharging when needed. In situations with limited access to charging facilities, robots need to be able to share and coordinate recharging activities, with guarantees that no robot will run out of energy. In this work, we present an approach based on Control Barrier Functions (CBFs) to enforce both energy sufficiency (assuring that no robot runs out of battery) and coordination constraints (guaranteeing mutual exclusive use of an available charging station), all in a mission agnostic fashion. Moreover, we investigate the system capacity in terms of the relation between feasible requirements of charging cycles and individual robot properties. We show simulation results, using a physics-based simulator and real robot experiments to demonstrate the effectiveness of the proposed approach.

Surgical navigation is challenging on complicated multi-branch structures such as intrarenal collecting systems or bronchi. The objective of this work is to help surgeons quickly establish the corresponding relationship between intraoperative endoscopic images and preoperative CT data. An endoscopic navigation method is proposed based on three-dimensional structure registration. It mainly includes three sections. First, a reconstruction method is presented to obtain three-dimensional information of porous structures from endoscopic images. It combines image enhancement, structure-from-motion and template matching. Second, a hole search strategy based on slicing is given for detecting and extracting three-dimensional porous structures from CT data. Third, a similarity measurement algorithm is developed for registering endoscopic images to CT data. The performance of this work is evaluated on the data from the ureteroscopic holmium laser lithotripsy and the results show its accuracy, robustness and time cost.

In this paper, we tackle the problem of detecting objects in 3D and forecasting their future motion in the context of self-driving. Towards this goal, we design a novel approach that explicitly takes into account the interactions between actors. To capture their spatial-temporal dependencies, we propose a recurrent neural network with a novel Transformer [1] architecture, which we call the Interaction Transformer. Importantly, our model can be trained end-to-end, and runs in real-time. We validate our approach on two challenging real-world datasets: ATG4D [2] and nuScenes [3]. We show that our approach can outperform the state-of-the-art on both datasets. In particular, we significantly improve the social compliance between the estimated future trajectories, resulting in far fewer collisions between the predicted actors.

Current autonomous driving systems are composed of a perception system and a decision system. Both of them are divided into multiple subsystems built up with lots of human heuristics. An end-to-end approach might clean up the system and avoid huge efforts of human engineering, as well as obtain better performance with increasing data and computation resources. Compared to the decision system, the perception system is more suitable to be designed in an end-to-end framework, since it does not require online driving exploration. In this paper, we propose a novel end-to-end approach for autonomous driving perception. A latent space is introduced to capture all relevant features useful for perception, which is learned through sequential latent representation learning. The learned end-to-end perception model is able to solve the detection, tracking, localization and mapping problems altogether with only minimum human engineering efforts and without storing any maps online. The proposed method is evaluated in a realistic urban driving simulator, with both camera image and lidar point cloud as sensor inputs. The codes and videos of this work are available at our github repo† and project website‡.

3D Point cloud registration is still a very challenging topic due to the difficulty in finding the rigid transformation between two point clouds with partial correspondences, and it's even harder in the absence of any initial estimation information. In this paper, we present an end-to-end deep-learning based approach to resolve the point cloud registration problem. Firstly, the revised LPD-Net is introduced to extract features and aggregate them with the graph network. Secondly, the self-attention mechanism is utilized to enhance the structure information in the point cloud and the cross-attention mechanism is designed to enhance the corresponding information between the two input point clouds. Based on which, the virtual corresponding points can be generated by a soft pointer based method, and finally, the point cloud registration problem can be solved by implementing the SVD method. Comparison results in ModelNet40 dataset validate that the proposed approach reaches the state-of-the-art in point cloud registration tasks and experiment resutls in KITTI dataset validate the effectiveness of the proposed approach in real applications.

We demonstrate a reinforcement learning agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. The input LTL formulas have never been seen before, yet the network performs zero-shot generalization to satisfy them. This is a novel form of multi-task learning for RL agents where agents learn from one diverse set of tasks and generalize to a new set of diverse tasks. The formulation of the network enables this capacity to generalize. We demonstrate this ability in two domains. In a symbolic domain, the agent finds a sequence of letters that is accepted. In a Minecraft-like environment, the agent finds a sequence of actions that conform to the formula. While prior work could learn to execute one formula reliably given examples of that formula, we demonstrate how to encode all formulas reliably. This could form the basis of new multitask agents that discover sub-tasks and execute them without any additional training, as well as the agents which follow more complex linguistic commands. The structures required for this generalization are specific to LTL formulas, which opens up an interesting theoretical question: what structures are required in neural networks for zero-shot generalization to different logics?

Human-robot collaborative assembly consists of humans and automated robots, who cooperate with each other to accomplish complex assembly tasks, which are difficult for either humans or robots to accomplish alone. There has been some success in statistics-based and optimization-based approaches to realize human-robot collaboration. However, they usually need a set of complex modeling and setup efforts and the robots usually need to be programmed by a well-trained expert. In this paper, we take a new approach by introducing convolutional neural networks (CNN) into the teaching- learning-collaboration (TLC) model for collaborative assembly tasks. The proposed approach can alleviate the need for complex modeling and setup compared to the existing approaches. It can collect and automatically label the data from human demonstrations and then train a CNN-based robot assistance model to make the robot assist humans in the assembly process in real-time. We have experimentally verified our proposed approach on a human-robot collaborative assembly platform and the results suggest that the robot can successfully learn from human demonstrations to automatically generate right actions to assist human in accomplishing assembly tasks.

Real-world applications require light-weight, energy-efficient, fully autonomous robots. Yet, increasing autonomy is oftentimes synonymous with escalating computational requirements. It might thus be desirable to offload intensive computation—not only sensing and planning, but also low-level whole-body control—to remote servers in order to reduce on-board computational needs. Fifth Generation (5G) wireless cellular technology, with its low latency and high bandwidth capabilities, has the potential to unlock cloud-based high performance control of complex robots. However, state-of-the-art control algorithms for legged robots can only tolerate very low control delays, which even ultra-low latency 5G edge computing can sometimes fail to achieve. In this work, we investigate the problem of cloud-based whole-body control of legged robots over a 5G link. We propose a novel approach that consists of a standard optimization-based controller on the network edge and a local linear, approximately optimal controller that significantly reduces on-board computational needs while increasing robustness to delay and possible loss of communication. Simulation experiments on humanoid balancing and walking tasks that includes a realistic 5G communication model demonstrate significant improvement of the reliability of robot locomotion under jitter and delays likely to be experienced in 5G wireless links.

Gait emergence and adaptation in animals is unmatched in robotic systems. Animals can create and recover locomotive functions "on-the-fly" after an injury whereas locomotion controllers for robots lack robustness to morphological changes. In this work, we extend previous research on emergent interlimb coordination of legged robots based on coupled phase oscillators with force feedback terms. We investigate how the coupling weights between these phase oscillators can be extracted from the morphology with a fast and computationally lightweight method based on a combination of twitching and Hebbian learning to form sensor-motor maps. The coefficients of these maps create naturally scaled weights, which not only lead to robust gait limit cycles, but can also adapt to morphological modifications such as sensor loss and limb injuries within a few gait cycles. We demonstrate the approach on a robotic quadruped and hexapod.

The bodies of quadrupeds have very complex muscle-tendon structure. In particular, it is known that in the horse hindlimb, multiple joints in the leg are remarkably interlocked due to the muscle-tendon structure. Although the function of these interlocking mechanisms during standing has been investigated in the field of anatomy, the function related to the emergence of limb trajectory during dynamic walking has not been revealed. To investigate the role of the interlocking mechanism, we developed a robot model imitating the muscle-tendon arrangement and the dynamics of a horse hindlimb. In the walking experiment, the robot autonomously generated a limb trajectory with a smooth transition between the swing phase and the stance phase by simply swinging the hip joint with sinusoidal input. Moreover, we compared the joint angles between successful and failed walking. The compared results indicate that the extension of the fetlock joint after hoof touchdown plays the crucial role in emergence of a function of supporting body.

Quadrotor flight in cluttered, unknown environments is challenging due to the limited range of perception sensors, challenging obstacles, and limited onboard computation. In this work, we directly address these challenges by proposing an efficient, reactive planning approach. We introduce the Bitwise Trajectory Elimination (BiTE) algorithm for efficiently filtering out in-collision trajectories from a trajectory library by using bitwise operations. Then, we outline a full receding-horizon planning approach for quadrotor flight in unknown environments demonstrated at up to 50 Hz on an onboard computer. This approach is evaluated extensively in simulation and shown to collision check up to 4896 trajectories in under 20μs, which is the fastest collision checking time for a MAV planner, to the best of the authors' knowledge. Finally, we validate our planner in over 120 minutes of flights in forest-like and urban subterranean environments.

The ability to search for objects is a precondition for various robotic tasks. In this paper, we address the problem of finding objects in partially known indoor environments. Using the knowledge of the floor plan and the mapped objects, we consider object-object and object-room co-occurrences as prior information for identifying promising locations where an unmapped object can be present. We propose an efficient search strategy that determines the best pose of the robot based on the analysis of the candidate locations. We optimize the probability of finding the target object and the distance travelled through a cost function.To evaluate our method, several experiments in simulated and real-world environments were performed. The results show that the robot successfully finds the target object in the environment while covering only a small portion of the search space. The real-world experiments with the TurtleBot 2 mobile robot validate the proposed approach and demonstrate that the method performs well also in real environments.

Fast, aerial navigation in cluttered environments requires a suitable map representation for path planning. In this paper, we propose the use of an efficient, structured multiresolution representation that expands the sensor range of dense local grids for memory-constrained platforms. While similar data structures have been proposed, we avoid processing redundant occupancy information and use the organization of the grid to improve efficiency. By layering 3D circular buffers that double in resolution at each level, obstacles near the robot are represented at finer resolutions while coarse spatial information is maintained at greater distances. We also introduce a novel method for efficiently calculating the Euclidean distance transform on the multiresolution grid by leveraging its structure. Lastly, we utilize our proposed framework to demonstrate improved stereo camera-based MAV obstacle avoidance with an optimization-based planner in simulation.