IROS 2020 论文列表

In this paper, we propose the "multi-vision hand", in which a number of small high-speed cameras are mounted on the robot hand of a common 7 degrees-of-freedom robot. Also, we propose visual-servoing control by using a multi-vision system that combines the multi-vision hand and external fixed high-speed cameras. The target task was ball catching motion, which requires high-speed operation. In the proposed catching control, the catch position of the ball, which is estimated by the external fixed high-speed cameras, is corrected by the multivision hand in real-time. In experiments, the catch operation was successfully implemented by correcting the catch position, thus confirming the effectiveness of the multi-vision hand system.

For the time varying optimization problem, the tracking error cannot converge to zero at the finite time because of the optimal solution changing over time. This paper proposes a novel varying parameter recurrent neural network (VPRNN) based hierarchical optimization of a 7-DoF surgical manipulator for Robot-Assisted Minimally Invasive Surgery (RAMIS), which guarantees task tracking, Remote Center of Motion (RCM) and manipulability index optimization. A theoretically grounded hierarchical optimization framework based is introduced to control multiple tasks based on their priority. Finally, the effectiveness of the proposed control strategy is demonstrated with both simulation and experimental results. The results show that the proposed VPRNN-based method can optimal three tasks at the same time and have better performance than previous work.

Behavioral decision making is an important aspect of autonomous vehicles (AV). In this work, we propose a behavior planning structure based on hierarchical reinforcement learning (HRL) which is capable of performing autonomous vehicle planning tasks in simulated environments with multiple sub-goals. In this hierarchical structure, the network is capable of 1) learning one task with multiple sub-goals simultaneously; 2) extracting attentions of states according to changing sub-goals during the learning process; 3) reusing the well-trained network of sub-goals for other tasks with the same sub-goals. A hybrid reward mechanism is designed for different hierarchical layers in the proposed HRL structure. Compared to traditional RL methods, our algorithm is more sample-efficient, since its modular design allows reusing the policies of sub-goals across similar tasks for various transportation scenarios. The results show that the proposed method converges to an optimal policy faster than traditional RL methods.

For robot swarms operating on complex missions in an uncertain environment, it is important that the decision-making algorithm considers both heterogeneity and uncertainty. This paper presents a stochastic programming framework for the vehicle routing problem with stochastic travel energy costs and heterogeneous vehicles and tasks. We represent the heterogeneity as linear constraints, estimate the uncertain energy cost through Gaussian process regression, formulate this stochasticity as chance constraints or stochastic recourse costs, and then solve the stochastic programs using branch and cut algorithms to minimize the expected energy cost. The performance and practicality are demonstrated through extensive computational experiments and a practical test case.

The majority of learning-based semantic segmentation methods are optimized for daytime scenarios and favorable lighting conditions. Real-world driving scenarios, however, entail adverse environmental conditions such as nighttime illumination or glare which remain a challenge for existing approaches. In this work, we propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. To this end, besides RGB images, we leverage thermal images, making our network significantly more robust. We avoid the expensive annotation of nighttime images by leveraging an existing daytime RGB-dataset and propose a teacher-student training approach that transfers the dataset's knowledge to the nighttime domain. We further adopt a domain adaptation method to align the learned feature spaces across the domains and propose a novel two-stage training scheme. Furthermore, due to a lack of thermal data for autonomous driving, we present a new dataset comprising over 20,000 time-synchronized and aligned RGB-thermal image pairs. In this context, we also present a novel target-less calibration method that allows for automatic robust extrinsic and intrinsic thermal camera calibration. Among others, we use our new dataset to show state-of-the-art results for nighttime semantic segmentation.

Continuous robot operation in extreme scenarios such as underground mines or sewers is difficult because exteroceptive sensors may fail due to fog, darkness, dirt or malfunction. So as to enable autonomous navigation in these kinds of situations, we have developed a type of proprioceptive localization which exploits the foot contacts made by a quadruped robot to localize against a prior map of an environment, without the help of any camera or LIDAR sensor. The proposed method enables the robot to accurately re-localize itself after making a sequence of contact events over a terrain feature. The method is based on Sequential Monte Carlo and can support both 2.5D and 3D prior map representations. We have tested the approach online and onboard the ANYmal quadruped robot in two different scenarios: the traversal of a custom built wooden terrain course and a wall probing and following task. In both scenarios, the robot is able to effectively achieve a localization match and to execute a desired pre-planned path. The method keeps the localization error down to 10 cm on feature rich terrain by only using its feet, kinematic and inertial sensing.

Humans learn about object properties using multiple modes of perception. Recent advances show that robots can use non-visual sensory modalities (i.e., haptic and tactile sensory data) coupled with exploratory behaviors (i.e., grasping, lifting, pushing, dropping, etc.) for learning objects' properties such as shape, weight, material and affordances. However, non-visual sensory representations cannot be easily transferred from one robot to another, as different robots have different bodies and sensors. Therefore, each robot needs to learn its task-specific sensory models from scratch. To address this challenge, we propose a framework for knowledge transfer using kernel manifold alignment (KEMA) that enables source robots to transfer haptic knowledge about objects to a target robot. The idea behind our approach is to learn a common latent space from multiple robots' feature spaces produced by respective sensory data while interacting with objects. To test the method, we used a dataset in which 3 simulated robots interacted with 25 objects and showed that our framework speeds up haptic object recognition and allows novel object recognition.

In this paper, we study the problem of computing grasping forces for quasi-static manipulation of large and heavy objects, by exploiting object-environment contacts. We present a general formulation of this problem as a Second-Order Cone Program (SOCP) that considers (i) contact friction constraints at the object-manipulator contacts and object-environment contacts, (ii) force/moment equilibrium constraints, and (iii) manipulator joint torque constraints. The SOCP formulation implies that the optimal grasping forces for manipulating objects with the help of the environment can be computed efficiently. Different optimization objectives like minimizing contact forces at the object-manipulator contacts or minimizing joint torques of manipulators can be considered. We evaluated our method by simulations of single-handed and dual-handed manipulation scenarios.

High-definition (HD) maps are emerging as an essential tool for autonomous driving since they provide high-precision semantic information about the physical environment. To function as a reliable source of map information, HD maps must be constantly updated with changes that occur to the state of the road. In this paper, we propose a novel framework for HD map change detection that can be used to maintain an up-to-date HD map. More specifically, we design our HD map change detection algorithm based on deep metric learning, providing a unified framework that directly maps an input image to estimated probabilities of HD map changes. To reduce the discrepancy between input domains, i.e., camera image and HD map, we propose an effective learning scheme for metric space based on adversarial learning. Finally, we augment our framework with a pixel-level local change detector that specifies the region of changes in the image. We verify the effectiveness of our framework by evaluating it on a city-scale urban HD map dataset. Experimental results show that our method can robustly detect changes against noises due to dynamic objects and error in vehicle poses.

To fluently collaborate with people, robots need the ability to recognize human activities accurately. Although modern robots are equipped with various sensors, robust human activity recognition (HAR) still remains a challenging task for robots due to difficulties related to multimodal data fusion. To address these challenges, in this work, we introduce a deep neural network-based multimodal HAR algorithm, HAMLET. HAMLET incorporates a hierarchical architecture, where the lower layer encodes spatio-temporal features from unimodal data by adopting a multi-head self-attention mechanism. We develop a novel multimodal attention mechanism for disentangling and fusing the salient unimodal features to compute the multimodal features in the upper layer. Finally, multimodal features are used in a fully connect neural-network to recognize human activities. We evaluated our algorithm by comparing its performance to several state-of-the-art activity recognition algorithms on three human activity datasets. The results suggest that HAMLET outperformed all other evaluated baselines across all datasets and metrics tested, with the highest top-1 accuracy of 95.12% and 97.45% on the UTD-MHAD [1] and the UT-Kinect [2] datasets respectively, and F1-score of 81.52% on the UCSD-MIT [3] dataset. We further visualize the unimodal and multimodal attention maps, which provide us with a tool to interpret the impact of attention mechanisms concerning HAR.

In haptic time delayed teleoperation as the time delay from the communication channel increases, teleoperation system stability and performance degrade. To increase performance and provide better stability margins, various estimation methods and observers have been implemented in literature to more accurately capture the force exerted by the remote system. Previously, solutions focused on environment force estimation methods that primarily rely on linearization of the Hunt-Crossley (HC) contact model, which has limiting assumptions for use. This work addresses the shortcomings of the aforementioned methods by investigating alternative HC parameter estimation techniques. A new application of Chebyshev polynomial approximation for adaptive parameter estimation of the HC model is proposed. This approximation is compared to current linearization methods as well as nonlinear estimation methods that are not well covered in literature. Moreover, the Chebyshev approximation is used in a new estimation approach that provides control via backstepping with adaptive parameter estimation using Lyapunov methods. This method reduces excitation requirements by using nonlinear swapping and the data accumulation concept to guarantee parameter convergence. A simulated full teleoperation system with time delay demonstrates the effectiveness of this approach.

Localization using ground texture images recorded with a downward-facing camera is a promising approach to achieve reliable high-accuracy vehicle positioning. A common way to accomplish the task is to focus on prominent features of the ground texture such as stones and cracks. Our results indicate that with an approximately known camera pose it is sufficient to use arbitrary ground regions, i.e. extracting features at random positions without significant loss in localization performance. Additionally, we propose a real-time capable CPU-only localization method based on this idea, and suggest possible improvements for further research.

Despite more than three decades of grasping research, many tools in our everyday life still pose a serious challenge for a robotic hand to grip. The level of dexterity for such a maneuver is surprisingly "high" that its execution may require a combination of closed loop controls and finger gaits. This paper studies the task of an anthropomorphic hand driven by a robotic arm to pick up and firmly hold a kitchen knife initially resting on the cutting board. In the first phase, the hand grasps the knife's handle at two antipodal points and then pivots it about the knife's point in contact with the board to leverage the latter's support. Desired contact forces exerted by the two holding soft fingers are calculated and used for dynamic control of both the hand and the arm. In the second phase, a sequence of gaits for all the five fingers is performed quasi-statically to reach a power grasp on the knife's handle, which remains still during the period. Simulation has been performed using models of the Shadow Hand and the UR10 Arm.

Data-efficient domain adaptation with only a few labelled data is desired for many robotic applications, e.g., in grasping detection, the inference skill learned from a grasping dataset is not universal enough to directly apply on various other daily/industrial applications. This paper presents an approach enabling the easy domain adaptation through a novel grasping detection network with confidence-driven semi-supervised learning, where these two components deeply interact with each other. The proposed grasping detection network specially provides a prediction uncertainty estimation mechanism by leveraging on Feature Pyramid Network (FPN), and the mean-teacher semi-supervised learning utilizes such uncertainty information to emphasizing the consistency loss only for those unlabelled data with high confidence, which we referred it as the confidence-driven mean teacher. This approach largely prevents the student model to learn the incorrect/harmful information from the consistency loss, which speeds up the learning progress and improves the model accuracy. Our results show that the proposed network can achieve high success rate on the Cornell grasping dataset, and for domain adaptation with very limited data, the confidence- driven mean teacher outperforms the original mean teacher and direct training by more than 10% in evaluation loss especially for avoiding the overfitting and model diverging.

We study the hierarchical knowledge transfer problem using a cloth-folding task, wherein the agent is first given a set of human demonstrations in the virtual world using an Oculus Headset, and later transferred and validated on a physical Baxter robot. We argue that such an intricate robot task transfer across different embodiments is only realizable if an abstract and hierarchical knowledge representation is formed to facilitate the process, in contrast to prior literature of sim2real in a reinforcement learning setting. Specifically, the knowledge in both the virtual and physical worlds are measured by information entropy built on top of a graph-based representation, so that the problem of task transfer becomes the minimization of the relative entropy between the two worlds. An And-Or-Graph (AOG) is introduced to represent the knowledge, induced from the human demonstrations performed across six virtual scenarios inside the Virtual Reality (VR). During the transfer, the success of a physical Baxter robot platform across all six tasks demonstrates the efficacy of the graph-based hierarchical knowledge representation.

Effective communication is key to successful, decentralized, multi-robot path planning. Yet, it is far from obvious what information is crucial to the task at hand, and how and when it must be shared among robots. To side-step these issues and move beyond hand-crafted heuristics, we propose a combined model that automatically synthesizes local communication and decision-making policies for robots navigating in constrained workspaces. Our architecture is composed of a convolutional neural network (CNN) that extracts adequate features from local observations, and a graph neural network (GNN) that communicates these features among robots. We train the model to imitate an expert algorithm, and use the resulting model online in decentralized planning involving only local communication and local observations. We evaluate our method in simulations by navigating teams of robots to their destinations in 2D cluttered workspaces. We measure the success rates and sum of costs over the planned paths. The results show a performance close to that of our expert algorithm, demonstrating the validity of our approach. In particular, we show our model's capability to generalize to previously unseen cases (involving larger environments and larger robot teams).

We present an algorithm for the realtime matching of wireframe extractions in pairs of images. Here we treat extracted wireframes as graphs and propose a simplified Graduated Assignment algorithm to use with this problem. Using this algorithm we achieve a 30% accuracy improvement over the baseline method. We show that, for this problem, the simplified Graduated Assignment algorithm can achieve realtime performance without a significant drop in accuracy as compared to the standard Graduated Assignment algorithm. We further demonstrate a method of utilizing this simplified Graduated Assignment algorithm for achieving a similar real-time improvement in the matching quality of standard features without wireframe detection.

It has been claimed that a main advantage of cognitive architectures (compared to other types of specialized robotic architectures) is that they are task-general and can thus learn to perform any task as long as they have the right perceptual and action primitives. In this paper, we provide empirical evidence for this claim by directly comparing a high-performing custom robotic architecture developed for the standardized robotic "FetchIt!" challenge task to a hybrid cognitive robotic architecture that allows for online one-shot task learning and task modifications through natural language instructions. The results show that there is no disadvantage of running the hybrid architecture (i.e., no significant difference in overall performance or computational overhead compared to the custom architecture) while adding the flexibility of online one-shot task instruction and modification not available in the custom architecture.

In this paper we address variable admittance control for human-robot physical interaction in manual guidance applications. In the proposed solution, the parameters of the admittance filter can change not only as a function of the current state of motion (i.e. whether the human guiding the robot ia accelerating or decelerating) but also with reference to a predefined goal position. The human is in fact gently guided towards the goal along some curved paths, where the damping is conveniently scaled in order to accommodate the motion towards the goal position. The algorithm also allows the human to reach goals that he/she cannot directly see because for example the transported object is bulky and obstructs the worker view. The performance of the proposed controller are evaluated by means of point to point cooperative motions with multiple volunteers using an ABB IRB140 robot.