ICRA 2021 论文列表

Naturally, finding joint trajectories for robotic manipulators involves competing optimization goals. On the one hand, the end-effector should move along a predictable and short path while on the other hand joint movement and acceleration should be kept to a minimum. Obstacles in the workspace or joint limits complicate the situation even further. Constructing a metric that makes undesired configurations more expensive to travel through, we equip the joint space with a notion of cost. The motion planning task then reduces to the problem of finding cheapest paths connecting two configurations – so-called geodesics. We show how to construct suitable metrics for a variety of typical scenarios and present an efficient algorithm for the computation of the corresponding geodesics. Our approach makes it very easy to balance different optimization goals and produces natural and smooth manipulator movement.

When faced with an execution failure, an intelligent robot should be able to identify the likely reasons for the failure and adapt its execution policy accordingly. This paper addresses the question of how to utilise knowledge about the execution process, expressed in terms of learned constraints, in order to direct the diagnosis and experience acquisition process. In particular, we present two methods for creating a synergy between failure diagnosis and execution model learning. We first propose a method for diagnosing execution failures of parameterised action execution models, which searches for action parameters that violate a learned precondition model. We then develop a strategy that uses the results of the diagnosis process for generating synthetic data that are more likely to lead to successful execution, thereby increasing the set of available experiences to learn from. The diagnosis and experience correction methods are evaluated for the problem of handle grasping, such that we experimentally demonstrate the effectiveness of the diagnosis algorithm and show that corrected failed experiences can contribute towards improving the execution success of a robot.

Modern navigation algorithms based on deep reinforcement learning (RL) show promising efficiency and robustness. However, most deep RL algorithms operate in a risk-neutral manner, making no special attempt to shield users from relatively rare but serious outcomes, even if such shielding might cause little loss of performance. Furthermore, such algorithms typically make no provisions to ensure safety in the presence of inaccuracies in the models on which they were trained, beyond adding a cost-of-collision and some domain randomization while training, in spite of the formidable complexity of the environments in which they operate. In this paper, we present a novel distributional RL algorithm that not only learns an uncertainty-aware policy, but can also change its risk measure without expensive fine-tuning or retraining. Our method shows superior performance and safety over baselines in partially- observed navigation tasks. We also demonstrate that agents trained using our method can adapt their policies to a wide range of risk measures at run-time.

Planning under uncertainty is a crucial capability for autonomous systems to operate reliably in uncertain and dynamic environments. The concern of safety becomes even more critical in healthcare settings where robots interact with human patients. In this paper, we propose a novel risk-aware planning framework to minimize the risk of falls by providing a patient with an assistive device. Our approach combines learning-based prediction with model-based control to plan for the fall prevention task. This provides advantages compared to end-to-end learning methods in which the robot's performance is limited to specific scenarios, or purely model-based approaches that use relatively simple function approximators and are prone to high modeling errors. We compare various risk metrics and the results from simulated scenarios show that using the proposed cost function, the robot can plan interventions to avoid high fall score events.

Deep Q learning (DQN) has enabled robot agents to accomplish vision based tasks that seemed out of reach. Despite recent success stories, there are still several sources of computational complexity that challenge the performance of DQN. We place the focus on vision manipulation tasks, where the correct action selection is often predicated on a small number of pixels. We observe that in some of these tasks DQN does not converge to the optimal Q function, and their values do not separate well optimal and suboptimal actions. In consequence, the policies obtained with DQN tend to be brittle and manifest a low success rate, especially in long horizon tasks. In this work we show the benefits of Reward Machines (RMs) for Deep Q learning (DQRM) in vision based robot manipulation tasks. Reward machines decompose the task at an abstract level, inform the agent about their current stage along task completion, and guide them via dense rewards. We show that RMs help DQN learn the optimal Q values in each abstract state. Their policies are more robust, manifest higher success rate, and are learned with fewer training steps compared with DQN. The benefits of RMs are more evident in long-horizon tasks, where we show that DQRM is able to learn good-quality policies with six times times fewer training steps than DQN, even when this is equipped with dense reward shaping.

This paper aims to study the reward-based policy exploration problem in a supervised learning approach and enable robots to form complex movement trajectories in challenging reward settings and search spaces. For this, the experience of the robot, which can be bootstrapped from demonstrated trajectories, is used to train a novel Neural Processes-based deep network that samples from its latent space and generates the required trajectories given desired rewards. Our framework can generate progressively improved trajectories by sampling them from high reward landscapes, increasing the reward gradually. Variational inference is used to create a stochastic latent space to sample varying trajectories in generating a population of trajectories given target rewards. We benefit from Evolutionary Strategies and propose a novel crossover operation, which is applied in the self-organized latent space of the individual policies, allowing blending of the individuals that might address different factors in the reward function. Using a number of tasks that require sequential reaching to multiple points or passing through gaps between objects, we showed that our method provides stable learning progress and significantly higher sample efficiency compared to a number of state-of-the-art robotic reinforcement learning methods. Finally, we show the real-world suitability of our method through real robot execution involving obstacle avoidance.

Accurate visual re-localization is very critical to many artificial intelligence applications, such as augmented reality, virtual reality, robotics and autonomous driving. To accomplish this task, we propose an integrated visual re-localization method called RLOCS by combining image retrieval, semantic consistency and geometry verification to achieve accurate estimations. The localization pipeline is designed as a coarse-to-fine paradigm. In the retrieval part, we cascade the architecture of ResNet101-GeM-ArcFace and employ DBSCAN followed by spatial verification to obtain a better initial coarse pose. We design a module called observation constraints, which combines geometry information and semantic consistency for filtering outliers. Comprehensive experiments are conducted on open datasets, including retrieval on R-Oxford5k and R-Paris6k, semantic segmentation on Cityscapes, localization on Aachen Day-Night and InLoc. By creatively modifying separate modules in the total pipeline, our method achieves many performance improvements on the challenging localization benchmarks.

The success of deep reinforcement learning (RL) and imitation learning (IL) in vision-based robotic manipulation typically hinges on the expense of large scale data collection. With simulation, data to train a policy can be collected efficiently at scale, but the visual gap between sim and real makes deployment in the real world difficult. We introduce RetinaGAN, a generative adversarial network (GAN) approach to adapt simulated images to realistic ones with object-detection consistency. RetinaGAN is trained in an unsupervised manner without task loss dependencies, and preserves general object structure and texture in adapted images. We evaluate our method on three real world tasks: grasping, pushing, and door opening. RetinaGAN improves upon the performance of prior sim-to-real methods for RL-based object instance grasping and continues to be effective even in the limited data regime. When applied to a pushing task in a similar visual domain, RetinaGAN demonstrates transfer with no additional real data requirements. We also show our method bridges the visual gap for a novel door opening task using imitation learning in a new visual domain. Visit the project website at retinagan.github.io

Self-healing function is a promising approach for damage management of high-load robot applications such as legged robots. Although the function is getting major in soft robotics, its application to life-sized "stiff" robots is of relatively minor interest. Although the authors have devised several self-healing tensile modules for tendon-driven robots, the design guideline to satisfy the large load endurance and large stroke is still unclear. The paper focuses on the parametric design for unleaked liquid-assisted healing of low melting point alloy structure. The method was validated with a benchtop module test. Moreover, the module enabled tendon-driven monopod testbed to perform squat motion three times after the landing impact fracture and the self-healing sequence, which was never accomplished.

This work presents the design and autonomous navigation policy of the Resilient Micro Flyer, a new type of collision-tolerant robot tailored to fly through extremely confined environments and manhole-sized tubes. The robot maintains a low weight (<500g) and implements a combined rigid-compliant design through the integration of elastic flaps around its stiff collision-tolerant frame. These passive flaps ensure compliant collisions, contact sensing and smooth navigation in contact with the environment. Focusing on resilient autonomy, capable of running on resource-constrained hardware, we demonstrate the beneficial role of compliant collisions for the reliability of the onboard visual-inertial odometry and propose a safe navigation policy that exploits both collision-avoidance using lightweight time-of-flight sensing and adaptive control in response to collisions. The robot further realizes an explicit manhole navigation mode that exploits the direct mechanical feedback provided by the flaps and a special navigation strategy to self-align inside manholes with non-straight geometry. Comprehensive experimental studies are presented to evaluate, both individually and as a whole, how resilience is achieved based on the robot design and its navigation scheme.

This paper proposes an efficient, fully convolutional neural network to generate robotic grasps by using 300×300 depth images as input. Specifically, a residual squeeze-and-excitation network (RSEN) is introduced for deep feature extraction. Following the RSEN block, a multi-scale spatial pyramid module (MSSPM) is developed to obtain multi-scale contextual information. The outputs of each RSEN block and MSSPM are combined as inputs for hierarchical feature fusion. Then, the fused global features are upsampled to perform pixel-wise learning for grasping pose estimation. The experimental results on Cornell and Jacquard grasping datasets indicate that the proposed method has a fast inference speed of 5ms while achieving high grasp detection accuracy of 96.4% and 94.8% on Cornell and Jacquard, respectively, which strikes a balance between accuracy and running speed. Our method also gets a 90% physical grasp success rate with a UR5 robot arm.

A majority of microrobots are constructed using compliant materials that are difficult to model analytically, limiting the utility of traditional model-based controllers. Challenges in data collection on microrobots and large errors between simulated models and real robots make current model-based learning and sim-to-real transfer methods difficult to apply. We propose a novel framework residual model learning (RML) that leverages approximate models to substantially reduce the sample complexity associated with learning an accurate robot model. We show that using RML, we can learn a model of the Harvard Ambulatory MicroRobot (HAMR) using just 12 seconds of passively collected interaction data. The learned model is accurate enough to be leveraged as "proxy-simulator" for learning walking and turning behaviors using model-free reinforcement learning algorithms. RML provides a general framework for learning from extremely small amounts of interaction data, and our experiments with HAMR clearly demonstrate that RML substantially outperforms existing techniques.

Reinforcement Learning (RL) algorithms can in principle acquire complex robotic skills by learning from large amounts of data in the real world, collected via trial and error. However, most RL algorithms use a carefully engineered setup in order to collect data, requiring human supervision and intervention to provide episodic resets. This is particularly evident in challenging robotics problems, such as dexterous manipulation. To make data collection scalable, such applications require reset-free algorithms that are able to learn autonomously, without explicit instrumentation or human intervention. Most prior work in this area handles single-task learning. However, we might also want robots that can perform large repertoires of skills. At first, this would appear to only make the problem harder. However, the key observation we make in this work is that an appropriately chosen multi-task RL setting actually alleviates the reset-free learning challenge, with minimal additional machinery required. In effect, solving a multi-task problem can directly solve the reset-free problem since different combinations of tasks can serve to perform resets for other tasks. By learning multiple tasks together and appropriately sequencing them, we can effectively learn all of the tasks together reset-free. This type of multi-task learning can effectively scale reset-free learning schemes to much more complex problems, as we demonstrate in our experiments. We propose a simple scheme for multi-task learning that tackles the reset-free learning problem, and show its effectiveness at learning to solve complex dexterous manipulation tasks in both hardware and simulation without any explicit resets. This work shows the ability to learn in-hand manipulation behaviors in the real world with RL without any human intervention.

Projecting high-dimensional environment observations into lower-dimensional structured representations can considerably improve data-efficiency for reinforcement learning in domains with limited data such as robotics. Can a single generally useful representation be found? In order to answer this question, it is important to understand how the representation will be used by the agent and what properties such a good representation should have. In this paper we systematically evaluate a number of common learnt and hand-engineered representations in the context of three robotics tasks: lifting, stacking and pushing of 3D blocks. The representations are evaluated in two use-cases: as input to the agent, or as a source of auxiliary tasks. Furthermore, the value of each representation is evaluated in terms of three properties: dimensionality, observability and disentanglement. We can significantly improve performance in both use-cases and demonstrate that some representations can perform commensurate to simulator states as agent inputs. Finally, our results challenge common intuitions by demonstrating that: 1) dimensionality strongly matters for task generation, but is negligible for inputs, 2) observability of task-relevant aspects mostly affects the input representation use-case, and 3) disentanglement leads to better auxiliary tasks, but has only limited benefits for input representations. This work serves as a step towards a more systematic understanding of what makes a good representation for control in robotics, enabling practitioners to make more informed choices for developing new learned or hand-engineered representations.

This paper presents replay overshooting (RO), an algorithm that uses properties of the extended Kalman filter (EKF) to learn nonlinear stochastic latent dynamics models suitable for long-horizon prediction. We build upon overshooting methods used to train other prediction models and recover a novel variational learning objective. Further, we use RO to extend another objective that acts as a surrogate for the true log-likelihood, and show that this objective empirically yields better models than the variational one. We evaluate RO on two tasks: prediction of synthetic video frames of a swinging motorized pendulum and prediction of the planar position of various objects being pushed by a real manipulator (MIT Push Dataset). Our model outperforms several other prediction models on both quantitative and qualitative metrics.

Brain needle intervention is a specific diagnosis and therapy procedure in brain disorders, such as brain tumors and Parkinson’s disease. Preoperative needle path planning is a vital step to guarantee the patient’s safety and reduce lesions. For positioning accuracy in the CT/MRI environment, we have developed a novel needle intervention robot in our previous work. Because the robot is currently designed for the rigid needle, the task of preoperative path-planning is to search for an optimal Remote Center of Motion (RCM) for needle insertion. Therefore, this work proposes an RCM recommendation system using deep reinforcement learning. Considering the robot kinematics, this system takes the following criteria/constraints into consideration: clinical obstacle (blood vessels, tissues) avoidance (COA), mechanically inverse kinematics (MIK) and mechanically less motion (MLM) for the robot. We design a reward function to combine the above three criteria based on their corresponding importance level and utilize proximal policy optimization (PPO) as the main agent of reinforcement learning (RL). RL methods are proved to be competent in searching the RCM, which satisfies the above criteria simultaneously. On the one hand, the results present that RL agents obtain the success rate of finishing the designed task at 93%, which has reached the human level in the tests. On the other hand, the RL agents have the remarkable capability of combining more complex criteria/constraints in future work.

In this paper, a novel navigation learning method in continuous action space among crowds based on relational graph is proposed which can be directly deployed on differential-drive mobile robots without any change. More specifically, in order to increase generalization ability in crowd sizes, Graph Convolutional Network (GCN) is at first adopted to extract the relationships between robot and pedestrians. Then the relation features are further utilized as the inputs of the pedestrian state prediction network, the actor network, and the critic network. To efficiently and safely learn the navigation policy, all networks are pretrained through imitating ORCA which is a state-of-the-art algorithm in crowd navigation, and then a model-based reinforcement learning (RL) method which combines the model prediction and the clipped advantage-weighted regression is proposed to finetune the networks. Finally, simulation experiments are performed and it’s verified that the proposed learning method performs significantly better than ORCA and the other state-of-the-art RL methods.

Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recognition accuracies. In this regard, we propose a novel online approach of multi-modal relational graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information through interactive message propagation in the latent feature space. In specific, we first extract embeddings from video and kinematics sequences with temporal convolutional networks and LSTM units. Next, we identify multi-relations in these multi-modal embeddings and leverage them through a hierarchical relational graph learning module. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset, outperforming current uni-modal and multi-modal methods on both suturing and knot typing tasks. Furthermore, we validated our method on in-house visual-kinematics datasets collected with da Vinci Research Kit (dVRK) platforms in two centers, with consistent promising performance achieved. Our code and data are released at: https://www.cse.cuhk.edu.hk/~yhlong/mrgnet.html.

Developing robust walking controllers for bipedal robots is a challenging endeavor. Traditional model-based locomotion controllers require simplifying assumptions and careful modelling; any small errors can result in unstable control. To address these challenges for bipedal locomotion, we present a model-free reinforcement learning framework for training robust locomotion policies in simulation, which can then be transferred to a real bipedal Cassie robot. To facilitate sim-to-real transfer, domain randomization is used to encourage the policies to learn behaviors that are robust across variations in system dynamics. The learned policies enable Cassie to perform a set of diverse and dynamic behaviors, while also being more robust than traditional controllers and prior learning-based methods that use residual control. We demonstrate this on versatile walking behaviors such as tracking a target walking velocity, walking height, and turning yaw. (Video1)