IROS 2024 论文列表

In transformer-based methods for point cloud instance segmentation, bipartite matching is used to establish one-to-one correspondences between predictions and ground truths. However, in early training stages, matches can be unstable and inconsistent between epochs, requiring the model to frequently adjust its learning path, thus reducing the quality of model convergence. To address this challenge, we propose the contrastive mask denoising transformer for 3D instance segmentation, which utilizes a mask denoising module to guide the model towards a more stable optimization path in early training stages. Furthermore, we introduce a multi-pattern-aware query selection module to assist the model learn multiple patterns at one position such that clustered objects can be discerned. In addition, the proposed modules are "plug and play", which can easily be integrated into transformer-based architectures. Experimental results on ScanNetv2 dataset show that the proposed modules improve the performance of multiple pipelines, notably achieving +1.0 mAP on the main pipeline.

Shape sensing of medical continuum robots is important both for closed-loop control as well as for enabling the clinician to visualize the robot inside the body. There is a need for inexpensive, but accurate shape sensing technologies. This paper proposes the use of magnetic ball chains as a means of generating shape-specific magnetic fields that can be detected by an external array of Hall effect sensors. Such a ball chain, encased in a flexible polymer sleeve, could be inserted inside the lumen of any continuum robot to provide real-time shape feedback. The sleeve could be removed, as needed, during the procedure to enable use of the entire lumen. To investigate this approach, a shape-sensing model for a steerable catheter tip is derived and an observability and sensitivity analysis are presented. Experiments show maximum estimation errors of 7.1% and mean of 2.9% of the tip position with respect to total length.

In the general field of collaborative robotics, one of the topics of greatest interest to the scientific community is the ability to learn to perform certain actions by imitating humans. If we think about humans, when someone teaches us how to perform a certain action, we often need to be shown just one time how to do it. Likewise, we believe that robotics should follow this line, using models that do not involve the capture of huge data sets or exhaustive training. Furthermore, while general models can typically be pretrained offline, the robot must quickly adapt to new knowledge without requiring an expensive retraining process. In this article we present a flexible neural learning architecture that allows a robot to learn how-to pick-up a given object just by watching how a human does it. Then, the robot will be able to pick up the current object, or other objects previously learned, anywhere in the work field, with a simple audible indication from the user. This is achieved based on continuous incremental learning techniques and generic segmentation networks integrated with Siamese network models according to the recently proposed CP-CVV method. Results are presented for the success rate in grasping a varied set of objects.

Domain Randomization (DR) is commonly used for sim2real transfer of reinforcement learning (RL) policies in robotics. Most DR approaches require a simulator with a fixed set of tunable parameters from the start of the training, from which the parameters are randomized simultaneously to train a robust model for use in the real world. However, the combined randomization of many parameters increases the task difficulty and might result in sub-optimal policies. To address this problem and to provide a more flexible training process, we propose Continual Domain Randomization (CDR) for RL that combines domain randomization with continual learning to enable sequential training in simulation on a subset of randomization parameters at a time. Starting from a model trained in a non-randomized simulation where the task is easier to solve, the model is trained on a sequence of randomizations, and continual learning is employed to remember the effects of previous randomizations. Our robotic reaching and grasping tasks experiments show that the model trained in this fashion learns effectively in simulation and performs robustly on the real robot while matching or outperforming baselines that employ combined randomization or sequential randomization without continual learning. Our code and videos are available at https://continual-dr.github.io/.

How does the person in the bounding box feel?" Achieving human-level recognition of the apparent emotion of a person in real world situations remains an unsolved task in computer vision. Facial expressions are not enough: body pose, contextual knowledge, and commonsense reasoning all contribute to how humans perform this emotional theory of mind task. In this paper, we examine two major approaches enabled by recent large vision language models: 1) image captioning followed by a language-only LLM, and 2) vision language models, under zero-shot and fine-tuned setups. We evaluate the methods on the Emotions in Context (EMOTIC) dataset and demonstrate that a vision language model, fine-tuned even on a small dataset, can significantly outperform traditional baselines. The results of this work aim to help robots and agents perform emotionally sensitive decision-making and interaction in the future.

Bounded rational agents often make decisions by evaluating a finite selection of choices, typically derived from a reference point termed the ‘default policy,’ based on previous experience. However, the inherent rigidity of the static default policy presents significant challenges for agents when operating in unknown environment, that are not included in agent’s prior knowledge. In this work, we introduce a context-generative default policy that leverages the region observed by the robot to predict unobserved part of the environment, thereby enabling the robot to adaptively adjust its default policy based on both the actual observed map and the imagined unobserved map. Furthermore, the adaptive nature of the bounded rationality framework enables the robot to manage unreliable or incorrect imaginations by selectively sampling a few trajectories in the vicinity of the default policy. Our approach utilizes a diffusion model for map prediction and a sampling-based planning with B-spline trajectory optimization to generate the default policy. Extensive evaluations reveal that the context-generative policy outperforms the baseline methods in identifying and avoiding unseen obstacles. Additionally, real-world experiments conducted with the Crazyflie drones demonstrate the adaptability of our proposed method, even when acting in environments outside the domain of the training distribution.

Effective localization is crucial for the reliable operation of autonomous delivery robots. This paper introduces ConLocGAN, a novel context-aware GAN, addressing challenges in Lidar-based localization. Our approach employs a two-step process, integrating image retrieval with Lidar-based localization. ConLocGAN extracts robust global descriptors for coarse pose estimator, which acts as a precursor for Lidar-based pose refinement. The discriminator in ConLocGAN identifies differences in images of the same scene under diverse conditions at the feature level. This information is then utilized to enhance localization-specific feature extraction by the generator in a self-supervised setting. Additionally, we present a simple data collection pipeline that is seamlessly integrated into routine robot operations. Using heatmaps for visualization, we demonstrate that our network learns robust descriptors by prioritizing static components of the scene while effectively disregarding environmental changes such as illumination and weather, as well as dynamic objects like people and vehicles. We further validate our method on the challenging CMU seasons dataset, where it outperforms state-of-the-art retrieval-based methods in coarse pose estimation.

Existing conversational robots are mostly reactive in that the interactions are usually initiated by the users. With the knowledge of the environmental context such as people’s daily activities, robots can be more intelligent and proactive. In this paper, we proposed a context-aware conversation adaptation system (CACAS) for human-robot interaction (HRI). First, a context recognition module and a language processing module are developed to obtain the context information, user intent and slots, which become part of the state. Second, a reinforcement learning algorithm is developed to train an initial policy with a simulated user. User feedback data is collected through HRI using the initial policy. Third, a policy combining the reinforcement learning-based policy with the neural network-based policy is adapted based on the user feedback. We conducted both simulated user tests and real human subject tests to evaluate the proposed system. The results show that CACAS achieved a success rate of 85% in the real human subject test and 87.5% of participants were satisfied with the adaptation results. For the simulation test, CACAS had the highest success rate compared with the baseline methods.

This paper presents a novel method for detecting and estimating contact forces only from human motions using machine learning techniques. Knowing the location of the contacts with the environment and the magnitude of the exerted force is critical for dynamic human motion analysis. However, their annotation is usually made manually from captured motion data especially in case of multiple contacts even if the data includes force measurement. Moreover, most existing human motion datasets do not include contact force. To overcome these bottlenecks, we introduce a network that leverages vector-quantized variational autoencoder (VQ-VAE) and self-attention that learns a small set of discrete feature values representing various contact states. These feature values, called contact codes, allow human motions to be converted to contact states and resulting forces. By applying an optimization for contact estimation with a reduced set of manual annotations, the existence of contacts can be automatically determined, which is essential information for dynamic analysis. We validated the effectiveness and potential usefulness of the proposed method with a human walking gait dataset, by converting the human motions into contact sequences and forces and applying the estimated contacts to dynamic motion analysis.

Robot-to-human object handover is an important step in many human robot collaboration tasks. A successful handover requires the robot to maintain a stable grasp on the object while making sure the human receives the object in a natural and easy-to-use manner. We propose ContactHandover, a robot to human handover system that consists of two phases: a contact-guided grasping phase and an object delivery phase. During the grasping phase, ContactHandover predicts both 6DoF robot grasp poses and a 3D affordance map of human contact points on the object. The robot grasp poses are reranked by penalizing those that block human contact points, and the robot executes the highest ranking grasp. During the delivery phase, the robot end effector pose is computed by maximizing human contact points close to the human while minimizing the human arm joint torques and displacements. We evaluate our system on 27 diverse household objects and show that our system achieves better visibility and reachability of human contacts to the receiver compared to several baselines. More results can be found on the project website.

Dexterous in-hand manipulation is an essential skill of production and life. However, the highly stiff and mutable nature of contacts limits real-time contact detection and inference, degrading the performance of model-based methods. Inspired by recent advances in contact-rich locomotion and manipulation, this paper proposes a novel model-based approach to control dexterous in-hand manipulation and overcome the current limitations. The proposed approach has an attractive feature, which allows the robot to robustly perform long-horizon in-hand manipulation without predefined contact sequences or separate planning procedures. Specifically, we design a high-level contact-implicit model predictive controller to generate real-time contact plans executed by the low-level tracking controller. Compared to other model-based methods, such a long-horizon feature enables replanning and robust execution of contact-rich motions to achieve large displacements in-hand manipulation more efficiently; Compared to existing learning-based methods, the proposed approach achieves dexterity and also generalizes to different objects without any pre-training. Detailed simulations and ablation studies demonstrate the efficiency and effectiveness of our method. It runs at 20Hz on the 23-degree-of-freedom, long-horizon, in-hand object rotation task.

The complex ways in which humans utilize their bodies in sports and martial arts are remarkable, and human motion analysis is one of the most effective tools for robot body design and control. On the other hand, motion analysis is not easy, and it is difficult to measure complex body motions in detail due to the influence of numerous muscles and soft tissues, mainly ligaments. In response, various musculoskeletal simulators have been developed and applied to motion analysis and robotics. However, none of them reproduce the ligaments but only the muscles, nor do they focus on the shoulder complex, including the clavicle and scapula, which is one of the most complex parts of the body. Therefore, in this study, a detailed simulation model of the shoulder complex including ligaments is constructed. The model will mimic not only the skeletal structure and muscle arrangement but also the ligament arrangement and maximum muscle strength. Through model predictive control based on the constructed simulation, we confirmed that the ligaments contribute to joint stabilization in the first movement and that the proper distribution of maximum muscle force contributes to the equalization of the load on each muscle, demonstrating the effectiveness of this simulation.

In this paper, we propose a robot skill-learning method that facilitates fast adaption to new tasks online. Our method is based on a hybrid learning from demonstration and reinforcement learning approach, which seeds learning with a compact and structured skill model, leading to efficient and stable behaviours. To facilitate fast skill adaption, we propose a bootstrapped learning framework that learns a policy for adapting a skill model across a wide range of initial conditions in simulation. This policy is then used to bootstrap a refinement process that quickly adapts the learnt skill model to new initial conditions in a few learning iterations. Our refined skill model is designed to be deployable on hardware and can correct for discrepancies between the simulation and the real world. Furthermore, we propose a novel method for constraining policy exploration to promising trajectories, which is crucial for enabling manipulation in complex environments. We evaluate our framework in simulation and hardware in multiple environments with varying task complexity. We showcase that compared to the state-of-the-art, which achieves an average success rate of only 56.6% across three different tasks of varying difficulty, our algorithm significantly outperforms it with an average success rate of 90%.

Efficiently generating grasp poses tailored to specific regions of an object is vital for various robotic manipulation tasks, especially in a dual-arm setup. This scenario presents a significant challenge due to the complex geometries involved, requiring a deep understanding of the local geometry to generate grasps efficiently on the specified constrained regions. Existing methods only explore settings involving table-top/small objects and require augmented datasets to train, limiting their performance on complex objects. We propose CGDF: Constrained Grasp Diffusion Fields, a diffusion-based grasp generative model that generalizes to objects with arbitrary geometries, as well as generates dense grasps on the target regions. CGDF uses a part-guided diffusion approach that enables it to get high sample efficiency in constrained grasping without explicitly training on massive constraint-augmented datasets. We provide qualitative and quantitative comparisons using analytical metrics and in simulation, in both unconstrained and constrained settings to show that our method can generalize to generate stable grasps on complex objects, especially useful for dual-arm manipulation settings, while existing methods struggle to do so. More results, code and an extended version of the paper can be found on the project page: https://constrained-grasp-diffusion.github.io/

This paper addresses the consistency issue of multi-robot distributed cooperative localization. We introduce a consistent distributed cooperative localization algorithm conducting state estimation in a transformed coordinate. The core idea involves a linear time-varying coordinated transformation to render the propagation Jacobian independent of the state and make it suitable for a distributed manner. This transformation is seamlessly integrated into a server-based distributed cooperative localization framework, in which each robot estimates its own state while the server maintains the cross-correlations. The transformation ensures the correct observability property of the entire framework. Moreover, the algorithm accommodates various types of robot-to-robot relative measurements, broadening its applicability. Through simulations and real-world dataset experiments, the proposed algorithm has demonstrated better performance in terms of both consistency and accuracy compared to existing algorithms.

Self-reflecting about our performance (e.g., how confident we are) before doing a task is essential for decision making, such as selecting the most suitable tool or choosing the best route to drive. While this form of awareness—thinking about our performance or metacognitive performance—is well-known in humans, robots still lack this cognitive ability. This reflective monitoring can enhance their embodied decision power, robustness and safety. Here, we take a step in this direction by introducing a mathematical framework that allows robots to use their control self-confidence to make better-informed decisions. We derive a mathematical closed-form expression for control confidence for dynamic systems (i.e., the posterior inverse covariance of the control action). This control confidence seamlessly integrates within an objective function for decision making, that balances the: i) performance for task completion, ii) control effort, and iii) self-confidence. To evaluate our theoretical account, we framed the decision-making within the tool selection problem, where the agent has to select the best robot arm for a particular control task. The statistical analysis of the numerical simulations with randomized 2DOF arms shows that using control confidence during tool selection improves both real task performance, and the reliability of the tool for performance under unmodelled perturbations (e.g., external forces). Furthermore, our results indicate that control confidence is an early indicator of performance and thus, it can be used as a heuristic for making decisions when computation power is restricted or decision-making is intractable. Overall, we show the advantages of using confidence-aware decision-making and control scheme for dynamic systems.

Robots rely on visual relocalization to estimate their pose from camera images when they lose track. One of the challenges in visual relocalization is repetitive structures in the operation environment of the robot. This calls for probabilistic methods that support multiple hypotheses for robot’s pose. We propose such a probabilistic method to predict the posterior distribution of camera poses given an observed image. Our proposed training strategy results in a generative model of camera poses given an image, which can be used to draw samples from the pose posterior distribution. Our method is streamlined and well-founded in theory and outperforms existing methods on localization in presence of ambiguities.

This paper presents a Human-Robot Blind Handover architecture within the context of Human-Robot Collaboration (HRC). The focus lies on a blind handover scenario where the operator is intentionally faced away, focused in a task, and requires an object from the robot. In this context, it is imperative for the robot to autonomously manage the entire handover process. Key considerations include ensuring safety while handing the object to the operator’s hand, and detect the proper timing to release the object. The article explores strategies to navigate these challenges, emphasizing the need for a robot to operate safely and independently in facilitating blind handovers, thereby contributing to the advancement of HRC protocols and fostering a natural and efficient collaboration between humans and robots.

Flexibility and softness play a significant role in dynamic human motions. This includes the flexibility owing to ligaments in the human body and the softness of external structures such as a leaf-spring-type prosthesis. Thus, robotic systems need to utilize such flexibility to achieve dynamic and energy-efficient motion. In this study, we proposed a compliance optimization-based control framework for a rigid-soft hybrid robot system where the continuous deformation of a flexible structure is represented using the piece-wise constant strain (PCS) model. We divided the hybrid system into two states: single support and double support. We validated the proposed method in these states using forward dynamics simulations, assuming a hybrid link system that consists of a humanoid robot with a flexible prosthesis.