ICRA 2024 论文列表

This paper introduces iRoCo (intuitive Robot Control) – a framework for ubiquitous human-robot collaboration using a single smartwatch and smartphone. By integrating probabilistic differentiable filters, iRoCo optimizes a combination of precise robot control and unrestricted user movement from ubiquitous devices. We demonstrate and evaluate the effectiveness of iRoCo in practical teleoperation and drone piloting applications. Comparative analysis shows no significant difference between task performance with iRoCo and gold-standard control systems in teleoperation tasks. Additionally, iRoCo users complete drone piloting tasks 32% faster than with a traditional remote control and report less frustration in a subjective load index questionnaire. Our findings strongly suggest that iRoCo is a promising new approach for intuitive robot control through smartwatches and smart-phones from anywhere, at any time. The code is available at www.github.com/wearable-motion-capture

Existing methods for place recognition in 3D point clouds either ignore partial structure information by converting 3D scans to 2D images or construct constrained bag-of-words (BoW) representations reliant on specific feature extraction algorithms. In this paper, we propose a novel method based on incremental and general bag of words. Incorporating an adaptable keypoint and 3D local feature extraction method, we employ an incremental BoW model that is updated regularly. This enables a coarse-to-fine candidate selection from the database. And a revisit can be identified following geometric verification. In addition, we propose a new supplementary metric that addresses the leaving-out issue of the conventional metric, enhancing the identification of true loops. Employing a state-of-the-art (SOTA) keypoint and feature extraction algorithm, we evaluate our method as well as SOTA place recognition methods using diverse datasets with varying qualities. Experimental results demonstrate that our method outperforms the baselines across all three datasets, showcasing robust performance and notable generalization capabilities.

Establishing the correspondences between newly acquired points and historically accumulated data (i.e., the map) through nearest neighbor search is crucial in numerous robotic applications. However, static tree data structures are inadequate to handle large and dynamically growing maps in real-time. To address this issue, we present the i-Octree, a dynamic octree data structure that supports both fast nearest neighbor search and real-time dynamic updates, such as point insertion, deletion, and on-tree down-sampling. The i-Octree is built upon a leaf-based octree and has two key features: a local spatially continuous storing strategy that allows for fast access to points while minimizing memory usage, and local on-tree updates that significantly reduce computation time compared to existing static or dynamic tree structures. The experiments show that the i-Octree outperforms contemporary state-of-the-art approaches by achieving, on average, a 19% reduction in runtime on real-world open datasets.

Accurate calibration is crucial for using multiple cameras to triangulate the position of objects precisely. However, it is also a time-consuming process that needs to be repeated for every displacement of the cameras. The standard approach is to use a printed pattern with known geometry to estimate the intrinsic and extrinsic parameters of the cameras. The same idea can be applied to event-based cameras, though it requires extra work. By using frame reconstruction from events, a printed pattern can be detected. A blinking pattern can also be displayed on a screen. Then, the pattern can be directly detected from the events. Such calibration methods can provide accurate intrinsic calibration for both frame- and event-based cameras. However, using 2D patterns has several limitations for multi-camera extrinsic calibration, with cameras possessing highly different points of view and a wide baseline. The 2D pattern can only be detected from one direction and needs to be of significant size to compensate for its distance to the camera. This makes the extrinsic calibration time-consuming and cumbersome. To overcome these limitations, we propose eWand, a new method that uses blinking LEDs inside opaque spheres instead of a printed or displayed pattern. Our method provides a faster, easier-to-use extrinsic calibration approach that maintains high accuracy for both event- and frame-based cameras.

This paper presents a multi-robot kinodynamic motion planner that enables a team of robots with different dynamics, actuation limits, and shapes to reach their goals in challenging environments. We solve this problem by combining Conflict-Based Search (CBS), a multi-agent path finding method, and discontinuity-bounded A*, a single-robot kinodynamic motion planner. Our method, db-CBS, operates in three levels. Initially, we compute trajectories for individual robots using a graph search that allows bounded discontinuities between precomputed motion primitives. The second level identifies inter-robot collisions and resolves them by imposing constraints on the first level. The third and final level uses the resulting solution with discontinuities as an initial guess for a joint space trajectory optimization. The procedure is repeated with a reduced discontinuity bound. Our approach is anytime, probabilistically complete, asymptotically optimal, and finds near-optimal solutions quickly. Experimental results with robot dynamics such as unicycle, double integrator, and car with trailer in different settings show that our method is capable of solving challenging tasks with a higher success rate and lower cost than the existing state-of-the-art.

Extrinsic calibration for LiDAR and camera is an essential prerequisite for sensor fusion. Recently, automatic and target-less extrinsic calibration has become the mainstream of academic research. However, geometric feature-based methods still have requirements on the scene. Deep learning methods, while achieving high accuracy and good adaptability, rely on large annotated dataset and need additional training. We propose a novel LiDAR-camera calibration method by using the Segment Anything Model(SAM) without additional training. With the automatically generated masks, we optimize the extrinsic parameters by maximizing the consistency score of the point attributes that fall on each mask. The point cloud attributes include intensity, normal vector and segmentation class. Experiments on different real-world dataset demonstrate the accuracy and robustness of our proposed method. The code is available at https://github.com/OpenCalib/CalibAnything.

The growing focus on indoor robot navigation utilizing wireless signals has stemmed from the capability of these signals to capture high-resolution angular and temporal measurements. Prior heuristic-based methods, based on radio frequency (RF) propagation, are intuitive and generalizable across simple scenarios, yet fail to navigate in complex environments. On the other hand, end-to-end (e2e) deep reinforcement learning (RL) can explore a rich class of policies, delivering surprising performance when facing complex wireless environments. However, the price to pay is the astronomical amount of training samples, and the resulting policy, without fine-tuning (zero-shot), is unable to navigate efficiently in new scenarios unseen in the training phase. To equip the navigation agent with sample-efficient learning and zero-shot generalization, this work proposes a novel physics-informed RL (PIRL) where a distance-to-target-based cost (standard in e2e) is augmented with physics-informed reward shaping. The key intuition is that wireless environments vary, but physics laws persist. After learning to utilize the physics information, the agent can transfer this knowledge across different tasks and navigate in an unknown environment without fine-tuning. The proposed PIRL is evaluated using a wireless digital twin (WDT) built upon simulations of a large class of indoor environments from the AI Habitat dataset augmented with electromagnetic radiation simulation for wireless signals. It is shown that the PIRL significantly outperforms both e2e RL and heuristic-based solutions in terms of generalization and performance. Source code is available at https://github.com/Panshark/PIRL-WIN.

Object tracking is central to robot perception and scene understanding, allowing robots to parse a video stream in terms of moving objects with names. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories [1], [2]. Recently, large-scale pre-trained models have shown promising advances in detecting and segmenting objects and parts in 2D static images in the wild. This raises the question: can we re-purpose these large-scale pre-trained static image models for open-vocabulary video tracking? In this paper, we combine an open-vocabulary detector [3], segmenter [4], and dense optical flow estimator [5], into a model that tracks and segments any object in 2D videos. Given a monocular video input, our method predicts object and part mask tracks with associated language descriptions, rebuilding the pipeline of Tractor [6] with modern large pre-trained models for static image detection and segmentation: we detect open-vocabulary object instances and propagate their boxes from frame to frame using a flow-based motion model, refine the propagated boxes with the box regression module of the visual detector, and prompt an open-world segmenter with the refined box to segment the objects. We decide the termination of an object track based on the objectness score of the propagated boxes as well as forward-backward optical flow consistency. We re-identify objects across occlusions using deep feature matching. We show that our model achieves strong performance on multiple established benchmarks [7], [8], [9], [10], and can produce reasonable tracks in manipulation data [11]. In particular, our model outperforms previous state-of-the-art in UVO and BURST, benchmarks for open-world object tracking and segmentation, despite never being explicitly trained for tracking. We hope that our approach can serve as a simple and extensible framework for future research and enable imitation learning from videos with unconventional objects. Show More

Constrained robot motion planning is a ubiquitous need for robots interacting with everyday environments, but it is a notoriously difficult problem to solve. Many sampled points in a sample-based planner need to be rejected as they fall outside the constraint manifold, or require significant iterative effort to correct. Given this, few solutions exist that present a constraint-satisfying trajectory for robots, in reasonable time and of low path cost. In this work, we present a transformer-based model for motion planning with task space constraints for manipulation systems. Vector Quantized-Motion Planning Transformer (VQ-MPT) is a recent learning-based model that reduces the search space for unconstrained planning for sampling-based motion planners. We propose to adapt a pre-trained VQMPT model to reduce the search space for constraint planning without retraining or finetuning the model. We also propose to update the neural network output to move sampling regions closer to the constraint manifold. Our experiments show how VQ-MPT improves planning times and accuracy compared to traditional planners in simulated and real-world environments. Unlike previous learning methods, which require task-related data, our method uses pre-trained neural network models and requires no additional data for training and finetuning the model making this a one-shot process. We also tested our method on a physical Franka Panda robot with real-world sensor data, demonstrating the generalizability of our algorithm. We anticipate this approach to be an accessible and broadly useful for transfering learned neural planners to various robotic-environment interaction scenarios.

As robotic systems increasingly encounter complex and unconstrained real-world scenarios, there is a demand to recognize diverse objects. The state-of-the-art 6D object pose estimation methods rely on object-specific training and therefore do not generalize to unseen objects. Recent novel object pose estimation methods are solving this issue using task-specific fine-tuned CNNs for deep template matching. This adaptation for pose estimation still requires expensive data rendering and training procedures. MegaPose for example is trained on a dataset consisting of two million images showing 20,000 different objects to reach such generalization capabilities. To overcome this shortcoming we introduce ZS6D, for zero-shot novel object 6D pose estimation. Visual descriptors, extracted using pre-trained Vision Transformers (ViT), are used for matching rendered templates against query images of objects and for establishing local correspondences. These local correspondences enable deriving geometric correspondences and are used for estimating the object's 6D pose with RANSAC- based PnP. This approach showcases that the image descriptors extracted by pre-trained ViTs are well-suited to achieve a notable improvement over two state-of-the-art novel object 6D pose estimation methods, without the need for task-specific fine-tuning. Experiments are performed on LMO, YCBV, and TLESS. In comparison to MegaPose, we improve the Average Recall on all three datasets and compared to OSOP we improve on two datasets. The code is available at https://github.com/PhilippAuss/ZS6D.

The past few years have seen immense progress on two fronts that are critical to safe, widespread mobile robot deployment: predicting uncertain motion of multiple agents, and planning robot motion under uncertainty. However, the numerical methods required on each front have resulted in a mismatch of representation for prediction and planning. In prediction, numerical tractability is usually achieved by coarsely discretizing time, and by representing multimodal multi-agent interactions as distributions with infinite support. On the other hand, safe planning typically requires very fine time discretization, paired with distributions with compact support, to reduce conservativeness and ensure numerical tractability. The result is, when existing predictors are coupled with planning and control, one may often find unsafe motion plans. This paper proposes ZAPP (Zonotope Agreement of Prediction and Planning) to resolve the representation mismatch. ZAPP unites a prediction-friendly coarse time discretization and a planning-friendly zonotope uncertainty representation; the method also enables differentiating through a zonotope collision check, allowing one to integrate prediction and planning within a gradient-based optimization framework. Numerical examples show how ZAPP can produce safer trajectories compared to baselines in interactive scenes.

In the realm of robotic grasping, achieving accurate and reliable interactions with the environment is a pivotal challenge. Traditional methods of grasp planning methods utilizing partial point clouds derived from depth image often suffer from reduced scene understanding due to occlusion, ultimately impeding their grasping accuracy. Furthermore, scene reconstruction methods have primarily relied upon static techniques, which are susceptible to environment change during manipulation process limits their efficacy in real-time grasping tasks. To address these limitations, this paper introduces a novel two-stage pipeline for dynamic scene reconstruction. In the first stage, our approach takes scene scanning as input to register each target object with mesh reconstruction and novel object pose tracking. In the second stage, pose tracking is still performed to provide object poses in real-time, enabling our approach to transform the reconstructed object point clouds back into the scene. Unlike conventional methodologies, which rely on static scene snapshots, our method continuously captures the evolving scene geometry, resulting in a comprehensive and up-to-date point cloud representation. By circumventing the constraints posed by occlusion, our method enhances the overall grasp planning process and empowers state-of-the-art 6-DoF robotic grasping algorithms to exhibit markedly improved accuracy.

Recently, tactile sensing has attracted great interests in robotics, especially for exploring unstructured objects. Sensor arrays play an important role in the exploration, which generates rich spatio-temporal information. In this work, we propose an efficient tactile recognition model, X-Tacformer. This model pays attention to both spatial and temporal features of tactile sequences from sensor arrays, which is verified by four public datasets, Ev-Objects, Ev-Containers, Augment8000 and BioTac-Dos. Comparative studies show that our model has resulted in a significant improvement of the recognition accuracy by 0.0223, 0.1416, 0.2735 and 0.1592 in these datasets. In order to verify its performances on dataset with rich spatio-temporal features, a self-designed dataset, ALU-Textures, was constructed with 10 fabrics from everyday textiles, aiming to extend the data collection action modes of current datasets by simulating human rubbing movements with the thumb and index fingers of an Allegro hand. Our model also demonstrates efficient salient feature learning capabilities on ALU-Textures, which is further augmented by tactile data augmentation methods.

In the paper, we address wireless communication infrastructure building by relay placement based on approaches utilized in wireless network sensors. The problem is motivated by search and inspection missions with mobile robots, where known sensing ranges may be exploited. We investigate the relay placement, establishing network connectivity to support robust food-based communication routing. The proposed method decomposes the given area into Open space and Corridor space where specific deployment patterns allow for guaranteed k-connectivity, making the resulting network redundant while keeping channel utilization bounded. In particular, a hexagonal tesselation coverage pattern with 3-connectivity is investigated in Open space and a linear 4-connectivity pattern in Corridor space, respectively. The proposed approach is empirically evaluated in a realistic scenario, and based on the reported results, it is found superior compared to the existing stochastic randomized dual sampling schema.

In multi-drone systems such as drone light shows, drones move in formation while avoiding collisions. However, few existing formation planning algorithms consider the wind fields of drones during planning. Since the wind field effect is prominent when drones have to fly close to each other, we cannot ignore the effect during planning. In this paper, we extend the reservation system in autonomous intersection management for grid-based formation planning by including a new type of reservation called non-exclusive reservations specifically for handling wind fields. We train a deep learning model to predict the deviation of a drone’s trajectory when the drone enters the wind field of another drone and then use the reservation grid to prevent collision. Based on the reservation system, we develop a new formation planning algorithm that focuses on adjusting the start times of motion plans to avoid collision. Our experimental results show that trajectory prediction can help make better decisions in task assignments for minimizing makespans.

In this paper, a variant of hybrid A* is developed to find the shortest path for a curvature-constrained robot, that is tethered at its start position, such that the tether satisfies user-defined winding angle constraints. A variant of tangent graphs is used as an underlying graph for searching a path using A* in order to reduce the overall computation and define appropriate cost metrics to ensure winding angle constraints are satisfied. Conditions are provided under which the proposed algorithm is guaranteed to find a winding angle constrained path. The effectiveness and performance of the proposed algorithm are studied in simulation.

Cleaning glass surfaces is a prevailing maintenance problem in high-rise buildings. In the traditional methods of cleaning windows, hanging on ropes poses significant occupational hazards to workers. Furthermore, most glass facades feature window frames to securely fasten the glass panels to the building structure, ensuring durability and elegance. In this context, existing robotic cleaning methods are limited by their capability to move-over window frames and need more flexibility to access tight corners and curved surfaces. This paper presents a novel reconfigurable glass cleaning robot called "WiBot" to address these limitations. WiBot is a kinematic chain comprising modular linkages with a prismatic joint and two revolute joints at each end. Each revolute joint has a suction unit that enables locomotion and adhesion. Window frames are detected using image processing with an onboard camera, and design optimizations were performed to improve the robot’s capabilities. The prototype WiBot 1.0 was developed, and several experiments were conducted to evaluate the feasibility of the proposed system focusing on robot motion, window frame detection and move-over mechanism. The results show that WiBot can overcome the limitations of existing window cleaning solutions. Finally, several promising research directions are mentioned involving the proposed reconfigurable robot architecture in cleaning operations.

This work explores the use of artificial whiskers as tactile sensors for enhancing the perception and navigation capabilities of mobile robots in challenging settings such as caves and underground mines. These environments exhibit inconsistent lighting conditions, locally self-similar textures, and general poor visibility conditions, that can cause the performance of state-of-the-art vision-based methods to decline. In order to evaluate the efficacy of tactile sensing in this context, three algorithms were developed and tested with simulated and physical experiments: a wall-follower, a navigation algorithm based on Theta*, and a hybrid approach that combines the two. The obtained results highlight the efficacy of tactile sensing for wall-following in intricate environments. When paired with an external method for pose estimation, it further aids in navigating unknown environments. Moreover, by integrating navigation with wall-following, the third, hybrid algorithm enhanced the map traversal speed by roughly 26−43% compared to standard navigation methods without wall-following.

The hierarchy of global and local planners is one of the most commonly utilized system designs in autonomous robot navigation. While the global planner generates a reference path from the current to goal locations based on the pre-built map, the local planner produces a kinodynamic trajectory to follow the reference path while avoiding perceived obstacles. To account for unforeseen or dynamic obstacles not present on the pre-built map, "when to replan" the reference path is critical for the success of safe and efficient navigation. However, determining the ideal timing to execute replanning in such partially unknown environments still remains an open question. In this work, we first conduct an extensive simulation experiment to compare several common replanning strategies and confirm that effective strategies are highly dependent on the environment as well as the global and local planners. Based on this insight, we then derive a new adaptive replanning strategy based on deep reinforcement learning, which can learn from experience to decide appropriate replanning timings in the given environment and planning setups. Our experimental results show that the proposed replanner can perform on par or even better than the current best-performing strategies in multiple situations regarding navigation robustness and efficiency.