IROS 2024 论文列表

This paper addresses the problem of autonomous robotic inspection in complex and unknown environments. This capability is crucial for efficient and precise inspections in various real-world scenarios, even when faced with perceptual uncertainty and lack of prior knowledge of the environment. Existing methods for real-world autonomous inspections typically rely on predefined targets and waypoints and often fail to adapt to dynamic or unknown settings. In this paper, we introduce the Semantic Belief Behavior Graph (SB2G) framework as a new approach to semantic-aware autonomous robot inspection. SB2G generates a control policy for the robot, using behavior nodes that encapsulate various semantic-based policies designed for inspecting different classes of objects. We design an active semantic search behavior to guide the robot in locating objects for inspection while reducing semantic information uncertainty. The edges in the SB2G encode transitions between these behaviors. We validate our approach through simulation and real-world urban inspections using a legged robotic platform. Our results show that SB2G enables a more efficient object inspection policy, exhibiting similar behaviors comparable to human-operated inspections.

Self-supervised depth estimation has attracted much attention due to its ability to improve the 3D perception capabilities of unmanned systems. However, existing unsupervised frameworks rely on the assumption of photometric consistency, which may not hold in challenging environments such as night-time, rainy nights, or snowy winters due to complex lighting and reflections, resulting in inconsistent photometry across different frames for the same pixel. To address this problem, we propose a self-supervised monocular depth estimation unified framework that can handle these complex scenarios, which has the following characteristics: (1) an Illumination Compensation PoseNet (ICP) is designed, which is based on the classic Phong illumination theory and compensates for lighting changes in adjacent frames by estimating per-pixel transformations; (2) a Dual-Axis Transformer (DAT) block is proposed as the backbone network of the depth encoder, which infers the depth of local repeat-texture areas through spatial-channel dual-dimensional global context information of images. Experimental results demonstrate that our approach achieves state-of-the-art depth estimation results in complex environments on the challenging Oxford RobotCar dataset.

This paper proposes a distributed on-orbit spacecraft assembly algorithm, where future spacecraft can assemble modules with different functions on orbit to form a spacecraft structure with specific functions. This form of spacecraft organization has the advantages of reconfigurability, fast mission response and easy maintenance. Reasonable and efficient on-orbit self-reconfiguration algorithms play a crucial role in realizing the benefits of distributed spacecraft. This paper adopts the framework of imitation learning combined with reinforcement learning for strategy learning of module handling order. A robot arm motion algorithm is then designed to execute the handling sequence. We achieve the self-reconfiguration handling task by creating a map on the surface of the module, completing the path point planning of the robotic arm using A*. The joint planning of the robotic arm is then accomplished through forward and reverse kinematics. Finally, the results are presented in Unity3D.

In driving scenarios, motion segmentation is a crucial and fundamental component that is needed for many tasks. Recently, a self-supervised multitasking framework was proposed for driving scenarios. It simultaneously trains motion segmentation, optical flow, depth, and ego-motion models without annotated data. The self-supervised architecture derives training signals from training data via loss functions. If these loss functions lack robustness, they may result in model inaccuracies. To reduce the bad influences of occlusion and optical flow estimation errors on motion segmentation, we propose two loss functions: (1) Soft-Per-Pixel-Minimum (Soft-PPM) loss that excludes occluded pixels while balancing the contribution of each frame on the loss function temporally; (2) Flow difference loss that excludes pixels with unclear motion states to diminish the effect of optical flow estimation errors. Our loss function design is based on the key insight that information such as depth and optical flow can be used to train motion segmentation models and act as a reliable measure for pixels during training. Our approach can improve segmentation accuracy for both moving and static objects and has achieved IoU scores on moving and static classes comparable to the state-of-the-art methods on the KITTI dataset.

Monocular depth estimation obtaining scene depth information from a single image is an important task in the field of computer vision. Constrained by the limitations of convolutional networks in conducting long-distance modeling and the underutilization of datasets, the generalization of existing models is not satisfactory. In this paper, we propose an adaptive backbone named Internal Fusion Transformer to improve generalization ability compared to convolutional backbone, like HRNet, and a Bilateral Attention module which pays more attention to low-level semantic features compared to previous fuse methods. Meanwhile, we introduce three data augmentation methods, namely cropping-resizing (cr), cropping-shuffling (cs), and mirroring (mi), for self distillation, as well as discuss their contributions to model performance improvement. Our model is trained on the KITTI dataset, and without fine-tuning, tested on NYUv2 and Make3D datasets to confirm the generalization. The experimental results illustrate the effectiveness of our design. Our model also demonstrates better performance compared to other models on the KITTI dataset.

Brain-computer interfaces (BCI) serve as an important tool in areas such as neurorehabilitation and constructing prostheses. Electroencephalogram (EEG) motor imagery (MI) signal is a common method used to communicate between the human brain and the computer interface. However, differentiating between multiple motor imagery signals may be challenging due to the presence of high noise-to-signal ratio and small dataset sizes. In this study, we propose a variational autoencoder and transformer-attention based convolutional neural network (SSTACNet) for multi-class EEG-based motor imagery classification. The SSTACNet model leverages upon variational autoencoders’ ability to measure the contrastive distance between two sets of inputs to perform data self-selection. The model further utilizes multi-head self-attention as well as spatial and temporal convolutional filters to achieve superior extraction of signal features. The model additionally utilizes the variational autoencoder’s ability to augment the dataset with feature-informed pseudo-data, achieving stronger classification results. The proposed model outperforms the current state-of-the-art techniques in the BCI Competition IV-2a dataset with an accuracy of 85.52% and 70.56% for the subject-dependent and subject-independent modes, respectively. Codes may be found at: https://github.com/NgHanWei/SSTACNet

This study explores the potential of automating robotic laboratory readiness assessment by integrating Large Language Models (LLMs) with robotic data acquisition. It investigates the capability of LLMs to detect equipment motion and operational status using visual and auditory information. Despite the challenges LLMs face in spatial analysis, this study also investigates LLM grounding methods to ensure accurate workspace assessment. By inspecting a robotic cooking setup with camera-equipped robotic arm, LLMs can detect the motion of custom equipment via color-coded marks, and identify the operational status of kitchen appliances from a single image without any physical augmentations. Additionally, device operation perceived through the emission of loud noises can be assessed by post-processing sound recordings and analyzing loudness and sound frequency metrics presented in a visual plot form. For simple spatial tasks like saucepan positioning, LLM provides accurate assessments when grounded with a single image, while complex workspace safety assessment task requires extensive knowledge of past experiences. By reviewing status of each checklist item, the LLM can decide whether experiment needs to be halted or requires human intervention, offering a set of troubleshooting steps. These findings demonstrate feasibility of the self-assessment approach for robotic laboratory systems, paving the way for future deployments.

Mobile self-reconfigurable robots (MSRRs), as a novel multi-robot system with flexible configurations and task adaptability, hold promising applications in unstructured task environments. However, existing autonomous docking strategies are primarily applied in laboratory settings and face numerous challenges and limitations in actual applications, including differences in sensor characteristics, safety threats, and saturation constraints. To address these issues, this paper proposes a segmented secure docking control framework based on global localization and local perception to achieve stable and reliable reconfiguration of MSRRs in practical applications. Specific contributions include the implementation of a dual-layer constraint framework for safeness of units in the long-distance phase against velocity and acceleration nested windups, and the integration of active line-of-sight (LOS) correction and adaptive windup driving mobile units to achieve precise and rapid locking of docked positions within the LOS in the close-range phase. Finally, the validity of the proposed method is verified via physical experiments, offering an innovative approach to deploying MSRRs in complex scenarios.

Current bin picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings.

Surgical scene understanding in Robot-assisted Minimally Invasive Surgery (RMIS) is highly reliant on visual cues and lacks tactile perception. Force-modulated surgical palpation with tactile feedback is necessary for localization, geometry/depth estimation, and dexterous exploration of abnormal stiff inclusions in subsurface tissue layers. Prior works explored surface-level tissue abnormalities or single layered tissue-tumor embeddings with more than 300 palpations for dense 2D stiffness mapping. Our approach focuses on 3D reconstructions of sub-dermal tumor surface profiles in multi-layered tissue (skin-fat-muscle) using a visually-guided novel tactile navigation policy. A robotic palpation probe with triaxial force sensing was leveraged for tactile exploration of the phantom. From a surface mesh of the surgical region initialized from a depth camera, the policy explores a surgeon’s region of interest through palpation, sampled from bayesian optimization. Each palpation includes contour following using a contact-safe impedance controller to trace the sub-dermal tumor geometry, until the underlying tumor-tissue boundary is reached. Projections of these contour following palpation trajectories allows 3D reconstruction of the subdermal tumor surface profile in less than 100 palpations. Our approach generates high-fidelity 3D surface reconstructions of rigid tumor embeddings in tissue layers with isotropic elasticities, although soft tumor geometries are yet to be explored. For more details, please refer to our open-source codebase1 and project website2.

This paper addresses the spatio-temporal areas restoration problem: a robot, with limited battery life, deployed in a known environment, needs to persistently plan a schedule to visit areas of interest and charge its battery as needed. The goal is to restore the areas’ properties that temporally decay—such as air quality—so that the time the measured property values are below a certain threshold is minimized. This problem is different from typical problems solved in the area of monitoring a spatio-temporal environment. A related problem is the orienteering problem, where a robot visits nodes to maximize the profit collected at each visited node within a time budget frame. That problem is NP-hard. The typical formulation considers static profit, while we consider a time-varying one. Given look-ahead time window or schedule length, we formulate the problem as an optimization search problem with a temporal objective, and devise a heuristic function that enables finding solutions in polynomial time. The heuristic evaluates the discounted opportunity costs of a visit—a concept borrowed from economics. We then develop a greedy algorithm that takes the immediate feasible visit that minimizes this heuristic. This strategy addresses a primary limitation of a recent approach in applications where being able to revisit highly urgent areas within the time window of the schedule is critical. We provide a theoretical analysis on lower and upper bounds for the problem. Extensive experimental results with a robotic simulator show that our method is able to keep the areas in the environment above the threshold better than other methods and closer to the optimal. This work can enable high-impact applications, such as environmental preservation.

Prediction beyond partial observations is crucial for robots to navigate in unknown environments because it can provide extra information regarding the surroundings beyond the current sensing range or resolution. In this work, we consider the inpainting of semantic Bird’s-Eye-View maps. We propose SePaint, an inpainting model for semantic data based on generative multinomial diffusion. To maintain semantic consistency, we need to condition the prediction for the missing regions on the known regions. We propose a novel and efficient condition strategy, Look-Back Condition (LB-Con), which performs one-step look-back operations during the reverse diffusion process. By doing so, we are able to strengthen the harmonization between unknown and known parts, leading to better completion performance. We have conducted extensive experiments on different datasets, showing our proposed model outperforms commonly used interpolation methods in various robotic applications.

Manipulating deformable objects remains a challenge within robotics due to the difficulties of state estimation, long-horizon planning, and predicting how the object will deform given an interaction. These challenges are the most pronounced with 3D deformable objects. We propose SculptDiff, a goal-conditioned diffusion-based imitation learning framework that works with point cloud state observations to directly learn clay sculpting policies for a variety of target shapes. To the best of our knowledge this is the first real-world method that successfully learns manipulation policies for 3D deformable objects. For sculpting videos and access to our dataset and hardware CAD models, see the project website: https://sites.google.com/andrew.cmu.edu/imitation-sculpting/home

This paper proposes a new Petri-net-based Q-learning scheduling method to schedule robotic cellular manufacturing (RCM) systems efficiently. First, we use generalized and place-timed Petri nets to model RCM systems. Then, we design a reinforcement learning method with a sparse Q-table to evaluate state-transition pairs of the net’s reachability graph. It uses the negative transition firing time as a reward for an action selection and adopts a large penalty for any encountered deadlock. In addition, it balances the state space exploration and the experience exploitation by using a dynamic ϵ-greedy policy to update the state values with an accumulative reward. Three different dynamic ϵ-greedy policies are designed for different application scenarios. Some benchmark RCM systems are tested with the proposed method and several popular PN-based online dispatching rules, such as FIFO and SRPT. Simulation results demonstrate that our method schedules RCM systems as quickly as the online dispatching rules while outperforming them in terms of schedule makespan. For readers’ reference, our source code and test data are available at https://github.com/PNOptimizer/PNQL.

When exploring new areas, robotic systems generally exclusively plan and execute controls over geometry that has been directly measured. This planning paradigm can lead to unintuitive exploration or replanning latency when entering areas that were previous obstructed from view. To address this we present SceneSense, a real-time 3D diffusion model for synthesizing 3D occupancy information from partial observations that effectively predicts these occluded or out of view geometries for use in future planning and control frameworks. SceneSense uses a running occupancy map and a single RGB-D camera to generate predicted geometry around the platform at runtime, even when the geometry is occluded or out of view. Our architecture ensures that SceneSense never overwrites observed free or occupied space. By preserving the integrity of the observed map, SceneSense mitigates the risk of corrupting the observed space with generative predictions. While SceneSense is shown to operate well using a single RGB-D camera, the framework is flexible enough to extend to additional modalities. Unlike existing models that necessitate multiple views and offline scene synthesis, or are focused on filling gaps in observed data, our findings demonstrate that SceneSense is an effective approach to estimating unobserved local occupancy information at runtime. Local occupancy predictions from SceneSense are shown to better represent the ground truth occupancy distribution during the test exploration trajectories than the running occupancy map. The source code can be found on our website: https://arpg.github.io/scenesense/

Interactive point cloud segmentation has become a pivotal task for understanding 3D scenes, enabling users to guide segmentation models with simple interactions such as clicks, therefore significantly reducing the effort required to tailor models to diverse scenarios and new categories. However, in the realm of interactive segmentation, the meaning of instance diverges from that in instance segmentation, because users might desire to segment instances of both thing and stuff categories that vary greatly in scale. Existing methods have focused on thing categories, neglecting the segmentation of stuff categories and the difficulties arising from scale disparity. To bridge this gap, we propose ClickFormer, an innovative interactive point cloud segmentation model that accurately segments instances of both thing and stuff categories. We propose a query augmentation module to augment click queries by a global query sampling strategy, thus maintaining consistent performance across different instance scales. Additionally, we employ global attention in the query-voxel transformer to mitigate the risk of generating false positives, along with several other network structure improvements to further enhance the model’s segmentation performance. Experiments demonstrate that ClickFormer outperforms existing interactive point cloud segmentation methods across both indoor and outdoor datasets, providing more accurate segmentation results with fewer user clicks in an open-world setting. Project page: https://sites.google.com/view/clickformer/

We address the problem of sparse selection of visual features for localizing a team of robots navigating in an unknown environment, where robots can exchange relative position measurements with neighbors. We select a set of the most informative features by anticipating their importance in robots localization by simulating trajectories of robots over a prediction horizon. Through theoretical proofs, we establish a crucial connection between graph Laplacian and the importance of features. We leverage a scalable randomized algorithm for sparse sums of positive semidefinite matrices to efficiently select a set of the most informative features.

The estimation of the 6D pose of industrial components is essential for smart manufacturing. Especially for complex units that require intensive manual operations, such as a concentrator photovoltaics solar panel, accurate spatial localization provides visual aids for industrial automation. In this paper, we propose an accurate and scalable framework to address the dimensional variability of industrial components and tackle practical implementation issues. First, we use the scalable architecture EfficientNet as the backbone coupled with an enhanced feature pyramid network to estimate the object’s pose. By introducing vertical and horizontal connections of shallow layers, the feature extraction of small objects is optimized for better detection accuracy. Second, leveraging the reliable 2D detection results and geometry information, an adaptive pose refinement module is designed to adjust the estimated 6D pose. The scaling of the backbone network and the computational complexity of refined modules are uniformly adjusted via a shared hyperparameter, resulting in a globally scalable framework. In terms of the pose estimation accuracy, the effectiveness of the refinement module and the real-time performance, validations are conducted both on the LINEMOD dataset and our customized datasets comprising of objects from the industrial photovoltaic system. Additionally, to further illustrate the effectiveness of the proposed method, a precision parallel robot is employed to validate the accuracy of real-time object pose tracking.