IROS 2023 论文列表

This paper proposes a novel Generative Adversarial Network (GAN)-based strategy to augment subjective haptic Quality of Experience (QoE) datasets for bilateral teleoperation with haptic feedback without conducting time-consuming subjective experiments. In our previous work, we proposed a multi-assessment fusion approach to predict subjective haptic quality using a collection of objective metrics. This method requires a sufficiently large haptic dataset with QoE labels. The proposed generative approach automatically expands the existing haptic quality dataset by combining a modified conditional GAN (CGAN) and Style GAN (StyleGAN) architecture. The most important feature of our method is that it learns from the labeled training data and focuses on synthesizing signals with artifacts according to new input labels containing the QoE score, time delay, control method, and data reduction information. Extensive experiments are conducted to validate the suitability of the expanded dataset. The results show that our approach is able to generate new data, which match the label and signal distribution of the original data with categorical rank and linear correlation of over 0.85.

Robotic in-hand manipulation increases the capability of robotic hands to interact with the world. The amount of manipulation that a robot is capable of is highly dependent on the design of the robot hand, and previous works have shown success in designing hands to improve performance for different types of grasping and manipulation. In this paper we present a method for designing a fully-actuated planar manipulator that optimizes for specific in-hand motions. We demonstrate that, with the Asterisk Benchmark and a light-weight IK controller, we can translate our results from simulation to the real world with minimal effort and high-fidelity. Using the simulated data (over 4,000 simulated hand-designs) we begin to analyze which features contribute to improved planar manipulation.

The Multi-Agent Path Finding (MAPF) problem is a critical challenge in dynamic multi-robot systems. Recent studies have revealed that multi-agent reinforcement learning (MARL) is a promising approach to solving MAPF problems in a fully decentralized manner. However, as the size of the multi-robot system increases, sample inefficiency becomes a major impediment to learning-based methods. This paper presents a hierarchical reinforcement learning (HRL) framework for large-scale multi-agent path finding, featuring applying spatial and temporal abstraction to capture intermediate reward and thus encourage efficient exploration. Specifically, we introduce a meta controller that partitions the map into interconnected regions and optimizes agents' region-wise paths towards globally better solutions. Additionally, we design a lower-level controller that efficiently solves each sub-problem by incorporating heuristic guidance and an inter-agent communication mechanism with RL-based policies. Our empirical results on test instances of various scales demonstrate that our method outperforms existing approaches in terms of both success rate and makespan.

We present the HANDAL dataset for category-level object pose estimation and affordance prediction. Unlike previous datasets, ours is focused on robotics-ready manipulable objects that are of the proper size and shape for functional grasping by robot manipulators, such as pliers, utensils, and screwdrivers. Our annotation process is streamlined, requiring only a single off-the-shelf camera and semi-automated processing, allowing us to produce high-quality 3D annotations without crowd-sourcing. The dataset consists of 308k annotated image frames from 2.2k videos of 212 real-world objects in 17 categories. We focus on hardware and kitchen tool objects to facilitate research in practical scenarios in which a robot manipulator needs to interact with the environment beyond simple pushing or indiscriminate grasping. We outline the usefulness of our dataset for 6-DoF category-level pose+scale estimation and related tasks. We also provide 3D reconstructed meshes of all objects, and we outline some of the bottlenecks to be addressed for democratizing the collection of datasets like this one. Project website: https://nvlabs.github.io/HANDAL/

This paper presents a novel uncrewed aerial vehicle (UAV) design named HALO, which stands for “harmless aerial limber robot”. HALO uses a swashplateless mechanism to generate a moment for pitch and roll control without requiring additional actuators such as servo, reducing the number of components needed for control and enhancing reliability. Its reduced weight and number of actuators improve payload capacity and maneuverability. Meanwhile, HALO's coaxial duct design improves safety and aerodynamic efficiency. Experimental tests, including figure-of-eight trajectory tracking, wind gust and stick poking disturbances, hover efficiency comparison, and actual flight with collision is conducted to confirm HALO's robustness and exceptional safety characteristics, suggesting it as a promising design for various applications.

The agricultural setting poses additional challenges for robotic manipulation, as fruit is firmly attached to plants and the environment is cluttered and occluded. Therefore, accurate feedback about the grasp state is essential for effective harvesting. This study examines the different states involved in fruit picking by a robot, such as successful grasp, slip, and failed grasp, and develops a learning-based classifier using low-cost, computationally light sensors (IMU and IR reflectance). The Random Forest multi-class classifier accurately determines the current state and along with the sensors can operate in the occluded environment of a plant. The classifier was successfully trained and tested in the lab and showed 100% success at identifying slip and grasp failure and 80% success identifying successful picks on a real cherry tomato plant. By using this classifier, corrective actions can be planned based on the current state, thus leading to more efficient fruit harvesting.

Extensive research has been conducted on assessing grasp stability, a crucial prerequisite for achieving optimal grasping strategies, including the minimum force grasping policy. However, existing works employ basic feature-level fusion techniques to combine visual and tactile modalities, resulting in the inadequate utilization of complementary information and the inability to model interactions between unimodal features. This work proposes an attention-guided cross-modality fusion architecture to comprehensively integrate visual and tactile features. This model mainly comprises convolutional neural networks (CNNs), self-attention, and cross-attention mechanisms. In addition, most existing methods collect datasets from real-world systems, which is time-consuming and high-cost, and the datasets collected are comparatively limited in size. This work establishes a robotic grasping system through physics simulation to collect a multimodal dataset. To address the sim-to-real transfer gap, we propose a migration strategy encompassing domain randomization and domain adaptation techniques. The experimental results demonstrate that the proposed fusion framework achieves markedly enhanced prediction performance (approximately 10%) compared to other baselines. Moreover, our findings suggest that the trained model can be reliably transferred to real robotic systems, indicating its potential to address real-world challenges.

Robotic grasping is a fundamental skill for robots, but it is quite challenging in cluttered scenes. In cluttered scenes, the precise prediction of high-quality grasp configurations such as rotation and grasping width while avoiding collisions is essential. To accomplish this, the grasp detection models require the capabilities of stronger fine-grained information extracted around the grasp points. However, due to the computational resource restriction, point clouds are usually downsampled in existing networks, which inevitably make some potentially important points discarded. To overcome this problem, we propose a Grasp Region Exploration module to explore the area covered by high-quality grasps. Based on the grasp region, we enhance the point density around the grasp points to mitigate the loss of information caused by downsampling. Furthermore, we devise the Grasp Region Attention module to dynamically aggregate features of various points within the grasp region, such as the grasp point and contact points. The proposed method achieves state-of-the-art performance on the large-scale GraspNet-1Billion dataset. We also conduct real-world experiments on a Franka Emika Panda robot and show that the robot can grasp objects in cluttered scenes with a high success rate.

Ease of calibration and high-accuracy task-space state-estimation purely based on onboard sensors is a key requirement for enabling easily deployable cable robots in real-world applications. In this work, we incorporate the onboard camera and kinematic sensors to drive a statistical fusion framework that presents a unified localization and calibration system which requires no initial values for the kinematic parameters. This is achieved by formulating a Monte-Carlo algorithm that initializes a factor-graph representation of the calibration and localization problem. With this, we are able to jointly identify both the kinematic parameters and the visual odometry scale alongside their corresponding uncertainties. We demonstrate the practical applicability of the framework using our state-estimation dataset recorded with the ARAS-CAM suspended cable driven parallel robot, and published as part of this manuscript.

Crop monitoring is crucial for maximizing agricultural productivity and efficiency. However, monitoring large and complex structures such as sweet pepper plants presents significant challenges, especially due to frequent occlusions of the fruits. Traditional next-best view planning can lead to unstructured and inefficient coverage of the crops. To address this, we propose a novel view motion planner that builds a graph network of viable view poses and trajectories between nearby poses, thereby considering robot motion constraints. The planner searches the graphs for view sequences with the highest accumulated information gain, allowing for efficient pepper plant monitoring while minimizing occlusions. The generated view poses aim at both sufficiently covering already detected and discovering new fruits. The graph and the corresponding best view pose sequence are computed with a limited horizon and are adaptively updated in fixed time intervals as the system gathers new information. We demonstrate the effectiveness of our approach through simulated and real-world experiments using a robotic arm equipped with an RGB-D camera and mounted on a trolley. As the experimental results show, our planner produces view pose sequences to systematically cover the crops and leads to increased fruit coverage when given a limited time in comparison to a state-of-the-art single next-best view planner.

In this paper, we propose a solution for legged robot localization using architectural plans. Our specific contributions towards this goal are several. Firstly, we develop a method for converting the plan of a building into what we denote as an architectural graph (A-Graph). When the robot starts moving in an environment, we assume it has no knowledge about it, and it estimates an online situational graph representation (S-Graph) of its surroundings. We develop a novel graph-to-graph matching method, in order to relate the S-Graph estimated online from the robot sensors and the A-Graph extracted from the building plans. Note the challenge in this, as the S-Graph may show a partial view of the full A-Graph, their nodes are heterogeneous and their reference frames are different. After the matching, both graphs are aligned and merged, resulting in what we denote as an informed Situational Graph (is-Graph), with which we achieve global robot localization and exploitation of prior knowledge from the building plans. Our experiments show that our pipeline shows a higher robustness and a significantly lower pose error than several LiDAR localization baselines. Paper Video: https://youtu.be/3Pv7y8aOsUY

Point Cloud Registration is a fundamental and challenging problem in 3D computer vision. Recent works often utilize geometric structure features in downsampled points (patches) to seek correspondences, then propagate these sparse patch correspondences to the dense level in the corresponding patches' neighborhood. However, they neglect the explicit global scale rigid constraint at the dense level point matching. We claim that the explicit isometry-preserving constraint in the dense level on a global scale is also important for improving feature representation in the training stage. To this end, we propose a Graph Matching Optimization based Network (GMONet for short), which utilizes the graph-matching optimizer to explicitly exert the isometry preserving constraints in the point feature training to improve the point feature representation. Specifically, we exploit a partial graph-matching optimizer to enhance the super point (i.e., down-sampled key points) features and a full graph-matching optimizer to improve the dense level point features in the overlap region. Meanwhile, we leverage the inexact proximal point method and the mini-batch sampling technique to accelerate these two graph-matching optimizers. Given high discriminative point features in the evaluation stage, we utilize the RANSAC approach to estimate the transformation between the scanned pairs. The proposed method has been evaluated on the 3DMatch/3DLoMatch and the KITTI datasets. The experimental results show that our method performs competitively compared to state-of-the-art baselines.

6-DoF object-agnostic grasping in unstructured environments is a critical yet challenging task in robotics. Most current works use non-optimized approaches to sample grasp locations and learn spatial features without concerning the grasping task. This paper proposes GraNet, a graph-based grasp pose generation framework that translates a point cloud scene into multi-level graphs and propagates features through graph neural networks. By building graphs at the scene level, object level, and grasp point level, GraNet enhances feature embedding at multiple scales while progressively converging to the ideal grasping locations by learning. Our pipeline can thus characterize the spatial distribution of grasps in cluttered scenes, leading to a higher rate of effective grasping. Furthermore, we enhance the representation ability of scalable graph networks by a structure-aware attention mechanism to exploit local relations in graphs. Our method achieves state-of-the-art performance on the large-scale GraspNet-1Billion benchmark, especially in grasping unseen objects (+11.62 AP). The real robot experiment shows a high success rate in grasping scattered objects, verifying the effectiveness of the proposed approach in unstructured environments.

State-of-the-art multi-agent collision avoidance algorithms face limitations when applied to cluttered public environments, where obstacles may have a variety of shapes and structures. The issue arises because most of these algorithms are agent-level methods. They concentrate solely on preventing collisions between the agents while the obstacles are handled merely out-of-policy. Obstacle-aware policies output an action considering both agents and obstacles. Current obstacle-aware algorithms, mainly based on Lidar sensor data, struggle to handle collision avoidance around complex obstacles. To resolve this issue, this paper investigates how to find a better way to travel around diverse obstacles. In particular, we present a global map assisted collision avoidance algorithm which, following the lead of a high-level goal guide and using an obstacle representation called distance map, considers other agents and obstacles simultaneously. Moreover, our model can be loaded into each agent individually, making it applicable to large maps or more agents. Simulation results indicate that our model outperforms the state-of-the-art algorithms, showing in scenarios with complex obstacles. We present a notion for incorporating global information in decentralized decision-making, along with a method for extending agent-level algorithms to cluttered environments in real-world scenarios.

Re-Iocalizing a camera from a single image in a previously mapped area is vital for many computer vision applications in robotics and augmented/virtual reality. In this work, we address the problem of estimating the 6 DoF camera pose relative to a global frame from a single image. We propose to leverage a novel network of relative spatial and temporal geometric constraints to guide the training of a Deep Network for Localization. We employ simultaneously spatial and temporal relative pose constraints that are obtained not only from adjacent camera frames but also from camera frames that are distant in the spatio-temporal space of the scene. We show that our method, through these constraints, is capable of learning to localize when little or very sparse ground-truth 3D coordinates are available. In our experiments, this is less than 1 % of available ground-truth data. We evaluate our method on 3 common visual localization datasets and show that it outperforms other direct pose estimation methods.

We present a novel framework for global localization and guided relocalization of a vehicle in an unstructured environment. Compared to existing methods, our pipeline does not rely on cues from urban fixtures (e.g., lane markings, buildings), nor does it make assumptions that require the vehicle to be navigating on a road network. Instead, we achieve localization in both urban and non-urban environments by robustly associating and registering the vehicle's local semantic object map with a compact semantic reference map, potentially built from other viewpoints, time periods, and/or modalities. Robustness to noise, outliers, and missing objects is achieved through our graph-based data association algorithm. Further, the guided relocalization capability of our pipeline mitigates drift inherent in odometry-based localization after the initial global localization. We evaluate our pipeline on two publicly-available, real-world datasets to demonstrate its effectiveness at global localization in both non-urban and urban environments. The Katwijk Beach Planetary Rover dataset [1] is used to show our pipeline's ability to perform accurate global localization in unstructured environments. Demonstrations on the KITTI dataset [2] achieve an average pose error of 3.8 m across all 35 localization events on Sequence 00 when localizing in a reference map created from aerial images. Compared to existing works, our pipeline is more general because it can perform global localization in unstructured environments using maps built from different viewpoints.

An accurate and uncertainty-aware 3D human body pose estimation is key to enabling truly safe but efficient human-robot interactions. Current uncertainty-aware methods in 3D human pose estimation are limited to predicting the uncertainty of the body posture, while effectively neglecting the body shape and root pose. In this work, we present GloPro, which to the best of our knowledge the first framework to predict an uncertainty distribution of a 3D body mesh including its shape, pose, and root pose, by efficiently fusing visual clues with a learned motion model. We demonstrate that it vastly outperforms state-of-the-art methods in terms of human trajectory accuracy in a world coordinate system (even in the presence of severe occlusions), yields consistent uncertainty distributions, and can run in real-time. Our code will be released upon acceptance at https: II Igithub. coml smartroboticslab/GloPro.

3D mapping is crucial for many applications in robotics and related industries. To build dense high-quality point clouds accurate depth estimation or completion is needed. This paper presents the development of a metric-semantic mapping pipeline based on Deep Neural Networks (DNN) which assures geometrical consistency with enhancements for chal-lenging environments with transparent and reflecting objects like glass walls, doors, and mirrors. The suggested approach uses camera ego-motion alongside its sparse visual features to avoid the scale ambiguity issue caused by monocular depth affine-invariant estimations and to able to restore metric consistent depth information. Visual-inertial odometry data is used for camera pose graph optimization with no need to use RGB-D cameras. The proposed pipeline incorporates semantic segmentation and robust filtering to refine point clouds by removing outliers associated with mirrors and glass surfaces. Latency-aware performance and quality evaluation of 3D scene reconstruction were carried out on both a specially prepared dataset that reflects office-like scenes with multiple transparent objects and a public ScanNet dataset. The quantitative and qualitative results show that the proposed solution outperforms other state-of-art DNN-based models and algorithms as well as RGB-D cameras in terms of metric depth geometric consistency, 3D reconstruction accuracy, and the ability to preserve mesh quality in challenging scenarios with transparent and reflective surfaces.

Inertia-dominated mechanical systems can achieve net displacement by 1) periodically changing their shape (known as kinematic gait) and 2) adjusting their inertia distribution to utilize the existing nonzero net momentum (known as momentum gait). Therefore, finding the gait that most effectively utilizes the two types of locomotion in terms of the magnitude of the net momentum is a significant topic in the study of locomotion. For kinematic locomotion with zero net momentum, the geometry of optimal gaits is expressed as the equilibria of system constraint curvature flux through the surface bounded by the gait, and the cost associated with executing the gait in the metric space. In this paper, we identify the geometry of optimal gaits with nonzero net momentum effects by lifting the gait description to a time-parameterized curve in shape-time space. We also propose the variational gait optimization algorithm corresponding to the lifted geometric structure, and identify two distinct patterns in the optimal motion, determined by whether or not the kinematic and momentum gaits are concentric. The examples of systems with and without fluid-added mass demonstrate that the proposed algorithm can efficiently solve forward and turning locomotion gaits in the presence of nonzero net momentum. At any given momentum and effort limit, the proposed optimal gait that takes into account both momentum and kinematic effects outperforms the reference gaits that each only considers one of these effects.