IROS 2023 论文列表

The recent increase in data availability and reliability has led to a surge in the development of learning-based model predictive control (MPC) frameworks for robot systems. Despite attaining substantial performance improvements over their non-learning counterparts, many of these frameworks rely on an offline learning procedure to synthesize a dynamics model. This implies that uncertainties encountered by the robot during deployment are not accounted for in the learning process. On the other hand, learning-based MPC methods that learn dynamics models online are computationally expensive and often require a significant amount of data. To alleviate these shortcomings, we propose a novel learning-enhanced MPC framework that incorporates components from C1 adaptive control into learning-based MPC. This integration enables the accurate compensation of both matched and unmatched uncertainties in a sample-efficient way, enhancing the control performance during deployment. In our proposed framework, we present two variants and apply them to the control of a quadrotor system. Through simulations and physical experiments, we demonstrate that the proposed framework not only allows the synthesis of an accurate dynamics model on-the-fly, but also significantly improves the closed-loop control performance under a wide range of spatio-temporal uncertainties.

Line information from urban structures can be exploited as an additional geometrical feature to achieve robust vision-based simultaneous localization and mapping (SLAM) systems in textureless scenes. Sometimes, however, conventional line tracking methods fail to track caused by image blur or occlusion. Even though these lost line features are just a subset of plenty of features, the failure in feature tracking can potentially lead to performance degradation of the SLAM system, particularly in textureless environments. To tackle this problem, we propose a robust line-tracking method for line-based monocular visual-inertial odometry. The proposed method generates a semi-dense map composed of depth and sparsity mesh using estimated 3D features. By leveraging the semi-dense map, our method performs a range-adaptive epipo-lar search to match the lines, allowing for robust line tracking while simultaneously reducing false positives. Furthermore, an algorithm to avoid conflicts is proposed, which occurs when the tracked lines from consecutive matching do not accord with the lines matched by our method. This algorithm discriminately maintains line features while appropriately aggregating lines spread across multiple frames. As evaluated in the EuRoC dataset and a more challenging textureless corridor scene, our proposed method shows substantial performance increases compared with other line-based visual (-inertial) approaches.

Robots operating in human-centered environments, such as retail stores, restaurants, and households, are often required to distinguish between similar objects in different contexts with a high degree of accuracy. However, fine-grained object recognition remains a challenge in robotics due to the high intra-category and low inter-category dissimilarities. In addition, the limited number of fine-grained 3D datasets poses a significant problem in addressing this issue effectively. In this paper, we propose a hybrid multi-modal Vision Transformer (ViT) and Convolutional Neural Networks (CNN) approach to improve the performance of fine-grained visual classification (FGVC). To address the shortage of FGVC 3D datasets, we generated two synthetic datasets. The first dataset consists of 20 categories related to restaurants with a total of 100 instances, while the second dataset contains 120 shoe instances. Our approach was evaluated on both datasets, and the results indicate that our hybrid multi-modal model outperforms both CNN-only and ViT-only baselines, achieving a recognition accuracy of 94.50% and 93.51% on the restaurant and shoe datasets, respectively. Additionally, we have made our FGVC RGB-D datasets available to the research community to enable further experimentation and advancement. Furthermore, we integrated our proposed method with a robot framework and demonstrated its potential as a fine-grained perception tool in both simulated and real-world robotic scenarios.

Future robotics systems aiming for true autonomy must be robust against dynamic and unstructured environments. The 5th generation (5G) mobile network is expected to provide ubiquitous, reliable and low-latency wireless communications to ground robots, especially in outdoor scenarios. Empowered by 5G, the digital transformation of robotics is emerging, enabled by the cloud-native paradigm and the adoption of edge-computing principles for heavy computational task offloading. However, wireless link quality fluctuates due to multiple aspects such as the topography of the deployment area, the presence of obstacles, robots' movement and the configuration of the serving base stations. This directly impacts not only the connectivity to the robots but also the performance of robot operations, resulting in severe challenges when targeting full robot autonomy. To address such challenges, in this paper, we propose a framework to build a semantic map based on radio quality. By means of our proposed approach, mobile robots can gain knowledge on up-to-date radio context map information of the surrounding environment, hence enabling reliable and efficient robotics operations.

Recently, robot navigation methods using human instructions have been actively studied, including visual language navigation. Although language is one of the most promising forms of instruction, words often contain ambiguities. To complement this problem, we propose to use geometric instruction as a clue to the task goal. Specifically, in our proposed system, we assume that the robot receives a rough position of the target from human gesture. The robot adaptively estimates the reliability of this geometric instruction, and switches between exploration and instruction-following modes depending on the reliability value. We conducted evaluation of our method using a 3D simulation environment, and show that the task success rate and other metrics improve compared with the baseline methods.

This paper presents an effective approach to enable performance improvement in human-robot collaboration scenarios. The problem is tackled from the perspective of speed and separation monitoring principle, which stems from the recently instituted safety standard. The proposed approach attempts to seek for performance gains, measured by the speed-up of the production cycle, without compromising the safety constraints consistent with the standard. The approach is based on the notion of braking surface - an abstraction of the swept volume described by the manipulator during braking motion. We address two types of braking behavior: general and path-consistent. In both cases, the braking surface can be evaluated in a receding horizon manner. The robot velocity is continuously scaled such that, in case of a controlled stop, the corresponding volume spanned by the robot (braking surface) does not interfere with the surrounding obstacles. The approach is entirely kinematic and does not require the knowledge of the robot's dynamic model. Simulation study indicates that the pro-posed approach offers performance improvements compared to other state of the art methods. Moreover, the experiments demonstrate the real-time applicability of the method with the real robot in human-shared environment.

Recently, many research studies have been carried out on using deep learning methods for 3D point cloud understanding. However, there is still no remarkable result on 3D point cloud semantic segmentation compared to those of 2D research. One important reason is that 3D data has higher dimensionality but lacks large datasets, which means that the deep learning model is difficult to optimize and easy to overfit. To overcome this, an essential method is to provide more priors to the learning of deep models. In this paper, we focus on semantic segmentation for point clouds in the real world. To provide priors to the model, we propose a novel loss function called Linearity and Planarity to enhance local feature consistency in the regions with similar local structure. Experiments show that the proposed method improves baseline performance on both indoor and outdoor datasets e.g. S3DIS and Semantic3D.

Autonomous Marine Vehicles (AMVs) have gained interest for scientific and commercial applications, including pipeline and algae bloom monitoring, contaminant tracking, and ocean debris removal. The Team Orienteering Problem (TOP) is relevant in this context as Multi-Robot Systems (MRSs) allow for better coverage of the area of interest, simultaneous data collection at different locations, and an increase in the overall robustness and efficiency of the mission. However, route planning for AMVs in dynamic ocean environments is challenging due to the coupling of environmental and vehicle dynamics. We propose a multi-objective formulation that accounts for the trade-offs between visiting multiple task locations and energy consumption by the vehicles subject to a time budget. This work focuses on vehicles that can maintain a constant net speed but can be adapted to vehicles with constant thrust. Different from existing approaches, our method is able to leverage time-varying ocean currents to improve the energy efficiency of resulting routes. We validate our approach experimentally by superimposing ocean flow models with benchmark instances of the TOP.

We address a coordination problem for a team of heterogeneous and energy-limited agents to achieve cooperative tasks given as team-level spatio-temporal specifications. We assume that agents have stochastic energy dynamics and do not have identical capabilities. We define the team-level specification using Signal Temporal Logic (STL) with integral predicates, which can express tasks that can be completed collectively in an asynchronous way. We first abstract the environment as a graph using sampling-based methods. This abstraction includes the possible paths of different types of agents and ensures the availability of recharging within a certain distance. Then, we formulate a mixed-integer program over this abstraction to find the high-level paths of the agents. Finally, we steer the agents in the environment according to the nominal plan under stochastic energy consumption models and a recharging policy. Such stochastic energy dynamics cause deviations from the nominal plan and delays in completing the tasks. Accordingly, we define and evaluate the expected delay (temporal relaxation) in achieving the STL specification under the proposed solution.

For multi-agent area coverage path planning problem, existing researches regard it as a combination of Traveling Salesman Problem (TSP) and Coverage Path Planning (CPP). However, these approaches have disadvantages of poor observation ability in online phase and high computational cost in offline phase, making it difficult to be applied to energy-constrained Unmanned Aerial Vehicles (UAVs) and adjust strategy dynamically. In this paper, we decompose the task into two sub-problems: multi-agent path planning and sub-region CPP. We model the multi-agent path planning problem as a Collective Markov Decision Process (C-MDP), and design an Energy Constrained Multi-Agent Reinforcement Learning (ECMARL) algorithm based on the centralized training and distributed execution concept. Taking into account energy constraint of UAVs, the UAV propulsion power model is established to measure the energy consumption of UAVs, and load balancing strategy is applied to dynamically allocate target areas for each UAV. If the UAV is under energy-depleted situation, ECMARL can adjust the mission strategy in real time according to environmental information and energy storage conditions of other UAVs. When UAVs reach each sub-region of interest, Back-an-Forth Paths (BFPs) are adopted to solve CPP problem, which can ensure full coverage, optimality and complexity of the sub-problem. Comprehensive theoretical analysis and experiments demonstrate that ECMARL is superior to the traditional offline TSP-CPP strategy in terms of solution quality and computational time, and can effectively deal with the energy-constrained UAVs.

Legged locomotion is arguably the most suited and versatile mode to deal with natural or unstructured terrains. Intensive research into dynamic walking and running controllers has recently yielded great advances, both in the optimal control and reinforcement learning (RL) literature. Hopping is a challenging dynamic task involving a flight phase and has the potential to increase the traversability of legged robots. Model based control for hopping typically relies on accurate detection of different jump phases, such as lift-off or touch down, and using different controllers for each phase. In this paper, we present a end-to-end RL based torque controller that learns to implicitly detect the relevant jump phases, removing the need to provide manual heuristics for state detection. We also extend a method for simulation to reality transfer of the learned controller to contact rich dynamic tasks, resulting in successful deployment on the robot after training without parameter tuning.

Point cloud registration (PCR) aims to recover the rigid transformation between two noisy, unordered point sets. This task is typically tackled by establishing point-wise correspondences, and solving the rigid transformation between the two sets. Since descriptor-based methods find correspondences by matching the feature space distance, a powerful and rotation-robust point feature extractor is critical to the success of this task. Existing methods assume soft rotation invariance/equivariance through the means of training augmentation, rotational discretization or pre-alignment of patches. In contrast, this paper proposes a new method which generates fully rotation invariant and equivariant descriptors by construction. For each keypoint patch, our network extracts not only a rotation invariant descriptor for establishing corre-spondences, but also a rotation equivariant one. The rotation equivariant descriptor allows relative transformation to be directly recovered from a single correspondence pair, unlike standard methods that require three correspondences. This design significantly reduces iteration number of RANSAC and guarantees high registration recall when the inlier ratio of estimated correspondences is low. Extensive experiments have demonstrated that the proposed method outperforms state-of-art methods in the same category even after much fewer RANSAC iterations.

Needle picking is a challenging manipulation task in robot-assisted surgery due to the characteristics of small slender shapes of needles, needles' variations in shapes and sizes, and demands for millimeter-level control. Prior works, heavily relying on the prior of needles (e.g., geometric models), are hard to scale to unseen needles' variations. In this paper, we present the first end- to-end learning method to train deep visuomotor policy for needle picking. Concretely, we propose DreamerfD to maximally leverage demonstrations to improve the learning efficiency of a state-of-the-art model-based reinforcement learning method, DreamerV2; Since Variational Auto-Encoder (VAE) in DreamerV2 is difficult to scale to high-resolution images, we propose Dynamic Spotlight Adaptation to represent control-related visual signals in a low-resolution image space; Virtual Clutch is also proposed to reduce per-formance degradation due to significant error between prior and posterior encoded states at the beginning of a rollout. We conducted extensive experiments in simulation to evaluate the performance, robustness, in-domain variation adaptation, and effectiveness of individual components of our method. Our method, trained by 8k demonstration timesteps and 140k online policy timesteps, can achieve a remarkable success rate of 80%. Furthermore, our method effectively demonstrated its superiority in generalization to unseen in-domain variations including needle variations and image disturbance, highlighting its robustness and versatility. Codes and videos are available at https://sites.google.com/view/DreamerfD.

Trajectory sampling in the Frenet(road-aligned) frame, is one of the most popular methods for motion planning of autonomous vehicles. It operates by sampling a set of behavioral inputs, such as lane offset and forward speed, before solving a trajectory optimization problem conditioned on the sampled inputs. The sampling is handcrafted based on simple heuristics, does not adapt to driving scenarios, and is oblivious to the capabilities of downstream trajectory planners. In this paper, we propose an end-to-end learning of behavioral input distribution from expert demonstrations or in a self-supervised manner. We embed a novel differentiable trajectory optimizer as a layer in neural networks, allowing us to update behavioral inputs by considering the optimizer's feedback. Moreover, our end-to-end approach also ensures that the learned behavioral inputs aid the convergence of the optimizer. We improve the state-of-the-art in the following aspects. First, we show that learned behavioral inputs substantially decrease collision rate while improving driving efficiency over handcrafted approaches. Second, our approach outperforms model predictive control methods based on sampling-based optimization.

Soft robotic devices have been popular in handling intricate grasping and dexterous manipulation tasks, serving as an alternative to conventional, rigid end-effectors. These devices are relatively simple, lightweight, and cost-effective. Recently, kirigami based structures have been used to create low-cost and disposable soft robotic grippers and hands. These grippers undergo a complex post-contact reconfiguration and conform to an object's shape and size during grasping. In this paper, we explore this new class of soft robotic grippers by utilising them for single-grasp object classification and grasping force estimation. We install simplistic sensors on both the gripper and the actuation system to estimate the state of the kirigami gripper, and the collected data features are employed to train Random Forest models for identifying the grasped object. The classifier trained exhibits a high accuracy of 98 % in discriminating objects of various shapes. When handling food items, the classifier achieves an accuracy of 94 %, while in classifying transparent objects, the classifier obtained again a high accuracy of 97 %. Finally, object-specific force estimation models are triggered based on the classification decision of the Random Forest model to estimate the grasping force exerted by the gripper. These positive outcomes demonstrate the kirigami based robotic gripper's potential for object classification in a variety of circumstances, particularly where vision systems are not available or not reliable.

Backchanneling models, designed to enhance the interactive capabilities of robots, have primarily been trained on human-human interaction data. However, applying such data directly to social robots raises concerns due to dissimilarities in the way humans and robots exhibit verbal and nonverbal behaviors, particularly in the domain of emotional backchannels. This research aims to address this gap by conducting an exploratory study on the differences in human backchanneling behaviors during interactions with humans and social robots in various emotional contexts (e.g., happy and sad). Our findings reveal significant variations in emotionally specific backchannels between human-human and human-robot interactions under different emotional contexts. These results highlight the importance of designing backchanneling models that are tailored for human-robot interactions.

Though they are compliant, nimble and morpho-logically intelligent, fluidic soft robots often rely on bulky components for power and actuation. This work contributes a design methodology which enables development of soft fluidic robots that move in a sequenced fashion, enabling lightweight devices with embodied intelligence. Bezier-curved beams were introduced as a design building block whose antagonistic placement results in Representative Auxetic Element (RAE) that can be patterned on inflatable shells. Kinematics and loading behaviour of these design building blocks were studied through Finite Element Analysis (FEA). We give a methodology for patterning RAEs on cylindrical and conic shells to create soft fluidic components that move (motion components) and those that delay fluid flow (pinch components). We verify the physical concepts governing the design methodology through two prototype devices that produce sequenced motion under a single fluidic input. Devices using this framework have the potential to perform complicated sequenced motions with lightweight control components.

Tracking multiple moving objects of interest (OOI) with multiple robot systems (MRS) has been addressed by active sensing that maintains a shared belief of OOIs and plans the motion of robots to maximize the information quality. Mobility of robots enables the behavior of pursuing better visibility, which is constrained by sensor field of view (FoV) and occlusion objects. We first extend prior work to detect, maintain and share occlusion information explicitly, allowing us to generate occlusion-aware planning even if à priori semantic occlusion information is unavailable. The efficacy of active sensing approaches is often evaluated according to estimation error and information gain metrics. However, these metrics do not directly explain the level of cooperative behavior engendered by the active sensing algorithms. Next, we extract different emergent cooperative behaviors that stem from the same underlying algorithms but manifest differently under differing scenarios. In particular, we highlight and demonstrate three emergent behavior patterns in active sensing MRS: (i) Change of tracking responsibility between agents when tracking trajectories with divergent directions or due to a re-allocation of the resource among heterogeneous agents; (ii) Awareness of occlusions to a trajectory and temporal leave-and-return of the sensing agent; (iii) Sharing of local occlusion objects in MRS that subsequently improves the awareness of occlusion.

Open-world Instance Segmentation (OIS) is a challenging task that aims to accurately segment every object instance appearing in the current observation, regardless of whether these instances have been labeled in the training set. This is important for safety-critical applications such as robust autonomous navigation. In this paper, we present a flexible and effective OIS framework for LiDAR point cloud that can accurately segment both known and unknown instances (i.e., seen and unseen instance categories during training). It first identifies points belonging to known classes and removes the back-ground by leveraging close-set panoptic segmentation networks. Then, we propose a novel ellipsoidal clustering method that is more adapted to the characteristic of LiDAR scans and allows precise segmentation of unknown instances. Furthermore, a diffuse searching method is proposed to handle the common over-segmentation problem presented in the known instances. With the combination of these techniques, we are able to achieve accurate segmentation for both known and unknown instances. We evaluated our method on the SemanticKITTI open-world LiDAR instance segmentation dataset. The experimental results suggest that it outperforms current state-of-the-art methods, especially with a 10.0% improvement in association quality. The source code of our method will be publicly available at https://github.com/nubot-nudt/ElC-OIS.