IROS 2023 论文列表

In a future with autonomous robots, visual and spatial perception is of utmost importance for robotic systems. Particularly for aerial robotics, there are many applications where utilizing visual perception is necessary for any real-world scenarios. Robotic aerial grasping using drones promises fast pick-and-place solutions with a large increase in mobility over other robotic solutions. Utilizing Mask R-CNN scene segmentation (detectron2), we propose a vision-based system for autonomous rapid aerial grasping which does not rely on mark-ers for object localization and does not require the appearance of the object to be previously known. Combining segmented images with spatial information from a depth camera, we generate a dense point cloud of the detected objects and perform geometry - based grasp planning to determine grasping points on the objects. In real-world experiments on a dynamically grasping aerial platform, we show that our system can replicate the performance of a motion capture system for object local-ization up to 94.5 % of the baseline grasping success rate. With our results, we show the first use of geometry-based grasping techniques with a flying platform and aim to increase the autonomy of existing aerial manipulation platforms, bringing them further towards real-world applications in warehouses and similar environments.††Code: https://github.com/srl-ethz/detectron-realsense

Deep reinforcement learning (DRL) has been widely applied in autonomous exploration and mapping tasks, but often struggles with the challenges of sampling efficiency, poor adaptability to unknown map sizes, and slow simulation speed. To speed up convergence, we combine curriculum learning (CL) with DRL, and first propose a Cumulative Curriculum Reinforcement Learning (CCRL) training framework to alleviate the issue of catastrophic forgetting faced by general CL. Besides, we present a novel state representation, which considers a local egocentric map and a global exploration map resized to the fixed dimension, so as to flexibly adapt to environments with various sizes and shapes. Additionally, for facilitating the fast training of DRL models, we develop a lightweight grid-based simulator, which can substantially accelerate simulation compared to popular robot simulation platforms such as Gazebo. Based on the customized simulator, comprehensive experiments have been conducted, and the results show that the CCRL framework not only mitigates the catastrophic forgetting problem, but also improves the sample efficiency and generalization of DRL models, compared to general CL as well as without a curriculum. Our code is available at https://github.com/BeamanLi/CCRL_Exploration.

Autonomous exploration in an unknown environment is widely studied, and many exploration strategies exist. However, in most of the works, safety is not usually given top priority. The reason behind the violation of safety by most of the exploration algorithms in real-world applications is the ignorance of some or all of the following factors (i) the mathematical model of the robot, (ii) practical constraints on states (like constrained steering angle, speed, etc.) and inputs (like actuator saturation), and (iii) hardware constraints such as sampling time, sensor noise, modelling uncertainties, etc. In this work, we propose an autonomous exploration framework for the 2-D exploration problem that considers the factors above to provide safety guarantees for the robot and the environment. The effectiveness of this method is shown in a high-fidelity simulation of different robots with onboard sensors in different simulation environments and real-world implementation.

Automotive radars have been reliably used in most assisted and autonomous driving systems due to their robustness to extreme weather conditions. With radial velocity measurements from automotive radars, moving targets such as cars, trucks, and buses can be tracked robustly. However, due to the lack of elevation angles in measurements from automotive radars, stationary targets at different heights, such as maintenance holes and bridges, cannot be distinguished. Most autonomous systems rely on sensor fusion or ignore stationary targets to tackle the problem of missing the elevation angle dimension, which derives safety issues. We propose a simple yet effective approach to estimate the elevation angles of stationary targets from relative velocity and radial velocity measurements from an automotive radar. In contrast to structure from motion in computer vision, we utilize the instantaneous velocity generated from the motion of the ego vehicle. The radial velocity of each target is the projection of relative velocity onto the radial direction from radar to target. The radial velocity of each target can be inferred given the target's azimuth, elevation angle, and relative velocity. Accordingly, the elevation angle of each stationary target can be uniquely calculated given the velocity of radar and the target's azimuth and radial velocity measurements. The radar's velocity is estimated with the existing radar odometry algorithm and IMU. The proposed method is verified with real-world data. We evaluate the system's performance with a pre-built point cloud map and a good localization module in a real-world scenario. The proposed elevation angle reconstruction can reach a 1.41-degree mean error and standard deviation of 0.6 degrees in elevation angle.

Many intelligent robots use a combination of radar and camera sensors to capture environmental information. Robust and accurate perception highly relies on the result of multi-sensor calibration. Most current spatial calibration methods require a calibration board or a special marker as the target. In this paper, we provide a novel calibration method for RGBD camera and millimeter-wave radar, which automatically estimates the extrinsic parameters. Our proposed method includes the following two stages: rough extrinsic parameters are estimated by using object contours as geometric constraints, and meanwhile, the optimum is reached via optimizing based on the difference of velocity obtained from camera and radar. It only needs an object moving past sensors, but does not require for a calibration board. We validate our method through simulation experiments and real-world experiments. We construct a simulation environment in CARLA to verify the performance of our proposed method against different angles. Furthermore, different levels of zero mean Gaussian noise are added to evaluate the stability of our method. In addition, real-world experiments with different hardware setups are taken to verify the feasibility of our method in real-world conditions.

Pelvic fractures are one of the most serious traumas in orthopedics, and the technical proficiency and expertise of the surgical team strongly influence the quality of reduction results. With the advancement of information technology and robotics, robot-assisted pelvic fracture reduction surgery is expected to reduce the impact caused by inexperienced doctors and improve the accuracy and stability of pelvic reduction. However, this requires the robot to detect key surgeon actions from time-series data, enabling the robot to independently perceive the surgical status, predict the surgeon's intentions, assess the demonstrated level of professional competence, and assess the progress of the surgery. Therefore, a multi-task deep learning neural network architecture is proposed, which incorporates Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) along with tri-modality fusion and feature extraction techniques. The proposed framework aims to achieve key action detection in closed reduction operations for pelvic fractures. Subsequently, a trimodal fine-grained dataset was constructed, wherein 29, 32, and 14 labels were marked on flexion, position, and pressure data for 14 key closed reduction actions. The experimental results show that the correct detection rate of closed reduction actions is 92.3 %, significantly higher than the commonly used recognition algorithms. This work provides a method for the robot to learn the surgeon's professional knowledge, provides the basis for the operation's motion perception, and contributes to the autonomy of the robot-assisted closed reduction surgery of pelvic fractures.

Gait generation for soft robots is challenging due to the nonlinear dynamics and high dimensional input spaces of soft actuators. Limitations in soft robotic control and perception force researchers to hand-craft open loop controllers for gait sequences, which is a non-trivial process. Moreover, short soft actuator lifespans and natural variations in actuator behavior limit machine learning techniques to settings that can be learned on the same time scales as robot deployment. Lastly, simulation is not always possible, due to heterogeneity and nonlinearity in soft robotic materials and their dynamics change due to wear. We present a sample-efficient, simulation free, method for self-generating soft robot gaits, using very minimal computation. This technique is demonstrated on a motorized soft robotic quadruped that walks using four legs constructed from 16 “handed shearing auxetic” (HSA) actuators. To manage the dimension of the search space, gaits are composed of two sequential sets of leg motions selected from 7 possible primitives. Pairs of primitives are executed on one leg at a time; we then select the best-performing pair to execute while moving on to subsequent legs. This method-which uses no simulation, sophisticated computation, or user input-consistently generates good translation and rotation gaits in as low as 4 minutes of hardware experimentation, outperforming hand-crafted gaits. This is the first demonstration of completely autonomous gait generation in a soft robot.

Tele-operating aerial vehicles without any automated assistance is challenging due to various limitations, especially for inexperienced users. Autocomplete addresses this problem by automatically identifying and completing the user's intended motion. Such a framework uses machine learning to recognize and classify human inputs as one of a set of motion primitives, and then, if the human operator accepts, synthesizes the motion in order to complete the desired motion. This has been shown to improve the performance of the system and reduce operator workload. Previous Autocomplete systems focused on different 2D motions (line, arc, sine,..). However, since most UAVs tasks are in a 3D world, this paper introduces 3D Autocomplete for 3D motions. Moreover, the proposed framework presents just-in-time prediction of the 3D motions by proposing a change point detection technique, which allows the framework to autonomously identify when to conduct a prediction. Also, it deals with variable motion sizes. Real time simulation results show that the proposed framework is capable of predicting the user intentions after change point detection.

Augmented reality (AR) is considered one of the most promising solutions for safer procedures in several surgical specialities. Fusing patient-specific pre-operative information, typically 3D models extracted from CT scans or MRI, with real-time surgical images allows the surgeon to have detailed information on the anatomical structure of the surgical target intra-operatively. The coupling of AR and Robotics represents the next step towards introducing awareness into the surgical room, thus enhancing the surgeon's perceptual, cognitive and manipulative capabilities. This paper presents a novel integrated system for real-time AR navigation in robotic minimally invasive surgery (RMIS), composed of a robotic endoscopic camera, a robotic teleoperation implementing a software-based Remote Center of Motion (RCM), and an AR navigation software based on an initial manual registration of virtual 3D models with the real anatomy. The integrated system, as well as the individual modules, were evaluated in simulated surgical-like setups for accuracy and repeatability. The proposed system can perform high-precision tasks (position accuracy around 1 mm1 mm and AR error lower than 7%), showing potential for application in different surgical procedures and setting the basis for autonomous robotic surgery operations.

A human being can communicate while working at the same time. However, teleoperated humanoid robots that can work and communicate simultaneously are currently too complex and expensive. We propose an “Augmented Avatar” that can perform both work and communication simultaneously at a low cost. The authors have developed two types of Avatars: a Manipulation Avatar with minimal functions tailored for work, and a Communication Avatar that mimics the human body structure to facilitate communication. The aim of this project is to develop an AVATAR system that enables operators to seamlessly operate the two avatars without using a wearable control interface. In this paper, we report on the construction and operation test results of this prototype. The prototype system has been well received in an international AVATAR competition in which 820 teams participated, placing 12th. This paper also discusses operability based on the experience gained at this competition.

We introduce a Learning from Demonstration (LID) approach for contact-rich manipulation tasks, i.e., tasks in which the manipulandum's motion is constrained by contact with the environment. Our approach is motivated by the insight that even a large number of demonstrations will often not contain sufficient information to obtain a general policy for the task. To obtain general policies, our approach augments the information contained in a single demonstration. This autonomous augmentation is based on the insight that environmental constraints play a central role in generalization. We validate our approach in real-world experiments with mechanisms with multiple, interdependent articulations, including latch locks, chain locks, and drawers with handles. The extracted policies, obtained from a single augmented human demonstration, generalize to different mechanisms of the same type and in varying environmental settings.

Facial animation in virtual reality environments is essential for applications that necessitate clear visibility of the user's face and the ability to convey emotional signals. In our scenario, we animate the face of an operator who controls a robotic Avatar system. The use of facial animation is particularly valuable when the perception of interacting with a specific individual, rather than just a robot, is intended. Purely keypoint-driven animation approaches struggle with the complexity of facial movements. We present a hybrid method that uses both keypoints and direct visual guidance from a mouth camera. Our method generalizes to unseen operators and requires only a quick enrolment step with capture of two short videos. Multiple source images are selected with the intention to cover different facial expressions. Given a mouth camera frame from the HMD, we dynamically construct the target keypoints and apply an attention mechanism to determine the importance of each source image. To resolve keypoint ambiguities and animate a broader range of mouth expressions, we propose to inject visual mouth camera information into the latent space. We enable training on large-scale speaking head datasets by simulating the mouth camera input with its perspective differences and facial deformations. Our method outperforms a baseline in quality, capability, and temporal consistency. In addition, we highlight how the facial animation contributed to our victory at the ANA Avatar XPRIZE Finals.

High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: tactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: https://sites.google.com/view/tactile-saliency/.

Cooperative multi-agent problems often require coordination between agents, which can be achieved through a centralized policy that considers the global state. Multi-agent policy gradient (MAPG) methods are commonly used to learn such policies, but they are often limited to problems with low-level action spaces. In complex problems with large state and action spaces, it is advantageous to extend MAPG methods to use higher-level actions, also known as options, to improve the policy search efficiency. However, multi-robot option executions are often asynchronous, that is, agents may select and complete their options at different time steps. This makes it difficult for MAPG methods to derive a centralized policy and evaluate its gradient, as centralized policy always select new options at the same time. In this work, we propose a novel, conditional reasoning approach to address this problem and demonstrate its effectiveness on representative option-based multi-agent cooperative tasks through empirical validation. Find code and videos at: https://sites.google.com/view/mahrlsupp/

Spectral mapping, typically performed at the orbital scale, is hard at the rover scale as it necessitates larger coverage and field operable spectrometers. In this work, we propose using RGB cameras to assist spectral mapping. RGB cameras placed on a wide range of robotic platforms, including aerial vehicles, which can explore large regions compared to ground rovers. Our method uses a spectral model which learns the spatial relationship of spectra in a compressed feature space. We show that RGB data can contribute to this feature space and thereby enhance spectral reconstruction accuracy.

We study the problem of assigning robots with actions to track targets. The objective is to optimize the robot team's tracking quality which can be defined as the reduction in the uncertainty of the targets' states. Specifically, we consider two assignment problems given the different sensing capabilities of the robots. In the first assignment problem, a single robot is sufficient to track a target. To this end, we present a greedy algorithm (Algorithm 1) that assigns a robot with its action to each target. We prove that the greedy algorithm has a 1/2-approximation bound and runs in polynomial time. Then, we study the second assignment problem where two robots are necessary to track a target. We design another greedy algorithm (Algorithm 2) that assigns a pair of robots with their actions to each target. We prove that the greedy algorithm achieves a 1/3-approximation bound and has a polynomial running time. Moreover, we illustrate the performance of the two greedy algorithms in the ROS-Gazebo environment where the tracking patterns of one robot following one target using Algorithm 1 and two robots following one target using Algorithm 2 are clearly observed. Further, we conduct extensive comparisons to demonstrate that the two greedy algorithms perform close to their optimal counterparts and much better than their respective (1/2 and 1/3) approximation bounds.

Optimal placement of charging stations in a workspace is a crucial problem to address, for efficient operation of battery-driven mobile robots. When the battery charge of a robot reaches a certain threshold, the robot must be able to reach a nearby charging station to recharge its battery. In this paper, we deal with two different versions of the optimization problem related to the optimal placement of charging stations in a robot workspace. The first problem is formulated to find an optimal number of charging stations given a battery threshold deciding the need to move to a charging station, and the second problem finds an optimal battery threshold for a given number of charging stations. Both the problems involve finding the locations of charging stations, such that from any obstacle-free location at least one charging station is reachable with at most threshold amount of battery charge remaining with the robot. In this paper, we prove these optimization problems to be NP-hard, i.e., computationally intractable. To handle intractability of the above minimization problems, we design two polynomial-time approximation algorithms to find near-optimal solutions. Our algorithms achieve significantly high scalability without compromising the quality of the solution beyond a certain factor of the optimal solution. Experimental results show that our algorithms run order-of-magnitude faster than a recently proposed Satisfiability Modulo Theory (SMT)-based approach and maintain solution quality within the theoretical bounds on the optimal solution.

This work devises an optimized machine learning approach for human arm pose estimation from a single smart-watch. Our approach results in a distribution of possible wrist and elbow positions, which allows for a measure of uncertainty and the detection of multiple possible arm posture solutions, i.e., multimodal pose distributions. Combining estimated arm postures with speech recognition, we turn the smartwatch into a ubiquitous, low-cost and versatile robot control interface. We demonstrate in two use-cases that this intuitive control interface enables users to swiftly intervene in robot behavior, to temporarily adjust their goal, or to train completely new control policies by imitation. Extensive experiments show that the approach results in a 40% reduction in prediction error over the current state-of-the-art and achieves a mean error of 2.56 cm for wrist and elbow positions.