IROS 2021 论文列表

We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding high bandwidth raw data streaming via the robots real-time data bus. The hardware acceleration allows fast detection and localization of objects based on finger-camera images and provides input for a grasping controller. To this end, we implement a resource-aware encoder-decoder Convolutional Neural Network (CNN) for pixel-wise object segmentation and run inference on the in-hand embedded system at 3.58 GOPS. We evaluate the system, consisting of the soft hand with in-finger vision and the in-hand FPGA-accelerated CNN in several experiments on the humanoid robot ARMAR-6. Specifically, we evaluate the overall system response time, the ability to perform precision grasps and test reactivity and reliability that are required for handover actions. We obtain an overall system response time of 154 ms for catching a falling object and obtain a success rate of 90 % reliability for the power drill handover tasks. Further, we successfully demonstrate ability of dexterous grasping and manipulation of a pencil from a cup.

Camera anomalies like rain or dust can severely degrade image quality and its related tasks, such as localization and segmentation. In this work we address this important issue by implementing a pre-processing step that can effectively mitigate such artifacts in a real-time fashion, thus supporting the deployment of autonomous systems with limited compute capabilities. We propose a shallow generator with aggregation, trained in an adversarial setting to solve the ill-posed problem of reconstructing the occluded regions. We add an enhancer to further preserve high-frequency details and image colorization. We also produce one of the largest publicly available datasets1 to train our architecture and use realistic synthetic raindrops to obtain an improved initialization of the model. We benchmark our framework on existing datasets and on our own images obtaining state-of-the-art results while enabling real-time performance, with up to 40x faster inference time than existing approaches.

This work considers the problem of generating computationally efficient quadrotor motion primitives between a given pose (position, velocity, and acceleration) and a goal plane in the presence of obstacles. A new motion primitive tool based on the logistic curve is proposed and a closed-form analytic approach is developed to satisfy constraints on starting pose, goal plane, velocity, acceleration, and jerk. The geometric obstacle avoidance problem is represented as a combinatorial set problem and a heuristic approach is proposed to accelerate the solution search. Numerical examples are presented to highlight the fast motion primitive generation in multi-obstacle pose-to-plane scenarios.

In this work, we fully define the existing relationships between traditional optimality criteria and the connectivity of the underlying pose-graph in Active SLAM, characterizing, therefore, the connection between Graph Theory and the Theory Optimal Experimental Design. We validate the proposed relationships in 2D and 3D graph SLAM datasets, showing a remarkable relaxation of the computational load when using the graph structure. Furthermore, we present a novel Active SLAM framework which outperforms traditional methods by successfully leveraging the graphical facet of the problem so as to autonomously explore an unknown environment.

Classic Minimally Invasive Surgery (MIS) is an ergonomic burden for assistants and surgeons. The former need to adopt uncomfortable positions for hours while holding a camera to track the latter’s gestures inside the patient. This incurs assistant’s muscle fatigue which can lead to tremor or drift of the video feedback. A backdrivable robotic holder can be attached to this device in order to compensate its weight. This allows the user to place the camera at a desired position which the robot will steadily keep once he/she releases it. However, endoscopic cameras present difficult-to-model accessories whose gravity parameters can change during the same surgery. If these changes are not foreseen by the gravity model of the robot this results in a fall of the endoscope each time it is released. Therefore, it is desired to firstly detect if there is a fall in order to be able to correct it. In this article a fall detection method for a comanipulated robotic endoscope holder is proposed. It evaluates smoothness of the robot end effector trajectory to identify whether the user manipulates the instrument or it has been released and poorly compensated. An experiment was carried out with 10 subjects where 240 releases of the endoscope were performed while it was poorly compensated. The algorithm succeeded to detect the falls with sensitivity up to 99.17%.

We need robots more aware of the unintended outcomes of their actions for ensuring safety. This can be achieved by an onboard failure detection system to monitor and detect such cases. Onboard failure detection is challenging with a limited set of onboard sensor setup due to the limitations of sensing capabilities of each sensor. To alleviate these challenges, we propose FINO-Net, a novel multimodal sensor fusion based deep neural network to detect and identify manipulation failures. We also introduce FAILURE, a multimodal dataset, containing 229 real-world manipulation data recorded with a Baxter robot. Our network combines RGB, depth and audio readings to effectively detect failures. Results indicate that fusing RGB with depth and audio modalities significantly improves the performance. FINO-Net achieves %98.60 detection accuracy on our novel dataset. Code and data are publicly available at https://github.com/ardai/fino-net.

Projecting the point cloud on the 2D spherical range image transforms the LiDAR semantic segmentation to a 2D segmentation task on the range image. However, the LiDAR range image is still naturally different from the regular 2D RGB image; for example, each position on the range image encodes the unique geometry information. In this paper, we propose a new projection-based LiDAR semantic segmentation pipeline that consists of a novel network structure and an efficient post-processing step. In our network structure, we design a FID (fully interpolation decoding) module that directly upsamples the multi-resolution feature maps using bilinear interpolation. Inspired by the 3D distance interpolation used in PointNet++, we argue this FID module is a 2D version distance interpolation on (θ, ϕ) space. As a parameter-free decoding module, the FID largely reduces the model complexity by maintaining good performance. Besides the network structure, we empirically find that our model predictions have clear boundaries between different semantic classes. This makes us rethink whether the widely used K-nearest-neighbor post-processing is still necessary for our pipeline. Then, we realize the many-to-one mapping causes the blurring effect that some points are mapped into the same pixel and share the same label. Therefore, we propose to process those occluded points by assigning the nearest predicted label to them. This NLA (nearest label assignment) post-processing step shows a better performance than KNN with faster inference speed in the ablation study. On SemanticKITTI dataset, our pipeline achieves the best performance among all projection-based methods with 64×2048 resolution and all point-wise solutions. With a ResNet-34 as the backbone, both the training and testing of our model can be finished on a single RTX 2080 Ti with 11G memory. The code is released here.1

The RGB-Thermal (RGB-T) information for semantic segmentation has been extensively explored in recent years. However, most existing RGB-T semantic segmentation usually compromises spatial resolution to achieve real-time inference speed, which leads to poor performance. To better extract detail spatial information, we propose a two-stage Feature-Enhanced Attention Network (FEANet) for the RGB-T semantic segmentation task. Specifically, we introduce a Feature-Enhanced Attention Module (FEAM) to excavate and enhance multi-level features from both the channel and spatial views. Benefited from the proposed FEAM module, our FEANet can preserve the spatial information and shift more attention to high-resolution features from the fused RGB-T images. Extensive experiments on the urban scene dataset demonstrate that our FEANet outperforms other state-of-the-art (SOTA) RGB-T methods in terms of objective metrics and subjective visual comparison (+2.6% in global mAcc and +0.8% in global mIoU). For the 480 × 640 RGB-T test images, our FEANet can run with a real-time speed on an NVIDIA GeForce RTX 2080 Ti card.

The development of aerial autonomy has enabled aerial robots to fly agilely in complex environments. However, dodging fast-moving objects in flight remains a challenge, limiting the further application of unmanned aerial vehicles (UAVs). The bottleneck of solving this problem is the accurate perception of rapid dynamic objects. Recently, event cameras have shown great potential in solving this problem. This paper presents a complete perception system including ego-motion compensation, object detection, and trajectory prediction for fast-moving dynamic objects with low latency and high precision. Firstly, we propose an accurate ego-motion compensation algorithm by considering both rotational and translational motion for more robust object detection. Then, for dynamic object detection, an event camera-based efficient regression algorithm is designed. Finally, we propose an optimization-based approach that asynchronously fuses event and depth cameras for trajectory prediction. Extensive real-world experiments and benchmarks are performed to validate our framework. Moreover, our code will be released to benefit related researches.

With over 10 million people currently suffering from significant long-term gait disability in the United States only, robot-assisted rehabilitation and wearable devices are increasingly gaining attention as a mean to regain functional mobility. Since these devices work collaborative and synchronously with the human gait, it is necessary to be able to detect gait events, such as heel-strikes, in real-time. Although many algorithms have been proposed for detecting heel-strikes with either wearable (e.g. Inertial Measurement Units (IMUs)) or non-wearable (e.g. force plates) sensors, there is a great need for employing less obtrusive and reliable sensors that rely only on recording the kinematics of the leg motion. This work proposes a novel and efficient kinematic algorithm, called the Foot VErtical & Sagittal Position Algorithm (F-VESPA), which has several advantages over existing methods. First, it accurately estimates heel-strike events using kinematic data without requiring access to future data points, rendering it the first to our knowledge kinematic algorithm capable of real-time implementation during treadmill walking. Moreover, it does not require tuning of the utilized parameters, rendering it robust to different subjects, conditions and equipment. The algorithm is tested in a large set of subjects across various treadmill speeds, and it is shown to outperform online and offline implementations of existing prominent kinematic algorithms. Using a 150 Hz data collection system, the F-VESPA achieved a total true error of 33 ms (median) in detecting heel-strike. The F-VESPA is the first to our knowledge kinematic algorithm that can detect heel-strike events during treadmill walking in real-time, with high accuracy, robustness and fast response, enabling real-time control of a variety of assistive platforms and devices, among others.

Simultaneous Localization and Mapping (SLAM) has wide robotic applications such as autonomous driving and unmanned aerial vehicles. Both computational efficiency and localization accuracy are of great importance towards a good SLAM system. Existing works on LiDAR based SLAM often formulate the problem as two modules: scan-to-scan match and scan-to-map refinement. Both modules are solved by iterative calculation which are computationally expensive. In this paper, we propose a general solution that aims to provide a computationally efficient and accurate framework for LiDAR based SLAM. Specifically, we adopt a non-iterative two-stage distortion compensation method to reduce the computational cost. For each scan input, the edge and planar features are extracted and matched to a local edge map and a local plane map separately, where the local smoothness is also considered for iterative pose optimization. Thorough experiments are performed to evaluate its performance in challenging scenarios, including localization for a warehouse Automated Guided Vehicle (AGV) and a public dataset on autonomous driving. The proposed method achieves a competitive localization accuracy with a processing rate of more than 10 Hz in the public dataset evaluation, which provides a good trade-off between performance and computational cost for practical applications. It is one of the most accurate and fastest open-sourced SLAM systems1 in KITTI dataset ranking.

In this paper, we study existing flocking models and propose extensions to improve their abilities to deal with environments having obstacles impacting the communication quality. Often depicted as robust systems, there is yet a lack of understanding how flocking models compare and how they are impacted by the communication quality when they exchange control data. We extend two standard models to improve their ability to stay connected while evolving in environments with different obstacles distributions. By taking into account the radio propagation, we model the obstacles impact on communications in a simulator that we use to optimize flocking parameters. The simulation results show the efficiency of the proposed models and how they adapt to different environmental constraints.

Being able to refer to an object, a person, or a place in a non-ambiguous manner is a need when one has to achieve collaborative activities with a partner. This is the so-called Referring Expression Generation (REG) problem. While widely used for Human-Robot Interaction, state of the art approaches restrict its use to the current environment. We propose a novel extension to the REG which takes full advantage of the Human-Robot shared knowledge about past actions as additional information to generate Referring Expressions. We show that our approach is usable with a domain-independent ontology as a knowledge base and that it can also use a semantic representation of past activity to generate RE. We illustrate our method through simulated situations and discuss its efficiency and pertinence.

We present a systematic approach called Extended VINS-Mono to utilize VINS-Mono, a state-of-the-art monocular visual-inertial relative localization method, targeting practical vehicle localization in large-scale outdoor road environments. Our proposed fusion approach associates multiple independent localization methods and provides multiple (projected) state estimates in a desired coordinate system to satisfy different accuracy, rate and latency requirements. We extend VINS-Mono with absolute localization methods like GNSS and relative localization methods like Kalman-filter-based INS to provide global state estimation for navigation/routing and local state estimation for planning/control. Additionally, Extended VINS-Mono addresses two significant drawbacks in VINS-Mono for use in large-scale outdoor road environments. First, motion on an almost planar road surface will make scale unobservable in VINS-Mono. Secondly, moving objects in dynamic scenarios will degrade accuracy. We handle the scale estimation problem of VINS-Mono by extending its (re-)initialization process with speed readings and introducing a speed factor for use with graph optimization. A dynamic feature-point filter method with masks from DNN-based object detection handles dynamic environments and re-collects feature points on stationary objects like parked cars. Better global accuracy is obtained with Extended VINS-Mono, compared to VINS-Mono, in a 25 km-trip journey through highways, tunnels, urban areas and suburban areas in Pittsburgh. Thus, Extended VINS-Mono can be used for reliable and accurate absolute localization in dynamic road environments. We also evaluate the accuracy, localization rate and latency of multiple (projected) state estimates in the global coordinate system from multiple localization methods. Our fusion method is therefore able to satisfy different localization requirements of various tasks on an intelligent vehicle.

Humans display the remarkable ability to sense the world through tools and other held objects. For example, we are able to pinpoint impact locations on a held rod and tell apart different textures using a rigid probe. In this work, we consider how we can enable robots to have a similar capacity, i.e., to embody tools and extend perception using standard grasped objects. We propose that vibro-tactile sensing using dynamic tactile sensors on the robot fingers, along with machine learning models, enables robots to decipher contact information that is transmitted as vibrations along rigid objects. This paper reports on extensive experiments using the BioTac micro-vibration sensor and a new event dynamic sensor, the NUSkin, capable of multi-taxel sensing at 4 kHz. We demonstrate that fine localization on a held rod is possible using our approach (with errors less than 1 cm on a 20 cm rod). Next, we show that vibro-tactile perception can lead to reasonable grasp stability prediction during object handover, and accurate food identification using a standard fork. We find that multi-taxel vibro-tactile sensing at a sufficiently high sampling rate (above 2 kHz) led to the best performance across the various tasks and objects. Taken together, our results provide both evidence and guidelines for using vibro-tactile perception to extend tactile perception, which we believe will lead to enhanced competency with tools and better physical human-robot interaction.

Trajectory tracking in the orientation space utilizing unit quaternions yields non linear error dynamics as opposed to Cartesian position. In this work, we study trajectory tracking in the orientation space utilizing the most popular quaternion error representations and angular velocity errors. By selecting error functions carefully we show exponential convergence in a region of attraction containing large initial errors. We further show that under certain conditions frequently en-countered in practice, the formulation respecting the geometric characteristics of the quaternion manifold and its tangent space yields linear tracking dynamics allowing us to guarantee a desired tracking performance by gain selection without tuning. Simulation and experimental results are provided.

For many tasks, including table tennis, catching, and sword fighting, a critical step is intercepting the incoming object with a robot arm or held tool. Solutions to robot arm interception via learning, specifically reinforcement learning (RL), have become prevalent, as they provide robust solutions to the robot arm interception problem, even for high degree of freedom robotic systems. Despite numerous solutions, there has been little exploration into the factors of learning that impact solution quality. Thus, there is little insight into what problem features lead to better learning success. In this paper, we explore the parameters that impact solution quality. We find that link position observations outperform joint angle observations in terms of learning speed, performance, ability to utilize more than one frame of observation, and generalization to situations not trained for. These results are immediately applicable to RL for robot arm interception tasks.

We present a learning-based planner that aims to robustly drive a vehicle by mimicking human drivers’ driving behavior. We leverage a mid-to-mid approach that allows us to manipulate the input to our imitation learning network freely. With that in mind, we propose a novel feedback synthesizer for data augmentation. It allows our agent to gain more driving experience in various previously unseen environments that are likely to encounter, thus improving overall performance. This is in contrast to prior works that rely purely on random synthesizers. Furthermore, rather than completely commit to imitating, we introduce task losses that penalize undesirable behaviors, such as collision, off-road, and so on. Unlike prior works, this is done by introducing a differentiable vehicle rasterizer that directly converts the waypoints output by the network into images. This effectively avoids the usage of heavyweight ConvLSTM networks, therefore, yields a faster model inference time. About the network architecture, we exploit an attention mechanism that allows the network to reason critical objects in the scene and produce better interpretable attention heatmaps. To further enhance the safety and robustness of the network, we add an optional optimization-based post-processing planner improving the driving comfort. We comprehensively validate our method’s effectiveness in different scenarios that are specifically created for evaluating self-driving vehicles. Results demonstrate that our learning-based planner achieves high intelligence and can handle complex situations. Detailed ablation and visualization analysis are included to further demonstrate each of our proposed modules’ effectiveness in our method.

All robots create consequential sound—sound produced as a result of the robot’s mechanisms—yet little work has explored how sound impacts human-robot interaction. Recent work shows that the sound of different robot mechanisms affects perceived competence, trust, human-likeness, and discomfort. However, the physical sound characteristics responsible for these perceptions have not been clearly identified. In this paper, we aim to explore key characteristics of robot sound that might influence perceptions. A pilot study from our past work showed that quieter and higher-pitched robots may be perceived as more competent and less discomforting. To better understand how variance in these attributes affects perception, we performed audio manipulations on two sets of industrial robot arm videos within a series of four new studies presented in this paper. Results confirmed that quieter robots were perceived as less discomforting. In addition, higher-pitched robots were perceived as more energetic, happy, warm, and competent. Despite the robot’s industrial purpose and appearance, participants seemed to prefer more "cute" (or "kawaii") sound profiles, which could have implications for the design of more acceptable and fulfilling sound profiles for human-robot interactions with practical collaborative robots.