IROS 2022 论文列表

We propose DeepFusion, a modular multi-modal architecture to fuse lidars, cameras and radars in different combinations for 3D object detection. Specialized feature extractors take advantage of each modality and can be exchanged easily, making the approach simple and flexible. Extracted features are transformed into bird's-eye-view as a common representation for fusion. Spatial and semantic alignment is performed prior to fusing modalities in the feature space. Finally, a detection head exploits rich multi-modal features for improved 3D detection performance. Experimental results for lidar-camera, lidar-camera-radar and camera-radar fusion show the flexibility and effectiveness of our fusion approach. In the process, we study the largely unexplored task of faraway car detection up to 225 meters, showing the benefits of our lidar-camera fusion. Furthermore, we investigate the required density of lidar points for 3D object detection and illustrate implications at the example of robustness against adverse weather conditions. Moreover, ablation studies on our camera-radar fusion highlight the importance of accurate depth estimation.

Ultra-Wide-Band (UWB) ranging sensors have been widely adopted for robotic navigation thanks to their extremely high bandwidth and hence high resolution. However, off-the-shelf devices may output ranges with significant errors in cluttered, severe non-line-of-sight (NLOS) environments. Recently, neural networks have been actively studied to improve the ranging accuracy of UWB sensors using the channel-impulse-response (CIR) as input. However, previous works have not systematically evaluated the efficacy of various packet types and their possible combinations in a two-way-ranging transaction, including poll, response and final packets. In this paper, we firstly investigate the utility of different packet types and their combinations when used as input for a neural network. Secondly, we propose two novel data-driven approaches, namely FMCIR and WMCIR, that leverage two-sided CIRs for efficient UWB error mitigation. Our approaches outperform state-of-the-art by a significant margin, further reducing range errors up to 45%. Finally, we create and release a dataset of transaction-level synchronized CIRs (each sample consists of the CIR of the poll, response and final packets), which will enable further studies in this area.

Blimps are well suited to perform long-duration aerial tasks as they are energy efficient, relatively silent and safe. To address the blimp navigation and control task, in previous work we developed a hardware and software-in-the-loop framework and a PID-based controller for large blimps in the presence of wind disturbance. However, blimps have a deformable structure and their dynamics are inherently non-linear and time-delayed, making PID controllers difficult to tune. Thus, often resulting in large tracking errors. Moreover, the buoyancy of a blimp is constantly changing due to variations in ambient temperature and pressure. To address these issues, in this paper we present a learning-based framework based on deep residual reinforcement learning (DRRL), for the blimp control task. Within this framework, we first employ a PID controller to provide baseline performance. Subsequently, the DRRL agent learns to modify the PID decisions by interaction with the environment. We demonstrate in simulation that DRRL agent consistently improves the PID performance. Through rigorous simulation experiments, we show that the agent is robust to changes in wind speed and buoyancy. In real-world experiments, we demonstrate that the agent, trained only in simulation, is sufficiently robust to control an actual blimp in windy conditions. We openly provide the source code of our approach at https://github.com/robot-perception-group/AutonomousBlimpDRL. Video demonstration is provided at https://youtu.be/EMC4KnlH0yI.

This paper presents a Deep Reinforcement Learning based navigation approach in which we define the occu-pancy observations as heuristic evaluations of motion primitives, rather than using raw sensor data. Our method enables fast mapping of the occupancy data, generated by multi-sensor fusion, into trajectory values in 3D workspace. The computationally efficient trajectory evaluation allows dense sampling of the action space. We utilize our occupancy observations in different data structures to analyze their effects on both training process and navigation performance. We train and test our methodology on two different robots within challenging physics-based simulation environments including static and dy-namic obstacles. We benchmark our occupancy representations with other conventional data structures from state-of-the-art methods. The trained navigation policies are also validated successfully with physical robots in dynamic environments. The results show that our method not only decreases the required training time but also improves the navigation performance as compared to other occupancy representations. The open-source implementation of our work and all related info are available at https://github.com/RIVeR-Lab/tentabot.

Object rearranging is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a goal configuration. Previous studies focus on designing an expert system for each specific task by model-based or data-driven approaches and the application scenarios are therefore limited. Some research has been attempting to design a general framework to obtain more advanced manipulation capabilities for deformable rearranging tasks, with lots of progress achieved in simulation. However, transferring from simulation to reality is difficult due to the limitation of the end-to-end CNN architecture. To address these challenges, we design a local GNN (Graph Neural Network) based learning method, which utilizes two representation graphs to encode keypoints detected from images. Self-attention is applied for graph updating and cross-attention is applied for generating manipulation actions. Extensive experiments have been conducted to demonstrate that our framework is effective in multiple 1-D (rope, rope ring) and 2-D (cloth) rearranging tasks in simulation and can be easily transferred to a real robot by fine-tuning a keypoint detector.

When humans socially interact with another agent (e.g., human, pet, or robot) through touch, they do so by applying varying amounts of force with different directions, locations, contact areas, and durations. While previous work on touch gesture recognition has focused on the spatio-temporal distribution of normal forces, we hypothesize that the addition of shear forces will permit more reliable classification. We present a soft, flexible skin with an array of tri-axial tactile sensors for the arm of a person or robot. We use it to collect data on 13 touch gesture classes through user studies and train a Convolutional Neural Network (CNN) to learn spatio-temporal features from the recorded data. The network achieved a recognition accuracy of 74% with normal and shear data, compared to 66% using only normal force data. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes.

Predicting future paths of vehicles or pedestrians is an essential task for automated vehicles to allow for planning the own trajectory. Using predicted paths, a planning algorithm can, e.g., react to anticipated manoeuvres of other traffic participants. For calculating risks of planned manoeuvres, it is essential that the predicted paths are generated with information about their uncertainty. Since today's state of the art trajectory prediction algorithms are based on deep neural networks (DNNs), the estimation of uncertainty is left to the neural networks as well, which usually provide no means of assessing how the uncertainty estimation works. In this paper, we present a combination of DNNs with Gaussian processes via Deep Kernel Learning (DKL), which combines the ability of DNNs to perform the prediction task with the advantage of Gaussian processes of having more interpretable probabilistic outputs. We propose and evaluate two different variants for the task of multimodal trajectory prediction using Stochastic Variational Gaussian Processes (SVGPs) and the recently proposed regression method Deep Sigma Point Processes (DSPPs), respectively. We evaluate the predictive distributions of both approaches on the publicly available Argoverse Motion Forecasting dataset and compare them to other, purely neural network based methods for uncertainty estimation.

Body language such as conversational gesture is a powerful way to ease communication. Conversational gestures do not only make a speech more lively but also contain semantic meaning that helps to stress important information in the discussion. In the field of robotics, giving conversational agents (humanoid robots or virtual avatars) the ability to properly use gestures is critical, yet remain a task of extraordinary difficulty. This is because given only a text as input, there are many possibilities and ambiguities to generate an appropriate gesture. Different to previous works we propose a new method that explicitly takes into account the gesture types to reduce these ambiguities and generate human-like conversational gestures. Key to our proposed system is a new gesture database built on the TED dataset that allows us to map a word to one of three types of gestures: “Imagistic” gestures, which express the content of the speech, “Beat” gestures, which emphasize words, and “No gestures.” We propose a system that first maps the words in the input text to their corresponding gesture type, generate type-specific gestures and combine the generated gestures into one final smooth gesture. In our comparative experiments, the effectiveness of the proposed method was confirmed in user studies for both avatar and humanoid robot.

Going beyond the traditional sparse multi-channel peripheral human-machine interface that has been used widely in neurorobotics, high-density surface electromyography (HD-sEMG) has shown significant potential for decoding upper-limb motor control. We have recently proposed heterogeneous temporal dilation of LSTM in a deep neural network architecture for a large number of gestures (>60), securing spatial resolution and fast convergence. However, several fundamental questions remain unanswered. One problem targeted explicitly in this paper is the issue of “electrode shift,” which can happen specifically for high-density systems and during doffing and donning the sensor grid. Another real-world problem is the question of transient versus plateau classification, which connects to the temporal resolution of neural interfaces and seamless control. In this paper, for the first time, we implement gesture prediction on the transient phase of HD-sEMG data while robustifying the human-machine interface decoder to electrode shift. For this, we propose the concept of deep data augmentation for transient HD-sEMG. We show that without using the proposed augmentation, a slight shift of 10mm may drop the decoder's performance to as low as 20%. Combining the proposed data augmentation with a 3D Convolutional Neural Network (CNN), we recovered the performance to 84.6% while securing a high spatiotemporal resolution, robustifying to the electrode shift, and getting closer to large-scale adoption by the end-users, enhancing resiliency.

Within the field of multi-robot systems, developing systems that rely only on onboard sensing without the use of external infrastructure (e.g. GNSS) has many potential applications. However, relying only on visual-based modalities for localization presents challenges in terms of accuracy and reliability. We introduce a decentralized multi-robot lateral velocity estimation method for Unmanned Aerial Vehicles (UAVs) to improve onboard measurements in case GNSS infrastructure is not available. This method relies on sharing the onboard measurements of neighbors, as well as the estimation of the relative motion of a focal UAV within the swarm, based on observation of coworking robots. The proposed velocity estimation method does not rely on centralized communication to achieve high reliability and scalability within the swarm system. The performance of the state estimation approach has been verified in simulations and real-world experiments. The results have shown that a swarm of UAVs using the proposed velocity estimator can stabilize individual robots when their primary onboard localization source is not reliable enough.

This paper presents an algorithm for a team of mobile robots to simultaneously learn a spatial field over a domain and spatially distribute themselves to optimally cover it. Drawing from previous approaches that estimate the spatial field through a centralized Gaussian process, this work leverages the spatial structure of the coverage problem and presents a decentralized strategy where samples are aggregated locally by establishing communications through the boundaries of a Voronoi partition. We present an algorithm whereby each robot runs a local Gaussian process calculated from its own measurements and those provided by its Voronoi neighbors, which are incorporated into the individual robot's Gaussian process only if they provide sufficiently novel information. The performance of the algorithm is evaluated in simulation and compared with centralized approaches.

We propose a decentralized control algorithm for a minimalistic robotic swarm with limited capabilities such that the desired global behavior emerges. We consider the problem of searching for and encapsulating various targets present in the environment while avoiding collisions with both static and dynamic obstacles. The novelty of this work is the guaranteed generation of desired complex swarm behavior with constrained individual robots which have no memory, no localization, and no knowledge of the exact relative locations of their neighbors. Moreover, we analyze how the emergent behavior changes with different parameters of the task, noise in the sensor reading, and asynchronous execution.

Multi-robot systems have been shown to build large-scale, user-specified structures using distributed, environmentally-mediated coordination in simulation. Little attention, however, has been devoted to error propagation and mitigation. In this paper, we introduce a detailed simulation of TERMES, a prototypical construction system, in which robots have realistic error profiles. We use this simulator and 32 randomly generated 250-brick blueprints to show that action errors can have significant long-term effects. We study the spatio-temporal error distribution and introduce and characterize the efficacy of a simple decay-based error correction mechanism. Although inefficient, this type of error correction is promising because it can be performed by robots with the same limited sensory capabilities as those who place bricks. To limit the impact on the construction rate, we also examine decay mechanisms informed by spatial and temporal error distributions. The incorporation of decay in our building process increases the probability of successful completion by ~ 4, at the expense of ~1/4 decrease in construction rate.

Designing the Kalman filter for nonlinear robot systems with theoretical guarantees is challenging, especially when the dynamics model is unavailable. This paper proposes a data-driven Kalman filter algorithm using kernel-based Koop-man operators for unknown nonlinear robot systems. First, the Koopman operator using sparse kernel-based extended dynamic decomposition (EDMD) is presented to learn the unknown dynamics with input-output datasets. Unlike classic EDMD, which requires manual selection of kernel functions, our approach automatically constructs kernel functions using an approximate linear dependency analysis method. The resulting Koopman model is a linear dynamic evolution in the kernel space, enabling us to address the nonlinear filtering problem using the standard linear Kalman filter design process. Despite this, our approach generates a nonlinear filtering law thanks to the adopted nonlinear kernel functions. Finally, the effectiveness of the proposed approach is validated by simulated experiments.

Although the average healthy adult transitions from sit to stand over 60 times per day, most research on powered prosthesis control has only focused on walking. In this paper, we present a data-driven controller that enables sitting, standing, and walking with minimal tuning. Our controller comprises two high level modes of sit/stand and walking, and we develop heuristic biomechanical rules to control transitions. We use a phase variable based on the user's thigh angle to parameterize both walking and sit/stand motions, and use variable impedance control during ground contact and position control during swing. We extend previous work on data-driven optimization of continuous impedance parameter functions to design the sit/stand control mode using able-bodied data. Experiments with a powered knee-ankle prosthesis used by a participant with above-knee amputation demonstrate promise in clinical outcomes, as well as trade-offs between our minimal-tuning approach and accommodation of user preferences. Specifically, our controller enabled the participant to complete the sit/stand task 20% faster and reduced average asymmetry by half compared to his everyday passive prosthesis. The controller also facilitated a timed up and go test involving sitting, standing, walking, and turning, with only a mild (10%) decrease in speed compared to the everyday prosthesis. Our sit/stand/walk controller enables multiple activities of daily life with minimal tuning and mode switching.

Cable-Driven Parallel Robots (CDPRs) have been proposed for a variety of applications such as material handling, rehabilitation, and instrumentation. However, the collision-free constraint of CDPRs limits the workspace of CDPRs and the feasible position of anchor points. To address the collision-free constraint of CDPRs, a data-driven kinematic control scheme is developed for CDPRs, enabling a CDPR to control its pose even if suffering collisions between a cable and the base or the end-effector. To deal with the collisions, the data-driven kinematic control scheme utilizes a motion model obtained based on data samples of the motion of the CDPR, rather than the Jacobian matrix of the CDPR, to map a control law in the task space to the time derivative of the length of cables in the joint space. To evaluate the effectiveness of the developed data-driven kinematic control scheme, experiments of controlling a suspended CDPR with two cables allowing collisions are conducted.

Accurate LiDAR-camera extrinsic calibration is a precondition for many multi-sensor systems in mobile robots. Most calibration methods rely on laborious manual operations and calibration targets. While working online, the calibration methods should be able to extract information from the environment to construct the cross-modal data association. Convolutional neural networks (CNNs) have powerful feature extraction ability and have been used for calibration. However, most of the past methods solve the extrinsic as a regression task, without considering the geometric constraints involved. In this paper, we propose a novel end-to-end extrinsic calibration method named DXQ-Net, using a differentiable pose estimation module for generalization. We formulate a probabilistic model for LiDAR-camera calibration flow, yielding a prediction of uncertainty to measure the quality of LiDAR-camera data association. Testing experiments illustrate that our method achieves a competitive with other methods for the translation component and state-of-the-art performance for the rotation component. Generalization experiments illustrate that the generalization performance of our method is significantly better than other deep learning-based methods.

Object manipulation in cluttered scenes is a difficult and important problem in robotics. To efficiently manipulate objects, it is crucial to understand their surroundings, especially in cases where multiple objects are stacked one on top of the other, preventing effective grasping. We here present DUQIM-Net, a decision-making approach for object manipulation in a setting of stacked objects. In DUQIM-Net, the hierarchical stacking relationship is assessed using Adj-Net, a model that leverages existing Transformer Encoder-Decoder object detectors by adding an adjacency head. The output of this head probabilistically infers the underlying hierarchical structure of the objects in the scene. We utilize the properties of the adjacency matrix in DUQIM-Net to perform decision making and assist with object-grasping tasks. Our experimental results show that Adj-Net surpasses the state-of-the-art in object-relationship inference on the Visual Manipulation Relationship Dataset (VMRD), and that DUQIM-Net outperforms comparable approaches in bin clearing tasks.

Ergonomics and human comfort are essential concerns in physical human-robot interaction applications. Defining an accurate and easy-to-use ergonomic assessment model stands as an important step in providing feedback for postural correction to improve operator health and comfort. Common practical methods in the area suffer from inaccurate ergonomics models in performing postural optimization. In order to retain assessment quality, while improving computational considerations, we propose a novel framework for postural assessment and optimization for ergonomically intelligent physical human-robot interaction. We introduce DULA and DEBA, differentiable and continuous ergonomics models learned to replicate the popular and scientifically validated RULA and REBA assessments with more than 99% accuracy. We show that DULA and DEBA provide assessment comparable to RULA and REBA while providing computational benefits when being used in postural optimization. We evaluate our framework through human and simulation experiments. We highlight DULA and DEBA's strength in a demonstration of postural optimization for a simulated pHRI task.