IROS 2022 论文列表

Drones have shown to be useful aerial vehicles for unmanned transport missions such as food and medical supply delivery. This can be leveraged to deliver life-saving nutrition and medicine for people in emergency situations. However, commercial drones can generally only carry 10 %–30 % of their own mass as payload, which limits the amount of food delivery in a single flight. One novel solution to noticeably increase the food-carrying ratio of a drone, is recreating some structures of a drone, such as the wings, with edible materials. We thus propose a drone, which is no longer only a food-transporting aircraft, but itself is partially edible, increasing its food-carrying mass ratio to 50 %, owing to its edible wings. Furthermore, should the edible drone be left behind in the environment after performing its task in an emergency situation, it will be more biodegradable than its non-edible counterpart, leaving less waste in the environment. Here we describe the choice of materials and scalable design of edible wings, and validate the method in a flight-capable prototype that can provide 300 kcal and carry a payload of 80 g of water.

Compared to their rigid counterparts, soft material robotic systems offer great advantages when it comes to flexibility and adaptability. Despite their advantages, modeling of soft systems is still a challenging task, due to the continuous and often highly nonlinear nature of deformation these systems exhibit. Tasks like motion planning or design optimization of soft robots require computationally cheap models of the system's behavior. In this paper we address this need by deriving operational point dependent Cosserat rod models from detailed volume finite element models (FEM). While the latter offer detailed simulations, they generally come with high computational burden that hinders them from being used in time critical model-based methods like motion planning or control. Basic Cosserat rod models promise to provide computationally efficient mechanical models of soft continuum robots. By using a detailed FE model in an offline stage to identify operational point dependent Cosserat rod models, we bring together the accuracy of volumetric FEM with the efficiency of Cosserat rod models. We apply the approach to a fiber reinforced soft pneumatic bending actuator module (SPA module) and evaluate the model's predictive capabilities for a single module as well as a two-module robot.

Current RGB-based 6D object pose estimation methods have achieved noticeable performance on datasets and real world applications. However, predicting 6D pose from single 2D image features is susceptible to disturbance from changing of environment and textureless or resemblant object surfaces. Hence, RGB-based methods generally achieve less competitive results than RGBD-based methods, which deploy both image features and 3D structure features. To narrow down this performance gap, this paper proposes a framework for 6D object pose estimation that learns implicit 3D information from 2 RGB images. Combining the learned 3D information and 2D image features, we establish more stable correspondence between the scene and the object models. To seek for the methods best utilizing 3D information from RGB inputs, we conduct an investigation on three different approaches, including Early-Fusion, Mid-Fusion, and Late-Fusion. We ascertain the Mid-Fusion approach is the best approach to restore the most precise 3D keypoints useful for object pose estimation. The experiments show that our method outperforms state-of-the-art RGB-based methods, and achieves comparable results with RGBD-based methods.

Learning-based visual odometry (VO) has gained increasing popularity in autonomous navigation of small robots. However, most methods in the category require computation resources not normally available on edge systems. We contend that specialized hardware accelerators are ideal solutions to this problem because of their superior energy efficiency. In this paper, we first propose a model to derive compute specifications for VO from physical characteristics of unmanned aerial vehicles (UAVs). These specifications serve as the basis to guide our accelerator design process. Based on the specifications derived from the DJI Mavic Air 2 and Crazyflie 2.0 UAVs, we explore the speed/flight-time design spaces for three target VO algorithms on two NVIDIA Jetson systems. Then, we propose a hardware accelerator architecture and present prototype implementations based on FPGAs. Additionally, we illustrate the algorithm/hardware co-design approach with a series of hardware-aware algorithmic redesigns targeting the FPGA prototypes, and quantify the throughput-accuracy tradeoff of them. Our FPGA implementation of DFVO is 2.7x more energy efficient compared to off-the-shelf embedded computers.

In this paper, we present a framework to assess the quality of a pedestrian detector in an autonomous driving scenario. To do this, we exploit performance metrics from the domain of computer vision on one side and so-called threat metrics from the motion planning domain on the other side. Based on a reachability analysis that accounts for the uncertainty in future motions of other traffic participants, we can determine the worst-case threat from the planning domain and relate it to the corresponding detection from the visual input. Our evaluation results for a RetinaNet on the Argoverse 1.1 [1] dataset show that already a rather simple threat metric such as time-to-collision (TTC) allows to select potentially dangerous interactions between the ego vehicle and a pedestrian when purely vision-based detections fail, even if they are passed to a subsequent object tracker. In addition, our results show that two different DNNs (Deep Neural Networks) with comparable performance differ significantly in the number of critical scenarios that we can identify with our method.

We present a Visual-Inertial Odometry (VIO) algorithm with multiple non-overlapping monocular cameras aiming at improving the robustness of the VIO algorithm. An initialization scheme and tightly-coupled bundle adjustment for multiple non-overlapping monocular cameras are proposed. With more stable features captured by multiple cameras, VIO can maintain stable state estimation, especially when one of the cameras tracked unstable or limited features. We also address the high CPU usage rate brought by multiple cameras by proposing a GPU-accelerated frontend. Finally, we use our pedestrian carried system to evaluate the robustness of the VIO algorithm in several challenging environments. The results show that the multi-camera setup yields significantly higher estimation robustness than a monocular system while not increasing the CPU usage rate (reducing the CPU resource usage rate and computational latency by 40.4% and 50.6% on each camera). A demo video can be found at https://youtu.be/r7QvPth1m10.

There has been a drastic shift to remote interaction for professional, industrial and personal interactions. Improving the overall quality of these interactions by removing any sense of distance between the users is the ultimate goal. Video conferencing has been widely adopted as an improvement to audio-only interactions. Having added visuals to audio communication, the next frontier is to add physical interaction to this remote communication. In this paper, we present an avatar system with the aim of tackling these necessities. The proposed system includes both hardware and software designs to ensure a real-time telemanipulation experience with tactile force feedback. We present a coupled hydrostatic actuated gripper and glove with high system bandwidth to reduce the inherent latency of the mechanical system. To account for latency over the network, the wave variable based method is adopted to maintain the stability of the closed-loop gripper control even under hundreds of milliseconds of delay. A bidirectional audiovisual communication system comprised of off-the-shelf hardware and software is incorporated to allow realtime conversation between the operator and the recipient for collaborative tasks. the proposed system has been validated in lab experiments and the global ana avatar xprize challenge semifinal.

Research attention on natural user interfaces (NUIs) for drone flights are rising. Nevertheless, NUIs are highly diversified, and primarily evaluated by different physical environments leading to hard-to-compare performance between such solutions. We propose a virtual environment, namely VRFlightSim, enabling comparative evaluations with enriched drone flight details to address this issue. We first replicated a state-of-the-art (SOTA) interface and designed two tasks (crossing and pointing) in our virtual environment. Then, two user studies with 13 participants demonstrate the necessity of VRFlightSim and further highlight the potential of open-data interface designs.

As Liszt once said “(a virtuoso) must call up scent and blossom, and breathe the breath of life”, a virtuoso plays the piano with passion, poetry, and extraordinary technical ability. Hence, piano playing, being a task that is quintessentially human, becomes a hallmark for roboticians and artificial intelligence researchers to pursue. In this paper, we advocate an end-to-end reinforcement learning (RL) paradigm to demonstrate how an agent can learn directly from machine-readable music score to play the piano with touch-augmented dexterous hands on a simulated piano. To achieve the desired tasks, we design useful touch- and audio-based reward functions and a series of tasks. Empirical results show that the RL agent can not only find the correct key position but also deal with the various rhythmic, volume, and fingering requirements. As a result, the agent demonstrates its effectiveness in playing simple pieces that have different musical requirements which show the potential of leveraging reinforcement learning approach for the piano playing tasks.

Robots and artificial agents that interact with humans should be able to do so without bias and inequity, but facial perception systems have notoriously been found to work more poorly for certain groups of people than others. In our work, we aim to build a system that can perceive humans in a more transparent and inclusive manner. Specifically, we focus on dynamic expressions on the human face, which are difficult to collect for a broad set of people due to privacy concerns and the fact that faces are inherently identifiable. Furthermore, datasets collected from the Internet are not necessarily representative of the general population. We address this problem by offering a Sim2Real approach in which we use a suite of 3D simulated human models that enables us to create an auditable synthetic dataset covering 1) underrepresented facial expressions, outside of the six basic emotions, such as confusion; 2) ethnic or gender minority groups; and 3) a wide range of viewing angles that a robot may encounter a human in the real world. By augmenting a small dynamic emotional expression dataset containing 123 samples with a synthetic dataset containing 4536 samples, we achieved an improvement in accuracy of 15% on our own dataset and 11 % on an external benchmark dataset, compared to the performance of the same model architecture without synthetic training data. We also show that this additional step improves accuracy specifically for racial minorities when the architecture's feature extraction weights are trained from scratch.

High-Definition (HD) maps are needed for robust navigation of autonomous vehicles, limited by the on-board storage capacity. To solve this, we propose a novel framework, Environment-Aware Normal Distributions Transform (EA-NDT), that significantly improves compression of standard NDT map representation. The compressed representation of EA-NDT is based on semantic-aided clustering of point clouds resulting in more optimal cells compared to grid cells of standard NDT. To evaluate EA-NDT, we present an open-source implementation that extracts planar and cylindrical primitive features from a point cloud and further divides them into smaller cells to represent the data as an EA-NDT HD map. We collected an open suburban environment dataset and evaluated EA-NDT HD map representation against the standard NDT representation. Compared to the standard NDT, EA-NDT achieved consistently at least 1.5× higher map compression while maintaining the same descriptive capability. Moreover, we showed that EA-NDT is capable of producing maps with significantly higher descriptivity score when using the same number of cells than the standard NDT.

Designing or learning an autonomous driving policy is undoubtedly a challenging task as the policy has to maintain its safety in all corner cases. In order to secure safety in autonomous driving, the ability to detect hazardous situations, which can be seen as an out-of-distribution (OOD) detection problem, becomes crucial. However, conventional datasets often only contain expert driving demonstrations, although some non-expert or uncommon driving behavior data are needed to implement a safety guaranteed autonomous driving platform. To this end, we present a dataset called the R3 Driving Dataset, composed of driving data with different qualities. The dataset categorizes abnormal driving behaviors into eight categories and 369 different detailed situations. The situations include dangerous lane changes and near-collision situations. To further enlighten how these abnormal driving behaviors can be detected, we utilize different uncertainty estimation and anomaly detection methods for the proposed dataset. From the results of the proposed experiment, it can be inferred that by using both uncertainty estimation and anomaly detection, most of the abnormal cases in the proposed dataset can be discriminated. https://rllab-snu.github.io/projects/R3-Driving-Dataset/doc.html

Autonomous navigation of a robot in agricultural fields is essential for every task from crop monitoring to weed management and fertilizer application. Many current approaches rely on accurate GPS, however, such technology is expensive and can be impacted by lack of coverage. As such, autonomous navigation through sensors that can interpret their environment (such as cameras) is important to achieve the goal of autonomy in agriculture. In this paper, we introduce a purely vision-based navigation scheme that is able to reliably guide the robot through row-crop fields using computer vision and signal processing techniques without manual intervention. Independent of any global localization or mapping, this approach is able to accurately follow the crop-rows and switch between the rows, only using onboard cameras. The proposed navigation scheme can be deployed in a wide range of fields with different canopy shapes in various growth stages, creating a crop agnostic navigation approach. This was completed under various illumination conditions using simulated and real fields where we achieve an average navigation accuracy of 3.82cm with minimal human intervention (hyper-parameter tuning) on BonnBot-I.

Surface grading is an integral part of the construction pipeline. Here, a bulldozer, which is a key machinery tool at any construction site, is required to level an uneven area containing pre-dumped sand piles. In this work, we aim to tackle the problem of autonomous surface grading on real-world scenarios. We design both a realistic physical simulation and a scaled real-world prototype environment mimicking real bulldozer dynamics and sensory information. In addition, we establish heuristics and learning strategies in order to solve the problem. Through extensive experiments, we show that although heuristics are capable of tackling the problem in a clean and noise-free simulated environment, they fail catastrophically when facing real-world scenarios. However, we show that the simulation can be leveraged to guide a learning agent, which can generalize and solve the task both in simulation and in a scaled prototype environment.

The ability to track surgical instruments in realtime is crucial for autonomous Robotic Assisted Surgery (RAS). Recently, the fusion of visual and kinematic data has been proposed to track surgical instruments. However, these methods assume that both sensors are equally reliable, and cannot successfully handle cases where there are significant perturbations in one of the sensors' data. In this paper, we address this problem by proposing an enhanced fusion-based method. The main advantage of our method is that it can adjust fusion weights to adapt to sensor perturbations and failures. Another problem is that before performing an autonomous task, these robots have to be repetitively recalibrated by a human for each new patient to estimate the transformations between the different robotic arms. To address this problem, we propose a self-calibration algorithm that empowers the robot to autonomously calibrate the transformations by itself in the beginning of the surgery. We applied our fusion and selfcalibration algorithms for autonomous ultrasound tissue scanning and we showed that the robot achieved stable ultrasound imaging when using our method. Our performance evaluation shows that our proposed method outperforms the state-of-art both in normal and challenging situations.

Although the soft robot is gaining considerable popularity in dexterous and safe manipulation, accurate motion control is still an open problem to be explored. Recent investigations suggest that reinforcement learning (RL) is a promising solution but lacks efficient adaptability for Sim2Real transfer or environment variations. In this paper, we present a deep deterministic policy gradient (DDPG)-based control system for the continuous task-space manipulation of soft robots. Domain randomization is adopted in simulation for fast control-policy initialization, while an offline retraining strategy is utilized to update the controller parameters for incremental learning. The experiments demonstrate that the proposed RL controller can track a moving target accurately (with RMSE of 1.26 mm), and accommodate to external varying load effectively (with ~30% RMSE reduction after retraining). Comparisons among the proposed RL controller and other supervised-learning-based controllers in handling additional tip load were also conducted. The results support that our RL method is appropriate for automatic learning such that there is no need of manual interference for data processing, particularly in cases with external disturbances and actuation redundancy.

The performance of a camera network monitoring a set of targets depends crucially on the configuration of the cameras. In this paper, we investigate the reconfiguration strategy for the parameterized camera network model, with which the sensing qualities of the multiple targets can be optimized globally and simultaneously. We first propose to use the number of pixels occupied by a unit-length object in image as a metric of the sensing quality of the object, which is determined by the parameters of the camera, such as intrinsic, extrinsic, and distortional coefficients. Then, we form a single quantity that measures the sensing quality of the targets by the camera network. This quantity further serves as the objective function of our optimization problem to obtain the optimal camera configuration. We verify the effectiveness of our approach through extensive simulations and experiments, and the results reveal its improved performance on the AprilTag detection tasks. Codes and related utilities for this work are open-sourced and available at https://github.com/sszxc/MultiCam-Simulation.

Complex needle shape prediction remains an issue for planning of surgical interventions of flexible needles. In this paper, we validate a theoretical method for flexible needle shape prediction allowing for non-uniform curvatures, extending upon a previous sensor-based model which combines curvature measurements from fiber Bragg grating (FBG) sensors and the mechanics of an inextensible elastic rod to determine and predict the 3D needle shape during insertion. We evaluate the model's effectiveness in single-layer isotropic tissue for shape sensing and shape prediction capabilities. Experiments on a four-active area, FBG-sensorized needle were performed in varying single-layer isotropic tissues under stereo vision to provide 3D ground truth of the needle shape. The results validate a viable 3D needle shape prediction model accounting for non-uniform curvatures in flexible needles with mean needle shape sensing and prediction root-mean-square errors of 0.479 mm and 0.892 mm, respectively.