IROS 2021 论文列表

The increasing interest for autonomous ships has motivated research in numerous areas. One such area is the safe navigation through ice infested waters, for which a sensor instrumentation and automated process are proposed for near-field, sea-ice 3D scanning and mapping using a ship mounted LiDAR, with attitude compensation from inertial and satellite positioning sensors. Data were collected both at the Aalto Ice Tank laboratory and on board the icebreaker S.A. Agulhas II during its voyage to the Antarctic waters. The implemented process enables automated acquisition of detailed 3D point cloud maps, containing highly valuable information for icy waters going ships currently operated by a human crew and, in the near future, supporting the development of autonomous ships. Compared to other methods using satellite, aerial or underwater data, the proposed method is a more cost-effective and easy to integrate solution into current and future icy waters going ships, thus enabling a higher level of situational awareness.

Deep reinforcement learning has achieved significant success in many fields, but will confront sampling efficiency and safety problems when applying to robot control in the real world. Sim-to-real transfer learning was proposed to make use of samples in the simulation and overcome the gap between simulation and real world. In this paper, we focus on improving Progressive Neural Network — an effective sim-to-real learning method, by proposing Interactive Progressive Network Learning (IPNL). IPNL integrates progressive network and interactive reinforcement learning (interactive RL) which learns from evaluative feedback provided by an observing human trainer. We test our method using five RL tasks with discrete or continuous actions in OpenAI Gym and a sinusoids curve following task with AUV simulator on the Gazebo platform. Our results suggest that while Progressive Network has good performance when transferring from tasks with low-dimensional state space to those with high-dimensional one but has little effect for transferring from high-dimensional tasks to low-dimensional ones, IPNL allows an agent to learn a more stable policy with better performance faster for both cases. More importantly, our further analysis indicate that there is a synergy between Progressive Network and interactive RL for improving the agent’s learning. Our results in the path following of AUV shed light on the potential of applying our method in the real world tasks.

Soft robot modeling tends to prioritize soft robot dynamics in order to recover how they might behave. Soft robot design tends to focus on how to use compliant elements with actuation to effect certain canonical movement profiles. For soft robot locomotors, these profiles should lead to locomotion. Naturally, there is a gap between the emphasis of computational modeling and the needs of locomotion design. This paper proposes to consider modeling and computation efforts directed more toward understanding soft robot-world interactions with locomotion in mind. With a SMA-actuated inchworm as the soft robot to model and control, the framework is a combination of shape identification and geometric modeling that culminates in control equations of motion. When applied to the task of gait-based locomotion, the equations operate in a low dimensional shape-based gait space. Simulated and experimentally applied gaits for an inchworm model showed qualitatively similar outcomes, while the measured net displacement per gait cycle coincided within 9%. This result advances the idea that a shape-centric approach to soft robot modeling for control and locomotion may provide predictive locomotive models.

Obstacle detection and avoidance plays a crucial role in autonomous navigation of unmanned ground vehicles. This becomes more challenging in off-road environments due to the higher probability of finding negative obstacles (e.g., holes, ditches, trenches, etc.) compared with on-road environments. One approach to solve this problem is to avoid the candidate path with a negative obstacle, but in off-road avoiding negative obstacles all the time is not possible. In such cases, the path planner may need to choose a candidate path with a negative obstacle that causes the least amount of damage to the vehicle. To deal better with these types of scenarios, this study introduces a novel approach to perform shape estimation of negative obstacles using LiDAR 3D point cloud data. The dimensions (width, diameter, and depth) and the location (center) of negative obstacles are calculated based on estimated shape. This approach is tested on different terrain types using the Mississippi Autonomous Vehicle Simulation (MAVS).

The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, non-amputees inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we constructed two fabric-based tactile sensors that measure contact location along the palmar and dorsal sides of the prosthetic fingers and grasp pressure at the tip of the prosthetic thumb. Inspired by the natural sensorimotor system, we use the measurements from these sensors to provide vibrotactile feedback of contact location and implement a tactile grasp controller with reflexes that prevent over-grasping and object slip. We compare this tactile system to a standard myoelectric prosthesis in a challenging reach-to-pick-and-place task conducted without direct vision; 17 non-amputee adults took part in this single-session between-subjects study. Participants in the tactile group achieved more consistent high performance compared to participants in the standard group. These results show that adding contact-location feedback and reflex control increases the consistency with which objects can be grasped and moved without direct vision in upper-limb prosthetics.

Suppose an agent asserts that it will move through an environment in some way. When the agent executes its motion, how does one verify the claim? The problem arises in a range of contexts including validating safety claims about robot behavior, applications in security and surveillance, and for both the conception and the (physical) design and logistics of scientific experiments. Given a set of feasible sensors to select from, we ask how to choose sensors optimally in order to ensure that the agent’s execution does indeed fit its pre-disclosed itinerary. Our treatment is distinguished from prior work in sensor selection by two aspects: the form the itinerary takes (a regular language of transitions) and that families of sensor choices can be grouped as a single choice. Both are intimately tied together, permitting construction of a product automaton because the same physical sensors (i.e., the same choice) can appear multiple times. This paper establishes the hardness of sensor selection for itinerary validation within this treatment, and proposes an exact algorithm based on an integer linear programming (ILP) formulation that is capable of solving problem instances of moderate size. We demonstrate its efficacy on small-scale case studies, including one motivated by wildlife tracking.

Under-actuated bionic hands have achieved tremendous popularity in many fields because of their advantages of lightweight, budget-friendly, satisfactory flexibility, and adaptability. Except for the bionic mechanical design, various anthropomorphic control strategies have been proposed and investigated in the last decades. However, due to its under-actuated characteristic, there are still many challenges for anthropomorphic control of all the degrees of freedom (DOFs) using less input. It is challenging to map the human hand kinematic synergies on robotic hands, particularly for a dynamic environment. Therefore, it is worth studying how to control the under-actuated bionic hand effectively in a dynamic environment. In this paper, an anthropomorphic control method is proposed using sensor fusion of hand kinematic inputs to control the under-actuated bionic hand. In order to map the kinematics of human fingers to the bionic hand, a novel finger bending angle is defined to represent the posture of human fingers. Multiple Leap Motion Controllers (LMC) are fused to estimate the stable and accurate finger bending angles to avoid the occlusion problem. Finally, experiments with real-time control of the under-actuated bionic hand are implemented to demonstrate the proposed approach’s effectiveness.

Venipuncture is an indispensable procedure for both diagnosis and treatment. In this paper, unlike existing solutions that fully or partially rely on professional assistance, a compact robotic system integrating both novel hardware and software developments is introduced. The hardware consists of a set of units to facilitate the supporting, positioning, puncturing, and imaging functionalities. To achieve full automation, a novel deep learning framework — semi-ResNeXt-Unet for semi-supervised vein segmentation from ultrasound images is proposed. The depth information of vein is calculated and enables the automated navigation for the puncturing unit. The algorithm is validated on 40 volunteers, and the proposed semi-ResNeXt-Unet improves the dice similarity coefficient (DSC) by 5.36%, decreases the centroid error by 1.38 pixels and decreases the failure rate by 5.60%, compared to fully-supervised ResNeXt-Unet.

Autonomous control of an emergency vehicle will save lives through faster transport and shorter response. Towards this goal, it must overcome the challenge of inter- acting with existing human drivers on the road. We present a game-theoretic approach for semi-cooperative control of an autonomous emergency vehicle that can interact efficiently with humans on the road. We model the interactions between autonomous and human driven cars with Social Value Orientation, a metric from social psychology, that allows the controller to leverage their influence on the trajectories of neighboring human drivers. In addition, by using a modified version of iterative best response, we direct the algorithm to converge to Nash equilibria that are cooperative. We demonstrate the efficacy of our algorithm in simulations of drivers in traffic, with a variety of traffic densities and driver personalities. In simulations of prosocial human drivers, our algorithm provides an 8% improvement in distance-traveled compared to egoistic human drivers.

To capture the geometry of an object by an autonomous system, next best view (NBV) planning can be used to determine the path a robot will take. However, current NBV planning algorithms do not distinguish between objects that need to be mapped and everything else in the environment; leading to inefficient search strategies. In this paper we present a novel approach for NBV planning that accounts for the importance of objects in the environment to inform navigation. Using weighted entropy to encode object utilities computed via semantic segmentation, we evaluate our approach over a set of virtual Gazebo environments comparable to construction scales. Our results show that using semantic information reduces the time required to capture a target object by at least 40 percent.

This work presents a semantic-aware path-planning pipeline for Unmanned Aerial Vehicles (UAVs) using deep reinforcement learning for vision-based navigation in challenging environments. Driven by the maturity of works in semantic segmentation, the proposed path-planning architecture uses reinforcement learning to distinguish the parts of the scene that are perceptually more informative using semantic cues, in effect guiding more robust, repeatable, and accurate navigation of the UAV to the predefined goal destination. Assuming that the UAV performs vision-based state estimation, such as keyframe-based visual odometry, and semantic segmentation onboard, the proposed deep policy network continuously evaluates the optimal relative perceptual informativeness of each semantic class in view. A perception-aware path planner uses these informativeness values to perform trajectory optimization in order to generate the next best action with respect to the current state and the perception quality of the surroundings, essentially guiding the UAV to avoid flying over perceptually degraded regions. Thanks to the use of semantic cues, the policy can be trained in a large number of non-photorealistic randomly-generated scenes, and results to an architecture that is generalizable to environments with the same semantic classes, independently of their visual appearance. Extensive evaluations on challenging, photorealistic simulations reveal a remarkable improvement in robustness and success rate with the proposed approach over the state of the art in active perception. Video – https://youtu.be/RaO3whUBVnc

When interacting in unstructured human environments, occasional robot failures are inevitable. When such failures occur, everyday people, rather than trained technicians, will be the first to respond. Existing natural language explanations hand-annotate contextual information from an environment to help everyday people understand robot failures. However, this methodology lacks generalizability and scalability. In our work, we introduce a more generalizable semantic explanation framework. Our framework autonomously captures the semantic information in a scene to produce semantically descriptive explanations for everyday users. To generate failure-focused explanations that are semantically grounded, we lever-ages both semantic scene graphs to extract spatial relations and object attributes from an environment, as well as pairwise ranking. Our results show that these semantically descriptive explanations significantly improve everyday users’ ability to both identify failures and provide assistance for recovery than the existing state-of-the-art context-based explanations.

Solving complex real-world tasks, e.g., autonomous fleet control, often involves a coordinated team of multiple agents which learn strategies from visual inputs via reinforcement learning. Many existing multi-agent reinforcement learning (MARL) algorithms however don’t scale to environments where agents operate on visual inputs. To address this issue, algorithmically, recent works have focused on non-stationarity and exploration. In contrast, we study whether scalability can also be achieved via a disentangled representation. For this, we explicitly construct an object-centric intermediate representation to characterize the states of an environment, which we refer to as ‘semantic tracklets.’ We evaluate ‘semantic tracklets’ on the visual multi-agent particle environment (VMPE) and on the challenging visual multi-agent GFootball environment. ‘Semantic tracklets’ consistently outperform baselines on VMPE, and achieve a +2.4 higher score difference than baselines on GFootball. Notably, this method is the first to successfully learn a strategy for five players in the GFootball environment using only visual data. For more, please see our project page: https://ioujenliu.github.io/SemanticTracklets

Outdoor scene completion is a challenging issue in 3D scene understanding, which plays an important role in intelligent robotics and autonomous driving. Due to the sparsity of LiDAR acquisition, it is far more complex for 3D scene completion and semantic segmentation. Since semantic features can provide constraints and semantic priors for completion tasks, the relationship between them is worth exploring. Therefore, we propose an end-to-end semantic segmentation-assisted scene completion network, including a 2D completion branch and a 3D semantic segmentation branch. Specifically, the network takes a raw point cloud as input, and merges the features from the segmentation branch into the completion branch hierarchically to provide semantic information. By adopting BEV representation and 3D sparse convolution, we can benefit from the lower operand while maintaining effective expression. Besides, the decoder of the segmentation branch is used as an auxiliary, which can be discarded in the inference stage to save computational consumption. Extensive experiments demonstrate that our method achieves competitive performance on SemanticKITTI dataset with low latency. Code and models will be released at https://github.com/jokester-zzz/SSA-SC.

Accurate and reliable localization is a fundamental requirement for autonomous vehicles to use map information in higher-level tasks such as navigation or planning. In this paper, we present a novel approach to vehicle localization in dense semantic maps, including vectorized high-definition maps or 3D meshes, using semantic segmentation from a monocular camera. We formulate the localization task as a direct image alignment problem on semantic images, which allows our approach to robustly track the vehicle pose in semantically labeled maps by aligning virtual camera views rendered from the map to sequences of semantically segmented camera images. In contrast to existing visual localization approaches, the system does not require additional keypoint features, handcrafted localization landmark extractors or expensive LiDAR sensors. We demonstrate the wide applicability of our method on a diverse set of semantic mesh maps generated from stereo or LiDAR as well as manually annotated HD maps and show that it achieves reliable and accurate localization in real-time.

Localization is an essential task for mobile autonomous robotic systems that want to use pre-existing maps or create new ones in the context of SLAM. Today, many robotic platforms are equipped with high-accuracy 3D LiDAR sensors, which allow a geometric mapping, and cameras able to provide semantic cues of the environment. Segment-based mapping and localization have been applied with great success to 3D point-cloud data, while semantic understanding has been shown to improve localization performance in vision based systems. In this paper we combine both modalities in SemSegMap, extending SegMap into a segment based mapping framework able to also leverage color and semantic data from the environment to improve localization accuracy and robustness. In particular, we present new segmentation and descriptor extraction processes. The segmentation process benefits from additional distance information from color and semantic class consistency resulting in more repeatable segments and more overlap after re-visiting a place. For the descriptor, a tight fusion approach in a deep-learned descriptor extraction network is performed leading to a higher descriptiveness for landmark matching. We demonstrate the advantages of this fusion on multiple simulated and real-world datasets and compare its performance to various baselines. We show that we are able to find 50.9 % more high-accuracy prior-less global localizations compared to SegMap on challenging datasets using very compact maps while also providing accurate full 6 DoF pose estimates in real-time.

Multi-sensor solution has been widely adopted in real-world robotics systems (e.g., self-driving vehicles) due to its better robustness. However, its performance is highly dependent on the accurate calibration between different sensors, which is very time-consuming (i.e., hours of human efforts) to acquire. Recent learning-based solutions partially address this yet still require costly ground-truth annotations as supervision. In this paper, we introduce a novel self-supervised semantic alignment loss to quantitatively measure the quality of a given calibration. It is well correlated with conventional evaluation metrics while it does not require ground-truth calibration annotations as the reference. Based on this loss, we further propose an annotation-free optimization-based calibration algorithm (SemAlign) that first estimates a coarse calibration with loss-guided initialization and then refines it with gradient-based optimization. SemAlign reduces the calibration time from hours of human efforts to only seconds of GPU computation. It not only achieves comparable performance with existing supervised learning frameworks but also demonstrates a much better generalization capability when transferred to a different dataset.

Forecasting future trajectories of multiple pedestrians in a crowded environment is a challenging problem due to the complex interactions among the pedestrians. The interactions can be asymmetric and their influences may vary over time. Moreover, each pedestrian can exhibit different behavior at any given time and context and thus they may have multiple future possible trajectories. In this work, we present a Gated Graph Convolutional Network (GatedGCN) based trajectory prediction model that explicitly deal with the asymmetric influences among the adjacent pedestrians through edge-wise gating mechanism. Through GatedGCN only, an overall average improvement of 16% and 18% was achieved on the two performance metrics over the state-of-the-art trajectory forecasting methods. Next, we tackle the problem of learning multi-modal distributions of each pedestrian trajectory using variational auto-encoders (VAEs). However, trajectories sampled from the learned distribution usually ignore the factors affecting the pedestrian motion such as collision avoidance and the target destination. While many of the existing approaches focus on learning such factors during the trajectory encoding process, we proposed a novel self-critical learning approach based on Actor-Critic framework to learn such factors during the trajectory generation process. We empirically show that our method creates fewer number of collisions than the existing methods on popular trajectory forecasting benchmarks.

Among various 3D capturing systems, since the system with line lasers based on the light sectioning method is simple and accurate, it has widely attracted many developers and used for many purposes. In addition, there is no need to synchronize the camera and the laser and also the configuration of the camera and the lasers is flexible, and thus, the system can be used for extreme conditions, such as underwater. There are two open problems for the system. The first problem is a low density of the 3D shape obtained from a single image, i.e., just several curves. The second problem is the accuracy of line detection in the wild. In this paper, we propose a self-calibration method using visual odometry (VO) to bundle a large number of frames to increase the density to solve the first problem. We also propose a robust line detection algorithm using CNN to solve the second problem. Comparative experiments prove the effectiveness of our proposed method. In addition, the system was tested in the extreme condition for demonstration.