IROS 2021 Papers

Autonomous delivery of suspended payloads with MAVs has many applications in rescue and logistics transportation. Robust and online estimation of the payload status is important but challenging especially in outdoor environments. The paper develops a novel real-time system for estimating the payload position; the system consists of a monocular fisheye camera and a novel encoder-based device. A Gaussian fusion-based estimation algorithm is developed to obtain the payload state estimation. Based on the robust payload position estimation, a payload controller is presented to ensure the re-liable tracking performance on aggressive trajectories. Several experiments are performed to validate the high performance of the proposed method.

This paper presents a vision-based control strategy for a rotary-wing unmanned aerial vehicle (RUAV) transporting an unknown suspended payload. The suspended payload parameters, which include its mass and cable length, are unknown and direct measurements of its states are not available. A feedforward-feedback adaptive control strategy, that consists of a notch filter and linear quadratic Gaussian (LQG) controller, is proposed to simultaneously avoid the excitation and actively damp the payload swing oscillations. The unknown payload mass is estimated using recursive least squares and the unknown cable length is estimated using a dedicated sine wave estimator. The payload parameter estimates are then used to adapt the control strategy for the specific suspended payload. A vision-based state estimator is implemented to provide payload state estimates for the optimal full-state feedback controller. Simulation results show that the control strategy successfully adapts for different suspended payloads and effectively damps the unwanted payload swing oscillations.

The visibility of targets determines performance and even success rate of various applications, such as active slam, exploration, and target tracking. Therefore, it is crucial to take the visibility of targets into explicit account in trajectory planning. In this paper, we propose a general metric for target visibility, considering observation distance and angle as well as occlusion effect. We formulate this metric into a differentiable visibility cost function, with which spatial trajectory and yaw can be jointly optimized. Furthermore, this visibility-aware trajectory optimization handles dynamic feasibility of position and yaw simultaneously. To validate that our method is practical and generic, we integrate it into a customized quadrotor tracking system. The experimental results show that our visibility-aware planner performs more robustly and observes targets better. In order to benefit related researches, we release our code to the public.

Modern agricultural applications require knowledge about the position and size of fruits on plants. However, occlusions from leaves typically make obtaining this information difficult. We present a novel viewpoint planning approach that builds up an octree of plants with labeled regions of interest (ROIs), i.e., fruits. Our method uses this octree to sample viewpoint candidates that increase the information around the fruit regions and evaluates them using a heuristic utility function that takes into account the expected information gain. Our system automatically switches between ROI targeted sampling and exploration sampling, which considers general frontier voxels, depending on the estimated utility. When the plants have been sufficiently covered with the RGB-D sensor, our system clusters the ROI voxels and estimates the position and size of the detected fruits. We evaluated our approach in simulated scenarios and compared the resulting fruit estimations with the ground truth. The results demonstrate that our combined approach outperforms a sampling method that does not explicitly consider the ROIs to generate viewpoints in terms of the number of discovered ROI cells. Furthermore, we show the real-world applicability by testing our framework on a robotic arm equipped with an RGB-D camera installed on an automated pipe-rail trolley in a capsicum glasshouse.

We propose the ViNet architecture for audio-visual saliency prediction. ViNet is a fully convolutional encoder-decoder architecture. The encoder uses visual features from a network trained for action recognition, and the decoder infers a saliency map via trilinear interpolation and 3D convolutions, combining features from multiple hierarchies. The overall architecture of ViNet is conceptually simple; it is causal and runs in real-time (60 fps). ViNet does not use audio as input and still outperforms the state-of-the-art audio-visual saliency prediction models on nine different datasets (three visual-only and six audio-visual datasets). ViNet also surpasses human performance on the CC, SIM and AUC metrics for the AVE dataset, and to our knowledge, it is the first model to do so. We also explore a variation of ViNet architecture by augmenting audio features into the decoder. To our surprise, upon sufficient training, the network becomes agnostic to the input audio and provides the same output irrespective of the input. Interestingly, we also observe similar behaviour in the previous state-of-the-art models [1] for audio-visual saliency prediction. Our findings contrast with previous works on deep learning-based audio-visual saliency prediction, suggesting a clear avenue for future explorations incorporating audio in a more effective manner. The code and pre-trained models are available at https://github.com/samyak0210/ViNet.

Marine vessels are subject to high wear and tear due to the conditions they operate in. To reduce risk of failure during operation, vessels are inspected periodically every five years. These inspections are prone to high subjectiveness that makes them hard to reproduce for the shipping owners. The purpose of this paper is to present a regressor to a Faster R-CNN network that can help alleviate some of the subjective assessment currently performed by human surveyors by estimating the severity of a corroded area, autonomously using drones. A feature pyramid backbone is shared between the Faster R-CNN and the added regression head. The goal of the regressor is to introduce a more objective assessment of the vessel that gives a consistent output for a consistent input. The system is evaluated on a real dataset, acquired in ballast tanks and the experimental results indicate that our deep learning approach can be used to detect and quantify corroded areas during the inspection process of marine vessels.

Functional autonomous systems often realize complex tasks by utilizing state machines comprised of discrete primitive behaviors and transitions between these behaviors. This architecture has been widely studied in the context of quasi-static and dynamics-independent systems. However, applications of this concept to dynamical systems are relatively sparse, despite extensive research on individual dynamic primitive behaviors, which we refer to as "motion primitives." This paper formalizes a process to determine dynamic-state aware conditions for transitions between motion primitives in the context of safety. The result is framed as a "motion primitive graph" that can be traversed by standard graph search and planning algorithms to realize functional autonomy. To demonstrate this framework, dynamic motion primitives— including standing up, walking, and jumping—and the transitions between these behaviors are experimentally realized on a quadrupedal robot.

On-demand ride-sharing is a promising way to improve mobility efficiency and reliability. The quality of passenger experience and the profit achieved by these platforms are strongly affected by the vehicle dispatch policy. However, existing ride-sharing research seldom considers travel time uncertainty, which leads to inaccurate dispatch allocations. This paper proposes a framework for dynamic vehicle dispatch that leverages stochastic travel time models to improve the performance of a fleet of shared vehicles. The novelty of this work includes: (1) a stochastic on-demand ride-sharing scheme to maximize the service rate (percentage of requests served) and reliability (probability of on-time arrival); (2) a technique based on approximate stochastic shortest path algorithms to compute the reliability for a ride-sharing trip; (3) a method to maximize the profit when a penalty for late arrivals is introduced. Based on New York City taxi data, it is shown that by considering travel time uncertainty, ride-sharing service achieves higher service rate, reliability and profit.

This paper presents a novel approach to the multigoal trajectory planning for vehicles with curvature-constrained trajectories such as fixed-wing aircraft. In the existing formulation called the Dubins Traveling Salesman Problem (DTSP), the vehicle speed is assumed to be constant over the whole trajectory, and that does not allow adaptation of the turning radius of the trajectory between the target locations. It does not support optimization of the overall flight time of the multi-goal trajectory by exploiting higher speeds for longer turning radii. Therefore, we propose a novel problem formulation called the Variable-Speed Traveling Salesman Problem (VS-TSP) that employs time-efficient trajectories with variable speed based on a generalization of the Dubins vehicle model, allowing multiple turning radii and change of the forward speed of the vehicle. The VS-TSP allows the vehicle to slow down if high maneuverability is necessary and speed up if high-speed turns with a large radius are beneficial to the overall tour cost. Based on the evaluation results for Cessna 172 aircraft model, the proposed VNS-based algorithm with variable speed provides up to about 20 % faster trajectories than a solution of the DTSP with a single turning radius.

Origami robots composed of rigid parts with flexible joints have inherent compliance that enables deployment and reconfiguration for various shape adaptations. The major drawback of such mechanical compliance is its intrinsic softness and lack of controllability of this stiffness. In this work, we propose a design of variable stiffness origami joints to be integrated into large scale origami systems with an inner controllable joint stiffness. This novel way of controlling compliance in origami structures demonstrates that the embedded variable stiffness joint in the prototype can provide rich haptic feedback to the user through its compact, collapsible interactive platform.

Estimating the 6D pose of objects is beneficial for robotics tasks such as transportation, autonomous navigation, manipulation as well as in scenarios beyond robotics like virtual and augmented reality. With respect to single image pose estimation, pose tracking takes into account the temporal information across multiple frames to overcome possible detection inconsistencies and to improve the pose estimation efficiency. In this work, we introduce a novel Deep Neural Network (DNN) called VIPose, that combines inertial and camera data to address the object pose tracking problem in real-time. The key contribution is the design of a novel DNN architecture which fuses visual and inertial features to predict the objects’ relative 6D pose between consecutive image frames. The overall 6D pose is then estimated by consecutively combining relative poses. Our approach shows remarkable pose estimation results for heavily occluded objects that are well known to be very challenging to handle by existing state-of-the-art solutions. The effectiveness of the proposed approach is validated on a new dataset called VIYCB with RGB image, IMU data, and accurate 6D pose annotations created by employing an automated labeling technique. The approach presents accuracy performances comparable to state-of-the-art techniques, but with the additional benefit of being real-time.

We present a decentralized collision avoidance method for dense environments based on buffered Voronoi cells (BVC) and reciprocal velocity obstacles (RVO). Our approach is designed for scenarios with a large number of agents in close proximity and provides passive-friendly collision avoidance guarantees. The Voronoi cells are superimposed with RVO cones to compute a suitable direction for each agent, and we use that direction to compute a local collision-free path. Our approach can also satisfy double-integrator dynamics, and we use the properties of the BVC to formulate a simple, decentralized deadlock resolution strategy. We demonstrate the benefits of V-RVO in complex scenarios with tens of agents in close proximity. In practice, V-RVO’s performance is comparable to prior velocity-obstacle methods, and the collision avoidance behavior is significantly less conservative than ORCA.

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot’s kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.

Many robotic manipulation problems are multi-modal—they consist of a discrete set of mode families (e.g., whether an object is grasped or placed) each with a continuum of parameters (e.g., where exactly an object is grasped). Core to these problems is solving single-mode motion plans, i.e., given a mode from a mode family (e.g., a specific grasp), find a feasible motion to transition to the next desired mode. Many planners for such problems have been proposed, but complex manipulation plans may require prohibitively long computation times due to the difficulty of solving these underlying single-mode problems. It has been shown that using experience from similar planning queries can significantly improve the efficiency of motion planning. However, even though modes from the same family are similar, they impose different constraints on the planning problem, and thus experience gained in one mode cannot be directly applied to another. We present a new experience-based framework, ALEF, for such multi-modal planning problems. ALEF learns using paths from single-mode problems from a mode family, and applies this experience to novel modes from the same family. We evaluate ALEF on a variety of challenging problems and show a significant improvement in the efficiency of sampling-based planners both in isolation and within a multi-modal manipulation planner.

Vision-based systems for terrain detection are ubiquitous in mobile robotics, while such systems recently emerged for locomotion assistance of disabled people. For instance, wearable devices embedding vision sensors can assist people in navigation; or guide lower-limb prosthesis or exoskeleton controller to retrieve gait patterns being adapted to the executed task (overground walking, stairs, slopes, etc.). In this research, we present a vision-based algorithm achieving the detection of flat ground, steps, and ramps, using a depth camera. The raw point cloud data obtained from the depth camera passes through multiple processing steps to extract environmental features, and then achieves terrain detection by feeding these features into a classification tree. The camera was mounted on a custom-made wearable tool that can be placed on the chest of human users. This contribution reports a pilot validation study with 6 healthy subjects moving in an indoor environment containing a rich set of different types of terrains. Our method can predict the locomotion modes up to three steps in front of the user. Moreover, it is able to perform terrain detection even if the path is partially occluded by another walker. The method accuracy with a cleared path was found to be above 90% for all locomotion modes.

The long-term goal of this research is to develop methods for training propulsion during walking using robotic exoskeletons that customize their intervention based on the response of an individual.In this study, we first determined the feasibility of modeling the relationship between propulsion mechanics and parameters of a robotic intervention applied at the hip and knee joints as a Gaussian process. Specifically, we used data obtained in a previous experiment that used pulses of torque applied at the hip and knee joint, at early and late stance, to establish the relationship between a 4D control parameter space and the resulting changes in hip extension and propulsive impulse at multiple strides following intervention. We estimated Gaussian models both at the group level and for each subject. Moreover, we used the estimated subject-specific models to simulate virtual human-in-the-loop optimization (HIL) experiments based on Bayesian optimization to establish the optimal settings of acquisition function and seed point selection methods.The estimated subject-specific optimal conditions have large between-subject variability in the kinetic component of propulsion mechanics (propulsive impulse), with only 31% of subjects featuring a subject-specific optimal point in the surroundings (within a sphere of radius 20% of each dimension’s range) of the group-level optimal point. Instead, variability of the effects on the kinematic component of propulsion (leg extension) were smaller (75% of the subjects within the surroundings of the group-optimal point). Virtual HIL experiments indicate that expected improvement is the most effective acquisition method, while no significant effect of seed point selection method was observed. Our study suggests that individualized training may be necessary for inducing desired effects in propulsive force generation during walking.

The tuning process for a robotic prosthesis is a challenging and time-consuming task both for users and clinicians. An automatic tuning approach using reinforcement learning (RL) has been developed for a knee prosthesis to address the challenges of manual tuning methods. The algorithm tunes the optimal control parameters based on the provided knee joint profile that the prosthesis is expected to replicate during gait safely. This paper presents an intuitive interface designed for the prosthesis users and clinicians to choose the preferred knee joint profile during gait and use the autotuner to replicate in the prosthesis. The interface-based approach is validated by observing the ability of the tuning algorithm to successfully converge to various alternate knee profiles by testing on two able-bodied subjects walking with a robotic knee prosthesis. The algorithm was found to converge successfully in an average duration of 1.15 min for the first subject and 2.31 min for the second subject. Further, the subjects displayed different preferences for optimal profiles reinforcing the need to tune alternate profiles. The implications of the results in the tuning of robotic prosthetic devices are discussed.

An efficient 3D scene perception algorithm is a vital component for autonomous driving and robotics systems. In this paper, we focus on semantic scene completion, which is a task of jointly estimating the volumetric occupancy and semantic labels of objects. Since the real-world data is sparse and occluded, this is an extremely challenging task. We propose a novel framework, named Up-to-Down network (UDNet), to achieve the large-scale semantic scene completion with an encoder-decoder architecture for voxel grids. The novel up-to-down block can effectively aggregate multi-scale context information to improve labeling coherence, and the atrous spatial pyramid pooling module is leveraged to expand the receptive field while preserving detailed geometric information. Besides, the proposed multi-scale fusion mechanism efficiently aggregates global background information and improves the semantic completion accuracy. Moreover, to further satisfy the needs of different tasks, our UDNet can accomplish the multi-resolution semantic completion, achieving faster but coarser completion. Detailed experiments in the semantic scene completion benchmark of SemanticKITTI illustrate that our proposed framework surpasses the state-of-the-art methods with remarkable margins and a real-time inference speed by using only voxel grids as input.

The key challenge of unsupervised vehicle re-identification (Re-ID) is learning discriminative features from unlabelled vehicle images. Numerous methods using domain adaptation have achieved outstanding performance, but those methods still need a labelled dataset as a source domain. This paper addresses an unsupervised vehicle Re-ID method, which no need any types of a labelled dataset, through a Self-supervised Metric Learning (SSML) based on a feature dictionary. Our method initially extracts features from vehicle images and stores them in a dictionary. Thereafter, based on the dictionary, the proposed method conducts dictionary-based positive label mining (DPLM) to search for positive labels. Pair-wise similarity, relative-rank consistency, and adjacent feature distribution similarity are jointly considered to find images that may belong to the same vehicle of a given probe image. The results of DPLM are applied to dictionary-based triplet loss (DTL) to improve the discriminativeness of learnt features and to refine the quality of the results of DPLM progressively. The iterative process with DPLM and DTL boosts the performance of unsupervised vehicle Re-ID. Experimental results demonstrate the effectiveness of the proposed method by producing promising vehicle Re-ID performance without a pre-labelled dataset. The source code for this paper is publicly available on https://github.com/andreYoo/VeRI_SSML_FD.git.