IROS 2021 Papers

In this paper, we propose micROS.BT, an event-driven behavior tree (BT) framework aiming at supporting swarm-robot coordination. Compared with other BT frame-works, micROS.BT implements the event-driven way under the multi-thread mode, which can effectively save computing resources. Moreover, in order to ensure swarm-robot coordination, we optimize the implementation of the traditional blackboard and propose the multi-mode blackboard, which supports inner-tree, inter-tree, and inter-robot data sharing. Furthermore, considering the limited modularity of a single tree, micROS.BT realizes a mechanism called hierarchical tree management which involves inter-tree notifying and waiting functionalities, while ensuring that each tree is independent and self-scheduled. The effectiveness of micROS.BT is verified by simulation and real-robot experiments for different system settings, showing that a substantial improvement is achieved in comparison with the traditional BT implementations.

We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural Radiance Field (NeRF). NeRFs have been shown to be remarkably effective for the task of view synthesis — synthesizing photorealistic novel views of real-world scenes or objects. In this work, we investigate whether we can apply analysis-by-synthesis via NeRF for mesh-free, RGB-only 6DoF pose estimation – given an image, find the translation and rotation of a camera relative to a 3D object or scene. Our method assumes that no object mesh models are available during either training or test time. Starting from an initial pose estimate, we use gradient descent to minimize the residual between pixels rendered from a NeRF and pixels in an observed image. In our experiments, we first study 1) how to sample rays during pose refinement for iNeRF to collect informative gradients and 2) how different batch sizes of rays affect iNeRF on a synthetic dataset. We then show that for complex real-world scenes from the LLFF dataset [21], iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF. Finally, we show iNeRF can perform categorylevel object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

We present iGibson 1.0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains 15 fully interactive home-sized scenes with 108 rooms populated with rigid and articulated objects. The scenes are replicas of real-world homes, with distribution and the layout of objects aligned to those of the real world. iGibson 1.0 integrates several key features to facilitate the study of interactive tasks: i) generation of high-quality virtual sensor signals (RGB, depth, segmentation, LiDAR, flow and so on), ii) domain randomization to change the materials of the objects (both visual and physical) and/or their shapes, iii) integrated sampling-based motion planners to generate collision-free trajectories for robot bases and arms, and iv) intuitive human-iGibson interface that enables efficient collection of human demonstrations. Through experiments, we show that the full interactivity of the scenes enables agents to learn useful visual representations that accelerate the training of downstream manipulation tasks. We also show that iGibson features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of human demonstrated (mobile) manipulation behaviors. iGibson 1.0 is open-source, equipped with comprehensive examples and documentation. For more information, visit our project website: http://svl.stanford.edu/igibson/.

Simulation is an essential tool for developing robotic systems; however it is computationally intensive and does not scale well, especially for multi-robot scenarios. In this paper we introduce daß – a system that facilitates distributable and scalable simulations of robotic applications. It overcomes many of the performance and quality limitations involved in common multi-robot simulation scenarios through horizontal scaling. Through extensive evaluation, we show that daß significantly outperforms the widely used Gazebo robotics simulator, and easily scales well beyond the current limitations with no significant loss in performance and simulation fidelity.

3D perception on point-cloud is a challenging and crucial computer vision task. A point-cloud consists of a sparse, unstructured, and unordered set of points. To understand a point-cloud, previous point-based methods, such as PointNet++, extract visual features through the hierarchical aggregation of local features. However, such methods have several critical limitations: 1) They require considerable sampling and grouping operations, which leads to low inference speed. 2) Despite redundancy among adjacent points, they treat all points alike with an equal amount of computation. 3) They aggregate local features together through downsampling, which causes information loss and hurts perception capability. To overcome these challenges, we propose a novel, simple, and elegant deep learning model called YOGO (You Only Group Once). YOGO divides a point-cloud into a small number of parts and extracts a high-dimensional token to represent points within each sub-region. Next, we use self-attention to capture token-to-token relations, and project the token features back to the point features. We formulate such a series of operations as a relation inference module (RIM). Compared with previous methods, YOGO is very efficient because it only needs to sample and group a point-cloud once. Instead of operating on points, YOGO operates on a small number of tokens, each of which summarizes the point features in a sub-region. This allows us to avoid redundant computation and thus boosts efficiency. Moreover, YOGO preserves pointwise features by projecting token features to point features although the RIM computes on tokens. This avoids information loss and enhances point-wise perception capability. We conduct thorough experiments to demonstrate that YOGO achieves at least 3.0x speedup over point-based baselines while delivering competitive classification and segmentation performance on a classification dataset and a segmentation dataset based on 3D Wharehouse, and S3DIS datasets. The c... Show More

We present a novel sensor-based learning navigation algorithm to compute a collision-free trajectory for a robot in dense and dynamic environments with moving obstacles or targets. Our approach uses deep reinforcement learning-based expert policy that is trained using a sim2real paradigm. In order to increase the reliability and handle the failure cases of the expert policy, we combine with a policy extraction technique to transform the resulting policy into a decision tree format. We use properties of decision trees to analyze and modify the policy and improve performance of navigation algorithm including smoothness, frequency of oscillation, frequency of immobilization, and obstruction of target. Overall, we are able to modify the policy to design an improved learning algorithm without retraining. We highlight the benefits of our approach in simulated environments and navigating a Clearpath Jackal robot among moving pedestrians. (Videos at this url: https://gamma.umd.edu/researchdirections/xrl/navviper)

Inspired by bat flight performance, we explore the advantages of wing twist and fold for flapping wing robots. For this purpose, we develop a dynamical model that incorporates these two degrees of freedom to the wing. The twist is assumed to be linearly-increasing along the wing, while the wing fold is modeled as a relative rotation of the handwing with respect to the armwing. An optimization scheme parameterizes the wing kinematics for 2, 5 and 8 m/s forward flight velocities. The intricate interplay between wing orientation, effective angle of attack and the ensuing lift and thrust generation are discussed. The results show that wing twist and fold alleviate negative lift and thrust in the upstroke, and in some cases producing persistent positive thrust throughout cycle for handwing. As a result, power consumption drops precipitously compared to the base case of a rigid flat plate. Another crucial realization is the relative importance of wing twist and fold in achieving efficient flight strongly depends on speeds. At slow flight, twist is significantly more effective in minimizing the power, but becomes energetically inefficient for fast speeds. The results also show that a 45° wing fold during upstroke is energetically beneficial for all speeds. The synergy of wing twist and fold are most prominent at slow flight. These findings provides useful guidelines for designing flapping wing robots.

The development of perception and control methods that allow bird-scale flapping-wing robots (a.k.a. ornithopters) to perform autonomously is an under-researched area. This paper presents a fully onboard event-based method for ornithopter robot visual guidance. The method uses event cameras to exploit their fast response and robustness against motion blur in order to feed the ornithopter control loop at high rates (100 Hz). The proposed scheme visually guides the robot using line features extracted in the event image plane and controls the flight by actuating over the horizontal and vertical tail deflections. It has been validated on board a real ornithopter robot with real-time computation in low-cost hardware. The experimental evaluation includes sets of experiments with different maneuvers indoors and outdoors.

Our paper proposes a model predictive controller as a single-task formulation that simultaneously optimizes wheel and torso motions. This online joint velocity and ground reaction force optimization integrates a kinodynamic model of a wheeled quadrupedal robot. It defines the single rigid body dynamics along with the robot’s kinematics while treating the wheels as moving ground contacts. With this approach, we can accurately capture the robot’s rolling constraint and dynamics, enabling automatic discovery of hybrid maneuvers without needless motion heuristics. The formulation’s generality through the simultaneous optimization over the robot’s whole-body variables allows for a single set of parameters and makes online gait sequence adaptation possible. Aperiodic gait sequences are automatically found through kinematic leg utilities without the need for predefined contact and lift-off timings, reducing the cost of transport by up to 85 %. Our experiments demonstrate dynamic motions on a quadrupedal robot with non-steerable wheels in challenging indoor and outdoor environments. The paper’s findings contribute to evaluating a decomposed, i.e., sequential optimization of wheel and torso motion, and single-task motion planner with a novel quantity, the prediction error, which describes how well a receding horizon planner can predict the robot’s future state. To this end, we report an improvement of up to 71 % using our proposed single-task approach, making fast locomotion feasible and revealing wheeled-legged robots’ full potential.

With the democratization of 3D LiDAR sensors, precise LiDAR odometries and SLAM are in high demand. New methods regularly appear, proposing solutions ranging from small variations in classical algorithms to radically new paradigms based on deep learning. Yet it is often difficult to compare these methods, notably due to the few datasets on which the methods can be evaluated and compared. Furthermore, their weaknesses are rarely examined, often letting the user discover the hard way whether a method would be appropriate for a use case.In this paper, we review and organize the main 3D LiDAR odometries into distinct categories. We implemented several approaches (geometric based, deep learning based, and hybrid methods) to conduct an in-depth analysis of their strengths and weaknesses on multiple datasets, guiding the reader through the different LiDAR odometries available. Implementation of the methods has been made publicly available at: https://github.com/Kitware/pyLiDAR-SLAM.

A 3D mesh offers a rich yet lightweight representation of geometry and topology for the metric and semantic understanding of a robot’s scene. Noisy features are often used to generate the mesh which furthers the need for accurate regularisation. Current approaches tightly couple front-end optimisation with regularisation making it difficult to evaluate the choice of discretisation and regularisation on mesh accuracy. In this work, we aim to explicitly query the performance of a set of well-known convex and non-convex regularisers on the mesh optimisation problem. We then apply these norms to dense depth estimation from a mesh representation and evaluate their performance in indoor and outdoor environments.While we show that the use of exotic, non-convex regularisers such as logTV and logTGV can result in more faithful structural reconstruction under noise, this comes at the cost of stronger outlier persistence that limits their performance when compared to their convex equivalents. This represents a significant departure from results achieved when the same regularisers are applied in denser "every-pixel" scenarios and suggests that current discretisation techniques adopted for this problem are more sensitive to triangulation. This sensitivity is often obscured in many practical robotic applications by a rigorous front-end that removes artefacts from the mesh to be optimised. By decoupling the front and back-ends we therefore show that further consideration must be taken to align current mesh discretisations with the classical definitions of variational regularisers to allow the full benefit of their well-documented properties to be realised.

Robotic technologies are becoming pervasive within industrial and domestic settings, resulting in more frequent interactions between humans and robots. To ensure these interactions are effective, Human-Robot Interaction (HRI) researchers have argued that robots and humans must establish a shared common ground by communicating fundamental pieces of information to each other, such as their intentions, goals, plans, status, etc. Although a large body of work has explored how robots might signal individual aspects of such information to users, we still know relatively little regarding the importance of such information overall (e.g., is communicating robot status more important than communicating robot goals?). Such information is necessary for robots acting in the wild to create prioritized lists of communicative goals as, at any given time, it is unlikely that a robot will be able to convey all possibly relevant or important aspects of information to users. Prioritizing information for users is a complex problem as many factors might influence information priority, including task context, user expertise, and robot capability. In this work, we first address the current state-of-the-art signaling methods for non-humanoid robots. Second, we take an initial step towards understanding prioritization by exploring what types of information users request, and how the rankings of informational importance that users assign change, in a prototypical shared-environment interaction with three different types of robots. Our results, collected from 150 participants on Amazon’s Mechanical Turk, generally show that users value information related to the robot’s battery, capabilities, task, safety, navigation, communication, and privacy, with user priorities of these items varying across a small ground robot, a large ground robot, and an aerial robot.

We propose a wearable tactile sensor suit, which can be regarded as tactile sensor networks, for monitoring natural body dynamics to be exploited as a computational resource for estimating the posture of a human or robot that wears it. We emulated the periodic motions of a wearer (a human and an android robot) using a novel sensor suit with a 9-channel fabric tactile sensor on the left arm. The emulation was conducted by using a linear regression (LR) model of sensor states as readout modules that predict the next wearer's movement using the current sensor data. Our result shows that the LR performance is comparable with other recurrent neural network approaches, suggesting that a fabric tactile sensor network can monitor the natural body motions, and further, this natural body dynamics itself can be used as an effective computational resource.

To solve the interference problems of wind and wave action and load movement when switching under water surface conditions in the marine environment, a study on the water surface stability prediction of the bio-inspired undulatory fin robot is carried out. Based on the fin motion equation and fluid drag theory, a water surface stability calculation model of the robot is established. The study compares the effects of different loads and heel angles on the stability of the robot's water surface under different calculation methods and verifies the validity of the model through computational fluid dynamics methods. The simulation results show that the water surface stability of the robot exhibits sinusoidal-like changes over time, which is equal to the undulatory fin period. The stability decreases with the increase of the drainage volume. When the drainage volume is constant, the stability first increases and then decreases with the increase in heel angle. The theoretical calculation results are consistent with the numerical results, which verify the effectiveness of the water surface stability prediction model proposed in this paper. It can provide a theoretical basis for the optimization design of water surface stability of the undulatory fin robot.

Gaussian processes (GPs) are a well-known nonparametric Bayesian inference technique, but they suffer from scalability problems for large sample sizes, and their performance can degrade for non-stationary or spatially heterogeneous data. In this work, we seek to overcome these issues through (i) employing variational free energy approximations of GPs operating in tandem with online expectation propagation steps; and (ii) introducing a local splitting step which instantiates a new GP whenever the posterior distribution changes significantly as quantified by the Wasserstein metric over posterior distributions. Over time, then, this yields an ensemble of sparse GPs which may be updated incrementally, and adapts to locality, heterogeneity, and non-stationarity in training data. We provide a 1-dimensional example to illustrate the motivation behind our approach, and compare the performance of our approach to other Gaussian process methods across various data sets, which often achieves competitive, if not superior predictive performance, relative to other locality-based GP regression methods in which hyperparameters are learned in an online manner.

In this paper, we consider periodic communication delays within the connected autonomous vehicles platoon. Periodic signals are fundamentally simple to create, and in this study we analyze whether certain amplitude or frequencies can cause instability. This is important as we discover in this study, the classical method of replacing time-varying delays with constant delays does not capture the complex stability boundary of periodic time delays which could be exploited by attackers to cause instability in the vehicle platoon. We use the semi-discretization method to obtain plant stability. Then, we take the average value of the time delay functions to characterize the maximal admissible delay region such that the time-delayed system remains stable and provide string stability analysis. Through numerical simulations, we verify the analytical results. We construct stability charts in different parameter spaces to explore the effects of the model parameters on the system stability. A specific range of frequencies was found that could destabilize the connected autonomous vehicle platoon.

Deep learning approaches to estimating 3D object pose and geometry present an attractive alternative to online estimation techniques, which can suffer from significant estimation latency. However, a practical hurdle to training state-of-the-art deep 3D bounding box estimators is collecting a sufficiently large dataset of 3D bounding box labels. In this work, we present a novel framework for weakly supervised volumetric monocular estimation (VoluMon) that requires annotations in the image space only, i.e., associated object bounding box detections and instance segmentation. By approximating object geometry as ellipsoids, we can exploit the dual form of the ellipsoid to optimize with respect to bounding box annotations and the primal form of the ellipsoid to optimize with respect to a segmented pointcloud. For a simulated dataset with access to ground-truth, we show monocular object estimation performance similar to a naive online depth based estimation approach and after online refinement when depth images are available, we also approach the performance of a learned deep 6D pose estimator, which is supervised with projected 3D bounding box keypoints and assumes known model dimensions. Finally, we show promising qualitative results generated from a real-world dataset collected using a stereo pair.

The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object’s shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object’s occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects’ reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object’s shape using a single arbitrary depth-view and a single dexterous-hand’s touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 643 for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (403, 643 and 1283) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.

Robots and autonomous systems need to know where they are within a map to navigate effectively. Thus, simultaneous localization and mapping or SLAM is a common building block of robot navigation systems. When building a map via a SLAM system, robots need to re-recognize places to find loop closure and reduce the odometry drift. Image-based place recognition received a lot of attention in computer vision, and in this work, we investigate how such approaches can be used for 3D LiDAR data. Recent LiDAR sensors produce high-resolution 3D scans in combination with comparably stable intensity measurements. Through a cylindrical projection, we can turn this information into a 360° panoramic range image. As a result, we can apply techniques from visual place recognition to LiDAR intensity data. The question of how well this approach works in practice has only partially been investigated. This paper provides an analysis of how such visual techniques can be with LiDAR data, and we provide an evaluation on different datasets. Our results suggest that this form of place recognition is possible and an effective means for determining loop closures.