IROS 2022 Papers

In this paper, we propose a hybrid visual-LiDAR odometry (H-VLO) framework that fuses predicted visual depth map and completed LiDAR map. Compared to the previous visual-LiDAR odometry methods, our approach leverages 2D feature matching and 3D association by utilizing deep depth map, deep flow map and deep LiDAR depth completion networks. Rather than extraction of the depth values from LiDAR measurements for each visual feature, our method first densifies a LiDAR scan with a deep depth completion network and then fuses it with visual deep depth map estimation in a Bayesian framework. This method reduces pose estimation drift by improving feature-to-feature and point-to-feature matching, as well as scale recovery. The evaluations on the public KITTI odometry benchmark show that our technique achieves better or at least comparable estimates than the state-of-the-art visual-LiDAR and monocular visual odometry approaches.

This paper presents a method for the robust selection of measurements in a simultaneous localization and mapping (SLAM) framework. Existing methods check consistency or compatibility on a pairwise basis, however many measurement types are not sufficiently constrained in a pairwise scenario to determine if either measurement is inconsistent with the other. This paper presents group-kk consistency maximization (\mathrm{G}k\text{CM}\mathrm{G}k\text{CM}) that estimates the largest set of measurements that is internally group-kk consistent. Solving for the largest set of group-kk consistent measurements can be formulated as an instance of the maximum clique problem on generalized graphs and can be solved by adapting current methods. This paper evaluates the performance of \mathrm{G}k\text{CM}\mathrm{G}k\text{CM} using simulated data and compares it to pairwise consistency maximization (PCM) presented in previous work.

Grounding a command to the visual environment is an essential ingredient for interactions between autonomous vehicles and humans. In this work, we study the problem of language grounding for autonomous vehicles, which aims to localize a region in a visual scene according to a natural language command from a passenger. Prior work only employs the top layer representations of a vision-and-language pretrained model to predict the region referred to by the command. However, such a method omits the useful features encoded in other layers, and thus results in inadequate understanding of the input scene and command. To tackle this limitation, we present the first layer fusion approach for this task. Since different visual regions may require distinct types of features to disambiguate them from each other, we further propose the region-specific dynamic (RSD) layer attention to adaptively fuse the multimodal information across layers for each region. Extensive experiments on the Talk2Car benchmark demonstrate that our approach helps predict more accurate regions and outperforms state-of-the-art methods.

Visual and lidar Simultaneous Localization and Mapping (SLAM) algorithms benefit from the Inertial Measurement Unit (IMU) modality. The high-rate inertial data complement the other lower-rate modalities. Moreover, in the absence of constant acceleration, the gravity vector makes two attitude angles out of three observable in the global coordinate frame. In visual odometry, this is already being used to reduce the 6-Degrees Of Freedom (DOF) pose estimation problem to 4-DOF. In lidar SLAM, the gravity measurements are often used as a penalty in the back-end global map optimization to prevent map deformations. In this work, we propose an Iterative Closest Point (ICP)-based front-end which exploits the observable DOF and provides pose estimates aligned with the gravity vector. We believe that this front-end has the potential to support the loop closure identification, thus speeding up convergences of global map optimizations. The presented approach has been extensively tested against accurate ground-truth localization in large-scale outdoor environments as well as in the Subterranean Challenge organized by Defense Advanced Research Projects Agency (DARPA). We show that it can reduce the localization drift by 30% when compared to the standard 6-DOF ICP. Moreover, the code is readily available to the community as a part of the libpointmatcher library.

Although the grasping state analysis is vital in the study of manipulators, the grasping state analysis of soft manipulators as an independent research topic is not much so far. This paper proposes a novel pneumatic soft manipulator with a flexible tactile sensor array (SM-FTSA). The flexible tactile sensor array comprises piezoresistive materials with a porous structure. An equal potential approach is adopted to realize the collection of tactile signals of the SM-FTSA. Inspired by the grasping analysis of rigid manipulators, we propose 4 grasping states for the SM-FTSA, including inflating, shaking, stable, and slipping. Based on the experimental data, we conduct grasping experiments on 12 objects with SM-FTSA, and we propose 10 features that reflect the grasping state. Several machine learning methods are utilized to classify the grasping state. Among them, the Random Forest method presents the best performance, and the average classification accuracy reaches 99%.

Robot simulation has been an essential tool for data-driven manipulation tasks. However, most existing simulation frameworks lack either efficient and accurate models of physical interactions with tactile sensors or realistic tactile simulation. This makes the sim-to-real transfer for tactile-based manipulation tasks still challenging. In this work, we integrate simulation of robot dynamics and vision-based tactile sensors by modeling the physics of contact. This contact model uses simulated contact forces at the robot's end-effector to inform the generation of realistic tactile outputs. To eliminate the sim-to-real transfer gap, we calibrate our physics simulator of robot dynamics, contact model, and tactile optical simulator with real-world data, and then we demonstrate the effectiveness of our system on a zero-shot sim-to-real grasp stability prediction task where we achieve an average accuracy of 90.7% on various objects. Experiments reveal the potential of applying our simulation framework to more complicated manipulation tasks. We open-source our simulation framework at https://github.com/CMURoboTouch/Taxim/tree/taxim-robot.

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results11https://github.com/hsp-iit/prosthetic-grasping-simulation.

In the lateral access environment, the robot be-havior should be planned considering surrounding objects and obstacles because object observation directions and approach angles are limited. To safely retrieve a partially occluded target object in these environments, we have to relocate objects using prehensile actions to create a collision-free path for the target. We propose a learning-based method for object rearrangement planning applicable to objects of various types and sizes in the lateral environment. We plan the optimal rearrangement sequence by considering both collisions and approach angles at which objects can be grasped. The proposed method finds the grasping order through Monte Carlo tree search, significantly reducing the tree search cost using point cloud states. In the experiment, the proposed method shows the best and most stable performance in various scenarios compared to the existing TAMP methods. In addition, we confirm that the proposed method trained in simulation can be easily applied to a real robot without additional fine-tuning, showing the robustness of the proposed method.

Robot assembly discovery (RAD) is a challenging problem that lives at the intersection of resource allocation and motion planning. The goal is to combine a predefined set of objects to form something new while considering task execution with the robot-in-the-loop. In this work, we tackle the problem of building arbitrary, predefined target structures entirely from scratch using a set of Tetris-like building blocks and a robotic manipulator. Our novel hierarchical approach aims at efficiently decomposing the overall task into three feasible levels that benefit mutually from each other. On the high level, we run a classical mixed-integer program for global optimization of block-type selection and the blocks' final poses to recreate the desired shape. Its output is then exploited to efficiently guide the exploration of an underlying reinforcement learning (RL) policy. This RL policy draws its generalization properties from a flexible graph-based representation that is learned through Q-learning and can be refined with search. Moreover, it accounts for the necessary conditions of structural stability and robotic feasibility that cannot be effectively reflected in the previous layer. Lastly, a grasp and motion planner transforms the desired assembly commands into robot joint movements. We demonstrate our proposed method's performance on a set of competitive simulated RAD environments, showcase real-world transfer, and report performance and robustness gains compared to an unstructured end-to-end approach.

Multi-goal policy learning for robotic manipu-lation is challenging. Prior successes have used state-based representations of the objects or provided demonstration data to facilitate learning. In this paper, by hand-coding a high-level discrete representation of the domain, we show that policies to reach dozens of goals can be learned with a single network using Q-learning from pixels. The agent focuses learning on simpler, local policies which are sequenced together by planning in the abstract space. We compare our method against standard multi-goal RL baselines, as well as other methods that leverage the discrete representation, on a challenging block construction domain. We find that our method can build more than a hundred different block structures, and demonstrate forward transfer to structures with novel objects. Lastly, we deploy the policy learned in simulation on a real robot.

Robotic manipulators navigating cluttered shelves or cabinets may find it challenging to avoid contact with obstacles. Indeed, rearranging obstacles may be necessary to access a target. Rather than planning explicit motions that place obstacles into a desired pose, we suggest allowing incidental contacts to rearrange obstacles while monitoring contacts for safety. Bypassing object identification, we present a method for categorizing object motions from tactile data collected from incidental contacts with a capacitive tactile skin on an Allegro Hand. We formalize tactile cues associated with categories of object motion, demonstrating that they can determine with > 90% accuracy whether an object is movable and whether a contact is causing the object to slide stably (safe contact) or tip (unsafe).

The current transformation towards smart manufacturing has led to a growing demand for human-robot collaboration (HRC) in the manufacturing process. Perceiving and understanding the human co-worker's behaviour introduces challenges for collaborative robots to efficiently and effectively perform tasks in unstructured and dynamic environments. Integrating recent data-driven machine vision capabilities into HRC systems is a logical next step in addressing these challenges. However, in these cases, off-the-shelf components struggle due to generalisation limitations. Real-world evaluation is required in order to fully appreciate the maturity and robustness of these approaches. Furthermore, understanding the pure-vision aspects is a crucial first step before combining multiple modalities in order to understand the limitations. In this paper, we propose GoferBot, a novel vision-based semantic HRC system for a real-world assembly task. It is composed of a visual servoing module that reaches and grasps assembly parts in an unstructured multi-instance and dynamic environment, an action recognition module that performs human action prediction for implicit communication, and a visual handover module that uses the perceptual understanding of human behaviour to produce an intuitive and efficient collaborative assembly experience. GoferBot is a novel assembly system that seamlessly integrates all sub-modules by utilising implicit semantic information purely from visual perception.

Using pole and plane objects in lidar SLAM can increase accuracy and decrease map storage requirements compared to commonly-used point cloud maps. However, place recognition and geometric verification using these landmarks is challenging due to the requirement for global matching without an initial guess. Existing works typically only leverage either pole or plane landmarks, limiting application to a restricted set of environments. We present a global data association method for loop closure in lidar scans using 3D line and plane objects simultaneously and in a unified manner. The main novelty of this paper is in the representation of line and plane objects extracted from Iidar scans on the manifold of affine subspaces, known as the affine Grassmannian. Line and plane correspondences are matched using our graph-based data association framework and subsequently registered in the least-squares sense. Compared to pole-only approaches and plane-only approaches, our 3D affine Grassmannian method yields a 71 % and 325 % increase respectively to loop closure recall at 100 % precision on the KITTI dataset and can provide frame alignment with less than 10 cm and 1 deg of error.

Robots that use natural language in collaborative tasks must refer to objects in their environment. Recent work has shown the utility of the linguistic theory of the Givenness Hierarchy (GH) in generating appropriate referring forms. But before referring expression generation, collaborative robots must determine the content and structure of a sequence of utterances, a task known as document planning in the natural language generation community. This problem presents additional challenges for robots in situated contexts, where described objects change both physically and in the minds of their interlocutors. In this work, we consider how robots can “think ahead” about the objects they must refer to and how to refer to them, sequencing object references to form a coherent, easy to follow chain. Specifically, we leverage GH to enable robots to plan their utterances in a way that keeps objects at a high cognitive status, which enables use of concise, anaphoric referring forms. We encode these linguistic insights as a mixed integer program within a planning context, formulating constraints to concisely and efficiently capture GH-theoretic cognitive properties. We demonstrate that this GH-informed planner generates sequences of utterances with high intersentential coherence, which we argue should enable substantially more efficient and natural human-robot dialogue.

Co-speech gestures are a principal component in conveying messages and enhancing interaction experiences between humans and critical ingredients in human-agent interaction, including virtual agents and robots. Existing machine learning approaches have yielded only marginal success in learning speech-to-motion at the frame level. Current methods generate repetitive gesture sequences that lack appropriateness with respect to the speech context. To tackle this challenge, we take inspiration from successes in natural language processing on context and long-term dependencies, and propose a new framework that views text-to-gesture as machine translation, where gestures are words in another (non-verbal) language. We propose a vector-quantized variational autoencoder structure as well as training techniques to learn a rigorous representation of gesture sequences. We then translate input text into a discrete sequence of associated gesture chunks in the learned gesture space. Ultimately, we use translated gesture tokens from the input text as an input to the autoencoder's decoder to produce gesture sequences. Subjective and objective evaluations confirm the success of our approach in terms of appropriateness, human-likeness, and diversity. We also introduce new objective metrics using the quantized gesture representation.

This paper focuses on the problem of smoothing a rotation trajectory corrupted by noise, while simultaneously estimating its corresponding angular velocity and angular acceleration. To this end, we develop a geometric version of the Savitzky-Golay filter on SO(3) that avoids following the conventional practice of first converting the rotation trajectory into Euler-like angles, performing the filtering in this new set of local coordinates, and finally converting the result back on SO (3). In particular, the estimation of the angular acceleration requires the computation of the right-trivialized second covariant derivative of the exponential map on SO (3) with respect to the (+) Cartan-Schouten connection. We provide an explicit expression for this derivative, creating a link to seemingly unrelated existing results concerning the first derivative of the exponential map on SE (3). A numerical example is provided in which we demonstrate the effectiveness and straightforward applicability of the proposed approach. An open implementation of the new geometric Savitzky-Golay filter is also provided.

We present two novel nonlinear MPC formulations for reduced attitude tracking on quadrotors with bidirectional thrust capabilities. Reduced attitude tracking is relevant to recovery from partial thrust loss, which can occur due to the failure of one or more motors. The first formulation builds on a linearization of the quadrotor attitude dynamics on S(2)S(2) to achieve simultaneous tracking of reduced attitude and total thrust targets. The second formulation, meanwhile, accomplishes the same goal using a linearization of the dynamics on the Lie algebra of SO(3)SO(3) and a proposed method for projecting Lie algebra errors onto reduced attitude errors. Both methods achieve global tracking on S(2)S(2) without requiring the use of computationally expensive sequential quadratic program solvers. Through simulations, we show that the second approach generally tracks aggressive attitude references better, while the first controller offers more reliable regulation.

Simultaneous orthophoto stitching during the flight of Unmanned Aerial Vehicles (UAV) can greatly promote the practicability and instantaneity of diverse applications such as emergency disaster rescue, digital agriculture, and cadastral survey, which is of remarkable interest in aerial photogrammetry. However, the inaccurately estimated camera poses and the intuitive fusion strategy of existing methods lead to misalignment and distortion artifacts in orthophoto mosaics. To address these issues, we propose a Georeferenced Real-time Orthophoto Stitching method (GeoROS), which can achieve efficient and accurate camera pose estimation through exploiting geolocation information in monocular visual simultaneous localization and mapping (SLAM) and fuse transformed images via orthogonality-preserving criterion. Specifically, in the SLAM process, georeferenced tracking is employed to acquire high-quality initial camera poses with a geolocation based motion model and facilitate non-linear pose optimization. Meanwhile, we design a georeferenced mapping scheme by introducing robust geolocation constraints in joint optimization of camera poses and the position of landmarks. Finally, aerial images warped with localized cameras are fused by considering both the orthogonality of camera orientation relative to the ground plane and the pixel centrality to fulfill global orthorectification. Besides, we construct two datasets with global navigation satellite system (GNSS) information of different scenarios and validate the superiority of our GeoROS method compared with state-of-the-art methods in accuracy and efficiency.

Personal air transportation on short distances, so-called Urban Air Mobility (UAM), is a trend in modern aviation that raises new challenges as flying in urban areas at low altitudes induces an additional risk to people and properties on the ground. Risk-aware trajectory planning can mitigate the risk by detouring and flying over less populated and thus less risky areas. Existing risk-aware trajectory planning approaches are computationally demanding single-query methods that are impractical for online usage. Moreover, coordinated planning for multiple aircraft is prohibitively expensive. Therefore, we propose to reduce computational demands by determining low-risk areas called safe corridors and creating a roadmap of safe corridors based on multiple least risky trajectories. The created roadmap can be used in graph-based multi-agent planning methods for coordinated trajectory planning. The proposed method has been evaluated in a realistic urban scenario, suggesting a significant computational burden reduction and less risky trajectories than the current state-of-the-art methods.