IROS 2022 Papers

We address the path planning problem for a team of robots satisfying a complex high-level mission specification given in the form of a Linear Temporal Logic (LTL) formula. The state-of-the-art approach to this problem employs the automata-theoretic model checking technique to solve this problem. This approach involves computation of a product graph of the Büchi automaton generated from the LTL specification and a joint transition system that captures the collective motion of the robots and then computation of the shortest path using Di-jkstra's shortest path algorithm. We propose MT*, an algorithm that reduces the computation burden for generating such plans for multi-robot systems significantly. Our approach generates a reduced version of the product graph without computing the complete joint transition system, which is computationally expensive. It then divides the complete mission specification among the participating robots and generates the trajectories for the individual robots independently. Our approach demonstrates substantial speedup in terms of computation time over the state-of-the-art approach and scales well with both the number of robots and the size of the workspace.

The existing manufacturing systems based on processes involving the transportation of workpiece are unsuitable for large products such as air mobility systems. This study proposes a novel ultra-complex manufacturing system called the “Modularized-Structure and Multiple-Points Simultaneous Machining System (MS-cubic)” based on the concept of intelligent space, which simultaneously performs multiple types of machining processes without moving a workpiece. The system can simultaneously process multiple points and flexibly change its workspace by modularizing its structure. This paper presents a discussion on the requirements and constraints to generate a feasible design of the rail module and the machining unit, which are two main elements of MS-cubic. The performance of the prototype MS-cubic is evaluated, and its stiffness is observed to be sufficient to perform drilling. Furthermore, the modularized design of system enables the fluid and electric power supply for the machining process.

Point cloud semantic segmentation is important for road scene perception, a task for driverless vehicles to achieve full fledged autonomy. In this work, we introduce Mask Point Transformer Network (MPT-Net), a novel architecture for point cloud segmentation that is simple to implement. MPT-Net consists of a local and global feature encoder and a transformer based decoder; a 3D Point-Voxel Convolution encoder backbone with voxel self attention to encode features and a Mask Point Transformer module to decode point features and segment the point cloud. Firstly, we introduce the novel MPT designed to specifically handle point cloud segmentation. MPT offers two benefits. It attends to every point in the point cloud using mask tokens to extract class specific features globally with cross attention, and provide inter-class feature information exchange using self attention on the learned mask tokens. Secondly, we design a backbone to use sparse point voxel convolutional blocks and a self attention block using transformers to learn local and global contextual features. We evaluate MPT-Net on large scale outdoor driving scene point cloud datasets, SemanticKITTI and nuScenes. Our experiments show that by replacing the standard segmentation head with MPT, MPT-Net achieves a state-of-the-art performance over our baseline approach by 3.8% in SemanticKITTI and is highly effective in detecting 'stuffs' in point cloud.

Our goal is to train a neural planner that can capture diverse driving behaviors in complex urban scenarios. We observe that even state-of-the-art neural planners are struggling to perform common maneuvers such as lane change, which is rather natural for human drivers. We propose to explore the multi-modalities in the planning problem and force the neural planner to explicitly consider different policies. This is achieved by generating the future trajectories conditioned on every possible reference line, which could simply be the centerline of the surrounding lanes. We find this simple strategy yet enables the planner to perform rich and complex behaviors. We train our model using real-world driving data and demonstrate the effectiveness of our method through both open-loop and closed-loop evaluations. Project website https://jchengai.github.io/mpnp.

Predicting object motion behaviour is a challenging but crucial task for safe decision making and path planning for an autonomous vehicle. It is challenging in large part due to the uncertain, multi-modal, and practically intractable set of possible agent-agent and agent-space interactions, especially in urban driving settings. Models solely based on constant velocity or social force have an inherent bias and may lead to inaccurate predictions across the prediction horizon whereas purely data driven approaches suffer from a lack of a holistic set of rules governing predictions. We tackle this problem by introducing MPC-PF: a novel potential field-based trajectory predictor that incorporates social interaction and is able to tradeoff between inherent model biases across the prediction horizon. Through evaluation on a variety of common urban driving scenarios, we show that our model is capable of producing accurate predictions for both short and long term timesteps. We also demonstrate the significance of our model architecture through an ablation study.

Visual SLAM systems targeting static scenes have been developed with satisfactory accuracy and robustness. Dynamic 3D object tracking has then become a significant capability in visual SLAM with the requirement of under-standing dynamic surroundings in various scenarios including autonomous driving, augmented and virtual reality. However, performing dynamic SLAM solely with monocular images remains a challenging problem due to the difficulty of asso-ciating dynamic features and estimating their positions. In this paper, we present MOTSLAM, a dynamic visual SLAM system with the monocular configuration that tracks both poses and bounding boxes of dynamic objects. MOTSLAM first performs multiple object tracking (MOT) with associated both 2D and 3D bounding box detection to create initial 3D objects. Then, neural-network-based monocular depth estimation is applied to fetch the depth of dynamic features. Finally, camera poses, object poses, and both static, as well as dynamic map points, are jointly optimized using a novel bundle adjustment. Our experiments on the KITTI dataset demonstrate that our system has reached best performance on both camera ego-motion and object tracking on monocular dynamic SLAM.

This paper proposes a new network for reconstructing multi-object point cloud. Different from previous networks which reconstruct multi-object point cloud as a whole, our network iteratively reconstructs each individual object point cloud from a frame of multi-object point cloud. To achieve this goal, we have designed MO-Transformer, a transformer-based autoregressive network. During training, MO-Transformer takes a frame of multi-object point cloud and individual object point clouds as input. During testing, MO-Transformer iteratively reconstructs individual object point clouds only based on the input multi-object point cloud. To train the proposed MO-Transformer, we design a new loss function called separate Chamfer distance (SCD). In addition, we prove that SCD is an upper bound of the traditional Chamfer distance calculated based on the entire multi-object point cloud. The reconstruction experiment verifies the efficacy of our network in multi-object point cloud reconstruction. Furthermore, the reconstruction experiment also investigates the effect of different dimensions using a series of datasets. The ablation study experiment verifies the necessity of SCD in training MO-Transformer.

Inspired by the fact that humans use diverse sensory organs to perceive the world, sensors with different modalities are deployed in end-to-end driving to obtain the global context of the 3D scene. In previous works, camera and LiDAR inputs are fused through transformers for better driving performance. These inputs are normally further interpreted as high-level map information to assist navigation tasks. Nevertheless, extracting useful information from the complex map input is challenging, for redundant information may mislead the agent and negatively affect driving performance. We propose a novel approach to efficiently extract features from vectorized High-Definition (HD) maps and utilize them in end-to-end driving tasks. In addition, we design a new expert to enhance the model performance by considering multi-road rules. Experimental results prove that both proposed improvements enable our agent to achieve superior performance compared with other methods.

This paper presents a framework for Multi-Modal SLAM (MIMOSA) that utilizes a nonlinear factor graph as the underlying representation to provide loosely-coupled fusion of any number of sensing modalities. Tailored to the goal of enabling resilient robotic autonomy in GPS-denied and perceptually-degraded environments, MIMOSA currently contains modules for pointcloud registration, fusion of multiple odometry estimates relying on visible-light and thermal vision, as well as inertial measurement propagation. A flexible back-end utilizes the estimates from various modalities as relative transformation factors. The method is designed to be robust to degeneracy through the maintenance and tracking of modality-specific health metrics, while also being inherently tolerant to sensor failure. We detail this framework alongside our implementation for handling high-rate asynchronous sensor measurements and evaluate its performance on data from autonomous subterranean robotic exploration missions using legged and aerial robots.

Simultaneous Localization and Mapping (SLAM) systems are fundamental building blocks for any autonomous robot navigating in unknown environments. The SLAM implementation heavily depends on the sensor modality employed on the mobile platform. For this reason, assumptions on the scene's structure are often made to maximize estimation accuracy. This paper presents a novel direct 3D SLAM pipeline that works independently for RGB-D and LiDAR sensors. Building upon prior work on multi-cue photometric frame-to-frame alignment [4], our proposed approach provides an easy-to-extend and generic SLAM system. Our pipeline requires only minor adaptations within the projection model to handle different sensor modalities. We couple a position tracking system with an appearance-based relocalization mechanism that handles large loop closures. Loop closures are validated by the same direct registration algorithm used for odometry estimation. We present comparative experiments with state-of-the-art approaches on publicly available benchmarks using RGB-D cameras and 3D LiDARs. Our system performs well in heterogeneous datasets compared to other sensor-specific methods while making no assumptions about the environment. Finally, we release an open-source C++ implementation of our system.

Portfolio-based algorithm selection can help in choosing the best suited algorithm for a given task while leveraging the complementary strengths of the candidates. Solving the Multi-Agent Path Finding (MAPF) problem optimally has been proven to be NP-Hard. Furthermore, no single optimal algorithm has been shown to have the fastest runtime for all MAPF problem instances, and there are no proven approaches for when to use each algorithm. To address these challenges, we develop MAPFASTER, a smaller and more accurate deep learning based architecture aiming to be deployed in fleet management systems to select the fastest MAPF solver in a multi-robot setting. MAPF problem instances are encoded as images and passed to the model for classification into one of the portfolio's candidates. We evaluate our model against state-of-the-art Optimal-MAPF-Algorithm selectors, showing +5.42% improvement in accuracy while being 7.1× faster to train. The dataset, code and analysis used in this research can be found at https://github.com/jeanmarcalkazzi/mapfaster.

Incorrectly sized balloon catheters can lead to increased post-surgical complications, yet even with preoperative imaging, correct selection remains a challenge. With limited feedback during surgery, it is difficult to verify correct deployment. We propose the use of integrated impedance measurements and Electrical Impedance Tomography (EIT) imaging to assess the deformation of the balloon and determine the size and shape of the surrounding lumen. Previous work using single impedance measurements, or pressure data and analytical models, whilst demonstrating high sizing accuracy, have assumed a circular cross section. Here we extend these methods by adding a multitude of electrodes to detect elliptical and occluded lumen and obtain EIT images to localise deformations. Using a 14 Fr (5.3 mm) catheter as an example, numerical simulations were performed to find the optimal electrode configuration of two rings of 8 electrodes spaced 10 mm apart. The simulations predicted that the maximum detectable aspect ratio decreased from 0.9 for a 14mm balloon to 0.5 at 30mm. The sizing and ellipticity detection results were verified experimentally. A prototype robotic balloon catheter was constructed to automatically inflate a compliant balloon while simultaneously recording EIT and pressure data. Data were collected in experiments replicating stenotic vessels with an elliptical and asymmetrical profile, and the widening of a lumen during angioplasty. After calibration, the system was able to correctly localise the occlusion and detect aspect ratios of 0.75. EIT images further localised the occlusion and visualised the dilation of the lumen during balloon inflation.

This study focuses on localization and mapping for UGVs when they are deployed in environments with non-level roads. In these scenarios, the vehicles need to travel through flat but not necessarily level grounds, i.e., ascent or descent, which may cause drifts of the robot pose and distortion of the map. We develop a low-drift LiDAR odometry and mapping approach for the UGV with LiDAR as the only exteroceptive sensor. A factor-graph based pose optimization method is developed with a specifically designed factor named slope factor. This factor includes the slope information that is estimated from a real-time LiDAR data stream. The slope information is also used to enhance the loop-closure detection procedure. Moreover, an incremental pitch estimation mechanism is designed to achieve further pose estimation refinement. We demonstrate the effectiveness of the developed framework in real-world environments. The odometry drift is lower and the map is more precise than experiments with the state-of-the-arts. Notably, on the Kitti dataset, our method also exhibits convincing performance, demonstrating its strength in more general application scenarios.

Several methods of semantic segmentation using light detection and ranging (LiDAR) sensors have been proposed for the recognition of surrounding objects by autonomous driving cars. LiDAR is a sensor that compensates for the weaknesses of other sensors, such as cameras or radar systems, and semantic segmentation assigns a class label to each point in the LiDAR point cloud. Recently, real-time semantic segmentation methods that are capable of processing LiDAR point clouds at frame rates have been proposed. Real-time semantic segmentation is essential for the autonomous driving system because it can output class labels for LiDAR point clouds at high speeds. However, this segmentation method suffers from a delay equal to processing time. To address this challenge, we propose a novel method that combines SalsaNext [1], a method of real-time LiDAR semantic segmentation, and semantic forecasting, which predicts the results of future semantic segmentation. We quantitatively evaluate our method using the Semantic-KITTI dataset, which comprises point cloud data acquired from the LiDAR sensor in the real world, and compare the latency and accuracy of our method with other semantic segmentation methods. Consequently, our method is found to be capable of operating in real-time and with low-latency, and it can achieve a performance similar to that of previously reported real-time semantic segmentation methods.

Despite the progress in legged robotic locomotion, autonomous navigation in unknown environments remains an open problem. Ideally, the navigation system utilizes the full potential of the robots' locomotion capabilities while operating within safety limits under uncertainty. The robot must sense and analyze the travers ability of the surrounding terrain, which depends on the hardware, locomotion control, and terrain properties. It may contain information about the risk, energy, or time consumption needed to traverse the terrain. To avoid hand-crafted traversability cost functions we propose to collect traversability information about the robot and locomotion policy by simulating the traversal over randomly generated terrains using a physics simulator. Thousand of robots are simulated in parallel controlled by the same locomotion policy used in reality to acquire 57 years of real-world locomotion experience equivalent. For deployment on the real robot, a sparse convolutional network is trained to predict the simulated traversability cost, which is tailored to the deployed locomotion policy, from an entirely geometric representation of the envi-ronment in the form of a 3D voxel-occupancy map. This rep-resentation avoids the need for commonly used elevation maps, which are error-prone in the presence of overhanging obstacles and multi-floor or low-ceiling scenarios. The effectiveness of the proposed travers ability prediction network is demonstrated for path planning for the legged robot ANY mal in various indoor and natural environments.

We present a locally optimal tracking controller for Cable Driven Parallel Robot (CDPR) control based on a time-varying Linear Quadratic Gaussian (TV-LQG) controller. In contrast to many methods which use fixed feedback gains, our time-varying controller computes the optimal gains depending on the location in the workspace and the future trajectory. Meanwhile, we rely heavily on offline computation to reduce the burden of online implementation and feasibility checking. Following the growing popularity of probabilistic graphical models for optimal control, we use factor graphs as a tool to formulate our controller for their efficiency, intuitiveness, and modularity. The topology of a factor graph encodes the relevant structural properties of equations in a way that facilitates insight and efficient computation using sparse linear algebra solvers. We first use factor graph optimization to compute a nominal trajectory, then linearize the graph and apply variable elimination to compute the locally optimal, time varying linear feedback gains. Next, we leverage the factor graph formulation to compute the locally optimal, time-varying Kalman Filter gains, and finally combine the locally optimal linear control and estimation laws to form a TV-LQG controller. We compare the tracking accuracy of our TV-LQG controller to a state-of-the-art dual-space feed-forward controller on a 2.9m x 2.3m, 4-cable planar robot and demonstrate improved tracking accuracies of 0.8° and 11.6 mm root mean square error in rotation and translation respectively.

Minimally invasive surgery requires real-time tool tracking to guide the surgeon where depth perception and visual occlusion present navigational challenges. Although vision-based and external sensor-based tracking methods exist, fibre-optic sensing can overcome their limitations as they can be integrated directly into the device, are biocompatible, small, robust and geometrically versatile. In this paper, we integrate a fibre Bragg grating-based shape sensor into a soft robotic device. The soft robot is the pneumatically attachable flexible (PAF) rail designed to act as a soft interface between manipulation tools and intra-operative imaging devices. We demonstrate that the shape sensing fibre can detect the location of the tools paired with the PAF rail, by exploiting the change in curvature sensed by the fibre when a strain is applied to it. We then validate this with a series of grasping tasks and continuous US swipes, using the system to detect in real-time the location of the tools interacting with the PAF rail. The overall location-sensing accuracy of the system is 64.6%, with a margin of error between predicted location and actual location of 3.75 mm.

Transformer-based visual object tracking has been utilized extensively. However, the Transformer structure is lack of enough inductive bias. In addition, only focusing on encoding the global feature does harm to modeling local details, which restricts the capability of tracking in aerial robots. Specifically, with local-modeling to global-search mechanism, the proposed tracker replaces the global encoder by a novel local-recognition encoder. In the employed encoder, a local-recognition attention and a local element correction network are carefully designed for reducing the global redundant information interference and increasing local inductive bias. Meanwhile, the latter can model local object details precisely under aerial view through detail-inquiry net. The proposed method achieves competitive accuracy and robustness in several authoritative aerial benchmarks with 316 sequences in total. The proposed tracker's practicability and efficiency have been validated by the real-world tests. The source code is available at https://github.com/vision4robotics/LPAT.

A large-scale dump truck that automatically transports earth and sand in cooperation with a human-operated backhoe is of interest to the construction industry. A human-operated dump truck generally drives slightly past the desired loading position and then backs up to it for loading the sediment. The turning and loading positions are subjectively decided according to the working posture of the backhoe and the surrounding environment, and the safety margin of cooperative works. Backhoe operators want to perform the same maneuvers for human-operated/automated dump trucks. The movements of the autonomous vehicle should be similar to those of a human-operated one. However, it is difficult to derive a human-like path that does more than minimize costs. This study proposes a path-planning method that generates a path including a turning back, according to the changing backhoe position and surrounding conditions. We modeled the positional relationship during loading between a backhoe and dump truck, determining the loading and turning positions and related parameters from operational data collected in trials with human-operated construction vehicles. The proposed method allowed the autonomous dump truck path to resemble a human-like one. The authors have retrofitted an existing large-scale six-wheeled dump truck for automatic operation. Automatic loading in cooperation with a human-operated backhoe was realized all 17 times using the retrofitted dump. The average stopping accuracy was 0.57 m and 9.7°.