IROS 2022 Papers

We present a proprioceptive teleoperation system that uses a reflexive grasping algorithm to enhance the speed and robustness of pick-and-place tasks. The system consists of two manipulators that use quasi-direct-drive actuation to provide highly transparent force feedback. The end-effector has bimodal force sensors that measure 3-axis force information and 2-dimensional contact location. This information is used for anti-slip and re-grasping reflexes. When the user makes contact with the desired object, the re-grasping reflex aligns the gripper fingers with antipodal points on the object to maximize the grasp stability. The reflex takes only 150ms to correct for inaccurate grasps chosen by the user, so the user's motion is only minimally disturbed by the execution of the re-grasp. Once antipodal contact is established, the anti-slip reflex ensures that the gripper applies enough normal force to prevent the object from slipping out of the grasp. The combination of proprioceptive manipulators and reflexive grasping allows the user to complete teleoperated tasks with precision at high speed.

Transfer learning based approaches have recently achieved promising results on the few-shot detection task. These approaches however suffer from “catastrophic forgetting” issue due to finetuning of base detector, leading to sub-optimal performance on the base classes. Furthermore, the slow convergence rate of stochastic gradient descent (SGD) results in high latency and consequently restricts real-time applications. We tackle the aforementioned issues in this work. We pose few-shot detection as a hierarchical learning problem, where the novel classes are treated as the child classes of existing base classes and the background class. The detection heads for the novel classes are then trained using a specialized optimization strategy, leading to significantly lower training times compared to SGD. Our approach obtains competitive novel class performance on few-shot MS-COCO benchmark, while completely retaining the performance of the initial model on the base classes. We further demonstrate the application of our approach to a new class-refined few-shot detection task.

To navigate safely, it is essential for a robot to detect all kinds of moving objects that could possibly interfere with the own trajectory. For common object classes, like cars, regular pedestrians, and trucks, there are large scale datasets as well as corresponding machine learning techniques, which provide remarkable results in commonly available detection benchmarks. A big challenge that remains, are less frequent classes, which are not part of a dataset in a sufficient number and variation. Dynamic occupancy grids are a promising approach for detection of moving objects in point clouds since they impose only a few assumptions about the objects' appearance and shape. Typically, they use particle filters to detect motion of occupancy in the grid. Existing approaches, however, often generate false positives at long obstacles because particles move along them. Therefore we propose a highly efficient approach, which performs the classification in a more structured and conservative way by making extensive use of available free space information. As a result, much less false positives are generated while the number of false negatives remains low. Our approach can be used to complement CNN-based object detections in order to detect both, frequent and uncommon object classes reliably. By using polar data structures that match the polar measurement principle, we are able to process even large point clouds of modern LiDARs with 128 lasers efficiently.

This paper presents a fast cost-aware any-angle path planning algorithm for aerial robots in 3D building-like environments. The approach integrates Euclidean Distance Fields (EDF) and Lazy Theta* algorithm to compute safe and smooth paths. We show how to consider the analytical proper-ties of EDFs for polygonal obstacles to get an approximation of the cost along the line of sight segments of the planner, reducing the computational requirements. Numerous tests in a realistic building-like environment are performed to evaluate the proposed algorithm with respect to other heuristic search algorithms considering the distance cost by using an EDF. The results show that the proposed algorithm considerably reduces the computation time in indoor and outdoor environments enabling fast, safe and smooth paths.

In recent years, mobile robots are becoming ambitious and deployed in large-scale scenarios. Serving as a high-level understanding of environments, a sparse skeleton graph is beneficial for more efficient global planning. Currently, existing solutions for skeleton graph generation suffer from several major limitations, including poor adaptiveness to different map representations, dependency on robot inspection trajectories and high computational overhead. In this paper, we propose an efficient and flexible algorithm generating a trajectory-independent 3D sparse topological skeleton graph capturing the spatial structure of the free space. In our method, an efficient ray sampling and validating mechanism are adopted to find distinctive free space regions, which contributes to skeleton graph vertices, with traversability between adjacent vertices as edges. A cycle formation scheme is also utilized to maintain skeleton graph compactness. Benchmark comparison with state-of-the-art works demonstrates that our approach generates sparse graphs in a substantially shorter time, giving high-quality global planning paths. Experiments conducted in real-world maps further validate the capability of our method in real-world scenarios. Our method will be made open source to benefit the community.

Mobility-on-demand systems are transforming the way we think about the transportation of people and goods. Most research effort has been placed on scalability issues for systems with a large number of agents and simple pickup/drop-off demands. In this paper, we consider fair multi-vehicle route planning with streams of complex, temporal logic transportation demands. We consider an approximately envy-free fair allocation of demands to limited-capacity vehicles based on agents' accumulated utility over a finite time horizon, representing for example monetary reward or utilization level. We propose a scalable approach based on the construction of assignment graphs that relate agents to routes and demands, and pose the problem as an Integer Linear Program (ILP). Routes for assignments are computed using automata-based methods for each vehicle and demands sets of size at most the capacity of the vehicle while taking into account their pickup wait time and delay tolerances. In addition, we integrate utility-based weights in the assignment graph and ILP to ensure approximative fair allocation. We demonstrate the computational and operational performance of our methods in ride-sharing case studies over a large environment in mid-Manhattan and Linear Temporal Logic demands with stochastic arrival times. We show that our method significantly decreases the utility deviation between agents and the vacancy rate.

Dynamic games feature a state-space complexity that scales superlinearly with the number of players. This makes this class of games often intractable even for a handful of players. We introduce the factorization process of dynamic games as a transformation leveraging the independence of players at equilibrium to build a leaner game graph. When applicable, it yields fewer nodes, fewer players per game node, hence much faster solutions. While for the general case checking for independence of players requires to solve the game itself, we observe that for dynamic games in the robotic domain there exist exact heuristics based on the spatio-temporal occupancy of the individual players. We validate our findings in realistic autonomous driving scenarios showing that already for a 4-player intersection we have a reduction of game nodes and solving time close to 99%.

Recent years have seen the introduction of more affordable but less accurate 2D range sensors whose field of view is 2\pi2\pi. Scan-matching with these has been insufficiently researched, while being a challenge due to these sensors' increased measurement uncertainty. This paper proposes a real-time method for matching scans extracted from panoramic 2D LIDAR sensors. The method leverages properties of the Fourier transform which arise due to the periodicity of the range signal. Matching is performed in a correspondenceless manner. The proposed method outperforms established scan-matching methods in terms of pose accuracy and robustness in tests on public domain data, and over noise levels of commercially available sensors. The source code is available for download.

We present a method for autonomous exploration of large-scale unknown environments under mission time con-straints. We start by proposing the Frontloaded Information Gain Orienteering Problem (FIG-OP) - a generalization of the traditional orienteering problem where the assumption of a reliable environmental model no longer holds. The FIG-OP ad-dresses model uncertainty by frontloading expected information gain through the addition of a greedy incentive, effectively expe-diting the moment in which new area is uncovered. In order to reason across multi-kilometer environments, we solve FIG-OP over an information-efficient world representation, constructed through the aggregation of information from a topological and metric map. Our method was extensively tested and field-hardened across various complex environments, ranging from subway systems to mines. In comparative simulations, we observe that the FIG-OP solution exhibits improved coverage efficiency over solutions generated by greedy and traditional orienteering-based approaches (i.e. severe and minimal model uncertainty assumptions, respectively).

In recent years, Visual-Inertial Odometry (VIO) has achieved many significant progresses. However, VIO meth-ods suffer from localization drift over long trajectories. In this paper, we propose a First-Estimates Jacobian Visual-Inertial-Ranging Odometry (FEJ-VIRO) to reduce the localization drifts of VIO by incorporating ultra-wideband (UWB) ranging measurements into the VIO framework consistently. Consid-ering that the initial positions of UWB anchors are usually unavailable, we propose a long-short window structure to initialize the UWB anchors' positions as well as the covariance for state augmentation. After initialization, the FEJ - VIRO estimates the UWB anchors' positions simultaneously along with the robot poses. We further analyze the observability of the visual-inertial-ranging estimators and proved that there are four unobservable directions in the ideal case, while one of them vanishes in the actual case due to the gain of spurious information. Based on these analyses, we leverage the FEJ technique to enforce the unobservable directions, hence reducing inconsistency of the estimator. Finally, we validate our analysis and evaluate the proposed FEJ-VIRO with both simulation and real-world experiments.

Hierarchical coordination of controllers often uses symbolic state representations that fully abstract their underlying low-level controllers, treating them as “black boxes” to the symbolic action abstraction. This paper proposes a framework to realize robust behavior, which we call Feasibility-based Control Chain Coordination (FC3). Our controllers expose the geometric features and constraints they operate on. Based on this, FC3 can reason over the controllers' feasibility and their sequence feasibility. For a given task, FC3 first automatically constructs a library of potential controller chains using a symbolic action tree, which is then used to coordinate controllers in a chain, evaluate task feasibility, as well as switching between controller chains if necessary. In several real-world experiments we demonstrate FC3, s robustness and awareness of the task's feasibility through its own actions and gradual responses to different interferences.

In this paper, we propose a novel variable-length estimation approach for shape sensing of extensible soft robots utilizing fiber Bragg gratings (FBGs). Shape reconstruction from FBG sensors has been increasingly developed for soft robots, while the narrow stretching range of FBG fiber makes it difficult to acquire accurate sensing results for extensible robots. Towards this limitation, we newly introduce an FBG-based length sensor by leveraging a rigid curved channel, through which FBGs are allowed to slide within the robot following its body extension/compression, hence we can search and match the FBGs with specific constant curvature in the fiber to determine the effective length. From the fusion with the above measurements, a model-free filtering technique is accordingly presented for simultaneous calibration of a variable-length model and temporally continuous length estimation of the robot, enabling its accurate shape sensing using solely FBGs. The performances of the proposed method have been experimentally evaluated on an extensible soft robot equipped with an FBG fiber in both free and unstructured environments. The results concerning dynamic accuracy and robustness of length estimation and shape sensing demonstrate the effectiveness of our approach.

To achieve accurate and robust pose estimation in Simultaneous Localization and Mapping (SLAM) task, multisensor fusion is proven to be an effective solution and thus provides great potential in robotic applications. This paper proposes FAST-LIVO, a fast LiDAR-Inertial-Visual Odometry system, which builds on two tightly-coupled and direct odometry subsystems: a VIO subsystem and a LIO subsystem. The LIO subsystem registers raw points (instead of feature points on e.g., edges or planes) of a new scan to an incrementally-built point cloud map. The map points are additionally attached with image patches, which are then used in the VIO subsystem to align a new image by minimizing the direct photometric errors without extracting any visual features (e.g., ORB or FAST corner features). To further improve the VIO robustness and accuracy, a novel outlier rejection method is proposed to reject unstable map points that lie on edges or are occluded in the image view. Experiments on both open data sequences and our customized device data are conducted. The results show our proposed system outperforms other counterparts and can handle challenging environments at reduced computation cost. The system supports both multi-line spinning LiDARs and emerging solid-state LiDARs with completely different scanning patterns, and can run in real-time on both Intel and ARM processors. We open source our code and dataset of this work on Github22https://github.com/hku-mars/FAST-LIVO to benefit the robotics community.

Path planning in unknown environments remains a challenging problem, as the environment is gradually observed during the navigation, the underlying planner has to update the environment representation and replan, promptly and constantly, to account for the new observations. In this paper, we present a visibility graph-based planning framework capable of dealing with navigation tasks in both known and unknown environments. The planner employs a polygonal representation of the environment and constructs the representation by extracting edge points around obstacles to form enclosed polygons. With that, the method dynamically updates a global visibility graph using a two-layered data structure, expanding the visibility edges along with the navigation, and removing edges that become occluded by newly observed obstacles. When navigating in unknown environments, the method is attemptable in discovering a way to the goal by picking up the environment layout on the fly, updating the visibility graph, and fast replanning corresponding to the newly observed environment. We evaluate the method in simulated and real-world settings. The method shows the capability to attempt and navigate through unknown environments, reducing travel time by up to 12-47% from search-based methods: A*, D* Lite, and more than 24-35% from sampling-based methods: RRT*, BIT*, and SPARS.

This paper explores a novel approach to dexterous manipulation, aimed at levels of speed, precision, robustness, and simplicity suitable for practical deployment. The enabling technology is a Direct-drive Hand (DDHand) comprising two fingers, two DOFs each, that exhibit high speed and a light touch. The test application is the dexterous manipulation of three small and irregular parts, moving them to a grasp suitable for a subsequent assembly operation, regardless of initial presentation. We employed four primitive behaviors that use ground contact as a “third finger”, prior to or during the grasp process: pushing, pivoting, toppling, and squeeze-grasping. In our experiments, each part was presented from 30 to 90 times randomly positioned in each stable pose. Success rates varied from 83% to 100%. The time to manipulate and grasp was 6.32 seconds on average, varying from 2.07 to 16 seconds. In some cases, performance was robust, precise, and fast enough for practical applications, but in other cases, pose uncertainty required time-consuming vision and arm motions. The paper concludes with a discussion of further improvements required to make the primitives robust, eliminate uncertainty, and reduce this dependence on vision and arm motion.

2D laser range-finders and depth-cameras are usually equipped on service robots. But there are rarely calibration methods of them. This paper proposes an extrinsic calibration method of a 2D laser range-finder and a depth-camera using an orthogonal trihedron. The trihedron with orthogonal assumptions is taken as a reference frame to roughly estimate the relative pose between the sensors by solving a perspective-three-point (P3P) problem and basis-to-basis correspondence. Then, the estimated relative pose is refined via non-linear optimization based on line-to-plane constraints. Unlike other works which require enough motion, only one-shot observation is required, and it is insensitive to sensor ranging noise and the manufacturing errors of calibration targets. Verified by simulation and real experiments, the proposed method is simple, effective and accurate.

Remotely controlled hydraulically driven robots are expected to play an important role in extreme environments such as disaster sites, and force feedback is effective for improving the fidelity of the remote environment and the work efficiency. However, it is not reasonable to attach a force sensor directly to the end-point of a hydraulically driven robot. In a previous study, the authors showed that the impact forces, which are important information to improve the fidelity of the remote environment and work efficiency, can be estimated by using the information of acceleration of each link in addition to the cylinder pressures. In this paper, we investigated how many accelerometers and where those accelerometers should be attached on each link by using an index called GDOP (Geometric Dilution Of Precision) to improve the accuracy of impact force estimation. The experimental results show that although the estimation accuracy could not be improved significantly by rearranging the accelerometers, the effect of reducing the noise of the estimated load due to the sensor noise was confirmed.

Learning from Demonstration (LfD) requires methodologies able to generalize tasks in new situations. This paper studies the use of virtual demonstrations to extend the extrapolation capabilities of probabilistic motion models such as the traPPCA method. Similarly to other LfD methods, traPPCA is able to calculate new trajectories very fast, but does not generalize well outside the area covered by the demonstrations. Another approach, the invariants method, shows outstanding generalization capabilities thanks to its shape-preserving prop-erties, while being limited by long computation times. The pro-posed methodology combines the advantages of the two methods by learning traPPCA models using virtual demonstrations generated by the invariants method. The proposed approach is analyzed in three case studies. Furthermore, a comparison is made between learning with virtual demonstrations and learning with only real demonstrations. The results encourage the use of virtual demonstrations to extend the extrapolation capabilities of probabilistic motion models and hence reduce the required number of real demonstrations. The latter has the potential of reducing the cost of commissioning robot tasks.

We address the joint task allocation and path planning problem whereby an operator with a fleet of vehicles must assign multiple tasks to each vehicle, while ensuring collision-free paths for them such that the total travel cost is minimized. Instead of sequentially solving the task allocation problem first, and then resolving all predicted collisions, i.e. conflicts between vehicles, we propose a novel method that solves in a simultaneous way task allocation and multi-agent path planning. Specifically, we introduce an extension of the Time Dependent Vehicle Routing Problem (TDVRP) whereby we propose to integrate conflicts information into a time dependent cost function used in the task allocation resolution. We compare our approach to two baseline approaches that both use a standard Capacitated VRP (CVRP) solver, a “one-shot” method and a “multi-shot” method. We perform simulations on benchmark realistic warehouse scenarios and the obtained results show that our proposed approach is able to generate improvements in the solutions costs compared to the baseline approaches.