IROS 2022 Papers

Visual Place Recognition (VPR) has become an indispensable capacity for mobile robots to operate in large-scale environments. Existing methods in this field mostly focus on exploring high-performance encoding strategies, while few attempts are devoted to lightweight models that balance per-formance and computational cost. In this work, we propose a Lightweight Self-attentional Distillation Network (LSDNet) aiming to obtain advantages of both performance and efficiency. (1) From a performance perspective, an attentional encoding strategy is proposed to integrate crucial information in the scene. It extends the NetVlad architecture with a self-attention module to facilitate non-local information interaction between local features. Through further visual word vector rescaling, the final image representation can benefit from both non-local spatial integration and cluster-wise weighting. (2) From an efficiency perspective, LSDNet is built upon a lightweight back-bone. To maintain comparable performance to large backbone models, a dual distillation strategy is introduced. It prompts LSDNet to learn both encoding patterns in the hidden space and feature distributions in the encoding space from the teacher model. Through distillation-augmented training, LSDNet is able to rival the teacher model and outperform SOTA global representations with the same lightweight backbone.

This paper proposes an online large-scale dense mapping method for UAVs with a height of 150–250 meters. We first fuse the GPS with the visual odometry to estimate the scaled poses and sparse points. In order to use the depth of sparse points for depth map, we propose Sparse Confidence Cascade View-Aggregation MVSNet (SCCVA-MVSNet), which projects the depth-converged points in the sliding window on keyframes to obtain a sparse depth map. To weigh the confidence of the depth of each sparse point, we construct sparse confidence by the photometric error. The images of all keyframes, coarse depth, and confidence as the input of CVA-MVSNet to extract features and construct 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Our proposed network utilizes sparse features point information, the output of the network better maintains the consistency of the scale. Our experiments show that MVSNet using sparse feature point information outperforms image-only MVSNet, and our online reconstruction results are comparable to offline reconstruction methods. To benefit the research community, we open our code at https://github.com/hjxwhy/LODM.git

Regular maintenance of pipelines is an important task to ensure oil transportation and other operation (sewers, nature gas). Precise localization of pipeline damage can greatly improve the efficiency of maintenance work. Since the texture similarity and illumination change of pipe, traditional local descriptors for image matching like SIFT, SURF and ORB are easy to suffer from false correspondences. As to remove the false matches, the local neighborhood constraints (LNC) that contain spatial constructs around feature points are proposed. Good correspondences are essential for the high-accuracy localization and mapping solution given the limited textures and illumination in the pipes. The LNC method is also integrated into the state-of-the-art visual SLAM system. The proposed LNC image matching method and the SLAM system are evaluated on datasets gathered from the pipe environment. Compared with other state-of-the-art methods, our LNC image matching method achieves similar or better performance in precision, recall and runtime. The SLAM system provides state estimation and map reconstruction of the pipe in real-time, and the localization error is within 1%.

Visual-inertial-odometry has attracted extensive attention in the field of autonomous driving and robotics. The size of Field of View (FoV) plays an important role in Visual-Odometry (VO) and Visual-Inertial-Odometry (VIO), as a large FoV enables to perceive a wide range of surrounding scene elements and features. However, when the field of the camera reaches the negative half plane, one cannot simply use [u, v, 1]^{T}[u, v, 1]^{T} to represent the image feature points anymore. To tackle this issue, we propose LF-VIO, a real-time VIO framework for cameras with extremely large FoV.We leverage a threedimensional vector with unit length to represent feature points, and design a series of algorithms to overcome this challenge. To address the scarcity of panoramic visual odometry datasets with ground-truth location and pose, we present the PALVIO dataset, collected with a Panoramic Annular Lens (PAL) system with an entire FoV of 36 0^{\circ}\times(40^{\circ}\sim 120^{\circ})0^{\circ}\times(40^{\circ}\sim 120^{\circ}) and an IMU sensor. With a comprehensive variety of experiments, the proposed LF-VIO is verified on both the established PALVIO benchmark and a public fisheye camera dataset with a FoV of 360^{\circ}\times(0^{\circ}\sim 93.5^{\circ})360^{\circ}\times(0^{\circ}\sim 93.5^{\circ}). LF-VIO outperforms state-of-the-art visual-inertial-odometry methods. Our dataset and code are made publicly available at https://github.com/flysoaryun/LF-VIO

This paper proposes a new regularization technique for reinforcement learning (RL) towards making policy and value functions smooth and stable. RL is known for the instability of the learning process and the sensitivity of the acquired policy to noise. Several methods have been proposed to resolve these problems, and in summary, the smoothness of policy and value functions learned mainly in RL contributes to these problems. However, if these functions are extremely smooth, their expressiveness would be lost, resulting in not obtaining the global optimal solution. This paper therefore considers RL under local Lipschitz continuity constraint, so-called L2C2. By designing the spatio-temporal locally compact space for L2C2 from the state transition at each time step, the moderate smoothness can be achieved without loss of expressiveness. Numerical noisy simulations verified that the proposed L2C2 outperforms the task performance while smoothing out the robot action generated from the learned policy.

We tackle the problem of tracking the human lower body as an initial step toward an automatic motion assessment system for clinical mobility evaluation, using a multimodal system that combines Inertial Measurement Unit (IMU) data, RGB images, and point cloud depth measurements. This system applies the factor graph representation to an optimization problem that provides 3-D skeleton joint estimations. In this paper, we focus on improving the temporal consistency of the estimated human trajectories to greatly extend the range of operability of the depth sensor. More specifically, we introduce a new factor graph factor based on Koopman theory that embeds the nonlinear dynamics of several lower-limb movement activities. This factor performs a two-step process: first, a custom activity recognition module based on spatial temporal graph convolutional networks recognizes the walking activity; then, a Koopman pose prediction of the subsequent skeleton is used as an a priori estimation to drive the optimization problem toward more consistent results. We tested the performance of this module a dataset composed of multiple clinical lower-limb mobility tests, and we show that our approach reduces outliers on the skeleton form by almost 1 m, while preserving natural walking trajectories at depths up to more than 10 m.

Whiskers are widely used by animals for sensing physical interactions with their environments. By combining the Kirigami skin pop-up feature and flexible conducting layer, we designed a deployable Kirigami whisker sensor. The sensor can deploy from a flat state to a sensing state while whisker stiffness and initial pop-up angle can be tuned by adjusting the pre-stretch strain. Preliminary results show that the sensor works well both in air and underwater. The sensor is capable of measuring both externally applied forces and water flow.

Kinesthetic teaching allows the direct skill transfer from the human to the robot and has been widely used to teach single arm tasks intuitively. In the bi-manual case, simultaneously moving both end-effectors is challenging due to the high physical and cognitive load imposed to the user. Thus, previous works on bi-manual task teaching resort to less intuitive methods by teaching each arm separately. This in turn requires motion synthesis and synchronization before execution. In this work, we leverage knowledge from the relative task space to facilitate a kinesthetic demonstration by guiding both end-effectors which is more human-like and intuitive way for performing bi-manual tasks. Our method utilizes the notion of virtual fixtures and inertia minimization in the null space of the task. The controller is experimentally validated in a bi-manual task which involves the drawing of a preset line on a workpiece utilizing two KUKA IIWA7 R800 robots. Results from ten participants were compared with a gravity compensation scheme demonstrating improved performance.

Suspended Cable-Driven Parallel Robots (SCDPR) have intriguing capabilities on large scales but still have open challenges in precisely estimating the end-effector pose. The cables exhibit a downward curved shape, also known as cable sag which needs to be accounted for in the pose estimation. The catenary equations can accurately describe this phenomenon but are only accurate in equilibrium conditions. Thus, pose estimation for large-scale SCDPR in dynamic motion is an open challenge. This work proposes a real-time pose estimation algorithm for dynamic trajectories of SCDPRs, which is accurate over large areas. We present a novel approach that considers cable sag to reduce the estimation error for large scales while also employing an Inertial Measurement Unit (IMU) to improve estimation accuracy for dynamic motion. Our approach reduces the RMSE to less than a third compared to standard methods not considering cable sag. Similarly, the inclusion of the IMU reduces the RMSE in dynamic situations by 40% compared to non-IMU aided approaches considering cable sag. Further-more, we evaluate our Extended Kalman Filter (EKF) based algorithm on a real system with ground truth pose information.

As the basics of Visual Simultaneous Localization And Mapping (VSLAM), keyframes play an essential role. In previous works, keyframes are selected according to a series of view change-based strategies for short-term data association (STDA). However, the texture enrichment of frames is always ignored, resulting in the failure of long-term data association (LTDA). In this paper, we propose an information enrichment selection strategy with an information occupancy grid model and a deep descriptor. Frame is expressed by a deep global descriptor for a statistical explainable abstraction, in which the texture enrichment is indicated. Based on the abstraction, an information occupancy grid model is established to measure the information enrichment and the potential LTDA ability. Evaluations on variant datasets are conducted, showing the advantage of our proposed method in terms of keyframe selection and tracking precision. Also, the statistical explainability of the deep descriptor is provided. The proposed keyframe selection strategy can improve LTDA and tracking precision, especially in situations with repeated observations and loop-closures.

Interactive Reinforcement Learning (IntRL) allows human teachers to accelerate the learning process of Reinforcement Learning (RL) robots. However, IntRL has largely been limited to tasks with discrete-action spaces in which actions are relatively slow. This limits IntRL's application to more complicated and challenging robotic tasks, the very tasks that modern RL is particularly well-suited for. We seek to bridge this gap by presenting Continuous Action-space Interactive Reinforcement learning (CAIR): the first continuous action-space IntRL algorithm that is capable of using teacher feedback to out-perform state-of-the-art RL algorithms in those tasks. CAIR combines policies learned from the environment and the teacher into a single policy that proportionally weights the two policies based on their agreement. This allows a CAIR agent to learn a relatively stable policy despite potentially noisy or coarse teacher feedback. We validate our approach in two simulated robotics tasks with easy-to-design and - understand heuristic oracle teachers. Furthermore, we validate our approach in a human subjects study through Amazon Mechanical Turk and show CAIR out-performs the prior state-of-the-art in Interactive RL.

Latch-mediated spring-actuation (LaMSA) is utilized in a majority of jumping robots for its ability to slowly load and quickly release energy to generate high-power movement. Such mechanisms are found in robots that jump off of solid surfaces and even off of water. However, no robot currently employs LaMSA to jump on air. This paper presents the design, modeling, and fabrication of the first LaMSA-driven air jumper, capable of jumping mid-air. Our model informs prototype design and provides insight into the scaling properties of the wing area, wing and fuselage mass, and energy. By successfully applying LaMSA to a new domain, this work lays the foundation for future investigations into high-power airreaction maneuvers, such as in fixed-wing unmanned aerial vehicle (UAV) flight, by enabling instantaneous changes in altitude without the addition of extra on-board motors.

This work presents an online trajectory generation algorithm using a sinusoidal jerk profile. The generator takes initial acceleration, velocity and position as input, and plans a multi-segment trajectory to a goal position under jerk, acceleration, and velocity limits. By analyzing the critical constraints and conditions, the corresponding closed-form solution for the time factors and trajectory profiles are derived. The proposed algorithm was first derived in Mathematica and then converted into a C++ implementation. Finally, the algorithm was utilized and demonstrated in ROS & Gazebo using a UR3 robot. Both the Mathematica and C++ implementations can be accessed at https://github.com/Haoran-Zhao/Jerk-continuous-online-trajectory-generator-with-constraints.git

Fibre jamming is a new and understudied soft robotic mechanism that has previously found success in stiffness-tunable arms and fingers. However, to date researchers have not fully taken advantage of the freedom offered by contemporary fabrication techniques including multi-material 3D printing in the creation of fibre jamming structures. In this research, we present a novel, modular, multi-material, 3D printed, fibre jamming tendon unit for use in a stiffness-tunable compliant robotic ankle, or Jammkle. Its multimaterial printed design offers unparalleled design freedom, enabling application specific tendon design. We develop analytical and finite element models of the tendon unit, showing good agreement with experimental data and numerically explore the design space. Finally, we demonstrate a practical application by integrating multiple tendon units into a robotic ankle and perform extensive testing and characterisation. We show that the Jammkle outperforms comparative leg structures in terms of compliance, damping, and slip prevention.

Depth estimation and semantic edge detection are two key tasks in computer vision, which have made great progress. To date, how to associatively predict the depth and the semantic edge is rarely explored. In this work, we first propose a flexible two-branch framework that can make the two tasks take advantage of each other, achieving a win-win situation. Specifically, for the semantic edge detection branch, an Enhanced Edge Weighting strategy (EEW) is designed, which learns weight information from the by-product of depth branch, depth edge, to enhance edge perception in features. Meanwhile, we make depth estimation benefit from semantic edge detection through introducing Depth Edge Semantic Classification module (DESC). Furthermore, a double reconstruction (D-reconstruction) approach is presented, together with semantic edge-guided disparity smoothing loss to mitigate the ambiguities of the self-supervised manner for depth estimation. Experiments on the Cityscapes dataset demonstrate that our framework outperforms the state-of-the-art method in depth estimation along with a significant improvement in semantic edge detection.

In this paper, we introduce a new concept of incorporating factorized flow maps as mid-level representations, for bridging the perception and the control modules in modular learning based robotic frameworks. To investigate the advantages of factorized flow maps and examine their interplay with the other types of mid-level representations, we further develop a configurable framework, along with four different environments that contain both static and dynamic objects, for analyzing the impacts of factorized optical flow maps on the performance of deep reinforcement learning agents. Based on this framework, we report our experimental results on various scenarios, and offer a set of analyses to justify our hypothesis. Finally, we validate flow factorization in real world scenarios.

Despite significant advancements, collision-free navigation in autonomous driving is still challenging, considering the navigation module needs to balance learning and planning to achieve efficient and effective control of the vehicle. We propose a novel framework of inverse reinforcement learning with hybrid-weight trust-region optimization and curriculum learning (IRL-HC) for autonomous maneuvering. Our method can incorporate both expert demonstration (from real driving) and domain knowledge (hard constraints such as collision avoidance, goal reaching, etc. encoded in reward functions) to learn an effective control policy. The hybrid-weight trustregion optimization is used to determine the difficulty of the task curriculum for fast incremental curriculum learning and improve the efficiency of inverse reinforcement learning by hybrid weight tuning of different sets of hyperparameters. IRL-HC is also compatible with domain-dependent techniques such as learn-from-accident, which can further boost performance. Overall, IRL-HC can reduce the number of collisions up to 48%, increase the training efficiency by 2.8x, and enable the vehicle to drive 10x further compared to other methods.

The combination of collaborative robots and end-to-end AI, promises flexible automation of human tasks in factories and warehouses. However, such promise seems a few breakthroughs away. In the meantime, humans and cobots will collaborate helping each other. For these collaborations to be effective and safe, robots need to model, predict and exploit human's intents for responsive decision making processes. Approximate Bayesian Computation (ABC) is approach to perform probabilistic predictions upon uncertain quantities. ABC includes priors conveniently, leverages sampling algorithms for inference and is flexible to benefit from complex models, e.g. via simulators. However, ABC is known to be computationally too intensive to run at interactive frame rates required for effective human-robot collaboration tasks. In this paper, we formulate human intent prediction as an ABC problem and describe two key performance innovations which allow computations at interactive rates. Our real-world experiments with a collaborative robot set-up, demonstrate the viability of our proposed approach. Experimental evaluations convey the advantages and value of human intent prediction for packing cooperative tasks. Qualitative results show how anticipating human's intent improves human-robot collaboration without compromising safety. Quantitative task fluency metrics confirm the qualitative claims.