IROS 2023 Papers

Traditional methods based on Lyapunov analysis and learning-based approaches such as reinforcement learning (RL) are two powerful tools in visual servo tasks. Traditional methods are interpretable and their stability can be guar-anteed by Lyapunov analysis. However, they tend to have a high dependency on an accurate system dynamic model. RL approaches learn to act based on past experiences and thus have higher adaptability on disturbances and errors. However, the training process is long and its safety or stability is generally hard to guarantee, making real-world training risky. In this paper, we propose a residual RL framework for training a multicopter to finish visual servo tasks under disturbances, guided by the system safety in terms of system Lyapunov function. Such an approach compensates for the lack of disturbance-rejection ability of the traditional method, and optimizes stability explicitly so the RL agent makes safer actions both during the training and in the final policy. A comparison between our approach and the baselines is provided in simulation, and real-world experiments on a multicopter are also carried out to show our effectiveness. We believe that this work moves one step toward achieving RL applications on real-world robotic systems.

To support sustainable infrastructure on the Moon, NASA must leverage robots to extract lunar resources for in-situ processing and construction. As part of this effort, NASA is launching the in-situ resource utilization (ISRU) Pilot Excavator later this decade to validate a robotic regolith excavator based on the Regolith Advanced Surface Systems Operations Robot (RASSOR). RASSOR is designed to extract and transport regolith to meet the needs of ISRU architectures. During its mission, Pilot Excavator will be tasked with driving in test patterns to demonstrate the operational concept. One possible test pattern is a circular trajectory around the lander while avoiding surface hazards such as lunar rocks. To this end, we utilize dynamic movement primitives to represent navigation sequences as primitive trajectories. We introduce a novel obstacle avoidance parameter, which is configured to avoid rocks throughout testing exercises. We demonstrate the effectiveness our method in a newly developed simulation tool called the Simulated Excavation Environment for Lunar Operations (SEELO) using models based on the NASA RASSOR 2.0 excavator. Our results show that the robot is able to safety and robustly navigate the lunar surface with densely populated rock obstacles while retaining the desired circle pattern behavior.

Predicting the future trajectories of other agents in the scene fast and effectively is crucial for autonomous driving systems. We note that high-quality predictions require us to take into account the subjective initiative of the target agents, which is reflected by the fact that they themselves make decisions based on their own predictions about the future, just like our ego vehicle's prediction-planning system. However, this characteristic has been neglected in previous studies. We introduce Look Before You Drive (LBYD), a two-stage approach that explicitly incorporates both past observations and future estimates to make predictions. To get a preliminary estimate of the future, we propose a neat and effective baseline capable of making predictions for multiple agents simultaneously. We use only the most basic structures, mainly Transformer, to ensure sufficient inference speed and room for expansion. On this basis, we cooperatively train two networks to enable the coarse estimates to boost final forecasting. Our experiments demonstrate that LBYD can significantly surpass the baseline performance. Moreover, while state-of-the-art methods rely on considering heterogeneity and artificially designed inductive biases for attention modeling, LBYD performs on par with SOTA without them on both the Argoverse 1 and the large scale Argoverse 2 datasets, and can run at 67 FPS on an RTX 3090 GPU.

The goal of visual object navigation for an agent is to find the target objects accurately. Recent works mainly focus on the feature of embedding, attempting to learn better features with different variants, such as object distribution and graph representations. However, some typical navigation problems in complex environments, such as partially known and obstacle problems, may not be effectively addressed by previous feature embedding methods. In this paper, we propose a framework with a long-short objective policy, where the hidden states are classified according to the navigation objectives at that moment and separately rewarded. Specifically, we consider two objectives: the long-term objective is to go closer to the target, and the short-term objective is for obstacle avoidance and exploration. To alleviate the effect of long-term and short-term alternation, we build a state memory and propose an adjustment gate to update the state memory. Finally, all past hidden states are reweighted and combined for action prediction with an action-boosting gate. Experimental results on RoboTHOR show that the proposed method can significantly outperform the state-of-the-art.

Onboard sensors, such as cameras and thermal sensors, have emerged as effective alternatives to Global Positioning System (GPS) for geo-Iocalization in Unmanned Aerial Vehicle (UAV) navigation. Since GPS can suffer from signal loss and spoofing problems, researchers have explored camera-based techniques such as Visual Geo-Iocalization (VG) using satellite RGB imagery. Additionally, thermal geo-Iocalization (TG) has become crucial for long-range UAV flights in low-illumination environments. This paper proposes a novel thermal geo-Iocalization framework using satellite RGB imagery, which includes multiple domain adaptation methods to address the limited availability of paired thermal and satellite images. The experimental results demonstrate the effectiveness of the proposed approach in achieving reliable thermal geo-Iocalization performance, even in thermal images with indistinct self-similar features. We evaluate our approach on real data collected onboard a UAV. We also release the code and Boson-nighttime, a dataset of paired satellite-thermal and unpaired satellite images for thermal geo-Iocalization with satellite imagery. To the best of our knowledge, this work is the first to propose a thermal geo-Iocalization method using satellite RGB imagery in long-range flights.

In-situ observation of the deep-sea floor is a fundamental need for marine sciences and ecosystem monitoring. This work proposes a novel robotic approach for benthic observations in the deep sea using underwater gliders. The glider is equipped with a downward looking camera system to acquire high resolution optical images of the seafloor. The system works fully autonomous and has the potential for long-endurance missions at low deployment costs. Key factors for long battery lifetime are a low power consumption during idle phases, a timely activation of the main imaging system and an efficient lighting setup. The results of our initial trials at sea using a 1000m glider show the applicability of the approach for marine science applications. We achieved optical image quality utilizable for seafloor classification and 3D reconstruction of underwater objects. Keeping the vertical zig-zag motion of underwater gliders in mind, our findings substantiate the feasibility of multi-week seafloor observation missions at long operating ranges.

Gestures form an important medium of communication between humans and machines. An overwhelming majority of existing gesture recognition methods are tailored to a scenario where humans and machines are located very close to each other. This short-distance assumption does not hold true for several types of interactions, for example gesture-based interactions with a floor cleaning robot or with a drone. Methods made for short-distance recognition are unable to perform well on long-distance recognition due to gestures occupying only a small portion of the input data. Their performance is especially worse in resource constrained settings where they are not able to effectively focus their limited compute on the gesturing subject. We propose a novel, accurate and efficient method for the recognition of gestures from longer distances. It uses a dynamic neural network to select features from gesture-containing spatial regions of the input sensor data for further processing. This helps the network focus on features important for gesture recognition while discarding background features early on, thus making it more compute efficient compared to other techniques. We demonstrate the performance of our method on the LD-ConGR long-distance dataset where it outperforms previous state-of-the-art methods on recognition accuracy and compute efficiency.

This paper presents a soft earthworm robot that is capable of both efficient locomotion and obstacle avoidance. The robot is designed to replicate the unique locomotion mechanisms of earthworms, which enable them to move through narrow and complex environments with ease. The robot consists of multiple segments, each with its own set of actuators, that are connected through rigid plastic joints, allowing for increased adaptability and flexibility in navigating different environments. The robot utilizes proprioceptive sensing and control algorithms to detect and avoid obstacles in real-time while maintaining efficient locomotion. The robot uses a pneumatic actuation system to mimic the circumnutation behavior exhibited by plant roots in order to navigate through complex environments. The results demonstrate the capabilities of the robot for navigating through cluttered environments, making this development significant for various fields of robotics, including search and rescue, environmental monitoring, and medical procedures.

This paper proposes a new locomotion algorithm for truss robots on irregular terrain, in particular, for the Variable Topology Truss (VTT) system. The previous Polygon-based Random Tree (PRT) search algorithm for support polygon generation is extended to irregular terrain while considering friction and internal force limitations. By characterizing terrain, unreachable areas are excluded from search to increase efficiency. A one-step rolling motion primitive is generated based on the kinematics, statics, and constraints of VTT. The locomotion planning is completed by transforming and connecting multiple motion primitives with respect to the desired support polygons. The algorithm's performance is verified by conducting simulations in multiple types of environments.

Most 6-DoF localization and SLAM systems use static landmarks but ignore dynamic objects because they cannot be usefully incorporated into a typical pipeline. Where dynamic objects have been incorporated, typical approaches have attempted relatively sophisticated identification and localization of these objects, limiting their robustness or general utility. In this research, we propose a middle ground, demonstrated in the context of autonomous vehicles, using dynamic vehicles to provide limited pose constraint information in a 6-DoF frame-by-frame PnP-RANSAC localization pipeline. We refine initial pose estimates with a motion model and propose a method for calculating the predicted quality of future pose estimates, triggered by whether or not the autonomous vehicle's motion is constrained by the relative frame-to-frame location of dynamic vehicles in the environment. Our approach detects and identifies suitable dynamic vehicles to define these pose constraints to modify a pose filter, resulting in improved recall across a range of localization tolerances from 0.25m to 5m, compared to a state-of-the-art baseline single image PnP method and its vanilla pose filtering. Our constraint detection system is active for approximately 35% of the time on the Ford AV dataset and localization is particularly improved when the constraint detection is active.

The aim of this paper is to study the influence of locality mechanisms in vision transformers. Transformers originated from machine translation and are particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking is a locality mechanism for infor-mation exchange within a local region. In this paper, locality mechanism is systematically investigated by carefully designed controlled experiments. We add locality to vision transformers into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to vision transformers with different architecture designs, which shows the generalization of the locality concept. For ImageNet2012 classification, the locality-enhanced transformers outperform the baselines Swin-T [1], DeiT-T [2] and PVT-T [3] by 1.0%, 2.6 % and 3.1 % with a negligible increase in the number of parameters and computational effort. Code is available at https://github.com/ofsoundof/LocalViT.

Reliable obstacle detection on railways could help prevent collisions that result in injuries and potentially damage or derail the train. Unfortunately, generic object detectors do not have enough classes to account for all possible scenarios, and datasets featuring objects on railways are challenging to obtain. We propose utilizing a shallow network to learn railway segmentation from normal railway images. The limited receptive field of the network prevents overconfident predictions and allows the network to focus on the locally very distinct and repetitive patterns of the railway environment. Additionally, we explore the controlled inclusion of global information by learning to hallucinate obstacle-free images. We evaluate our method on a custom dataset featuring railway images with artificially augmented obstacles. Our proposed method outperforms other learning-based baseline methods.

Game-theoretic motion planners are a powerful tool for the control of interactive multi-agent robot systems. Indeed, contrary to predict-then-plan paradigms, game-theoretic planners do not ignore the interactive nature of the problem, and simultaneously predict the behaviour of other agents while considering change in one's policy. This, however, comes at the expense of computational complexity, especially as the number of agents considered grows. In fact, planning with more than a handful of agents can quickly become intractable, disqualifying game-theoretic planners as possible candidates for large scale planning. In this paper, we propose a planning algorithm enabling the use of game-theoretic planners in robot systems with a large number of agents. Our planner is based on the reality of locality of information and thus deploys local games with a selected subset of agents in a receding horizon fashion to plan collision avoiding trajectories. We propose five different principled schemes for selecting game participants and compare their collision avoidance performance. We observe that the use of Control Barrier Functions for priority ranking is a potent solution to the player selection problem for motion planning.

Heavy quadrupedal drives have great potential for overcoming obstacles, showing great possibilities for transportation industries in complex environments. Ground reaction force (GRF) is a crucial state variable for quadrupedal control. Most GRF observations are implemented in lightweight quadrupeds, with little consideration of the loading being static or slippery on the body. However, the load information is vital to the heavy-duty quadruped applied in transportation tasks. In this paper, we disassembled the whole-body dynamics into the body dynamics combined with the individual floating single-leg dynamics and completed observing the virtual coupling effects between the body and legs. Based on the observed coupling force and centroidal dynamics (CD), the GRF of a stance leg is obtained without the awareness of body weight, movement, and load information. Furthermore, we utilized the body dynamics and the observed virtual force to obtain the body's unknown payload. By reconstructing the moment balance equation, we obtained the payload's position concerning the body coordinate. Compared to conventional quadrupedal GRF observation methods, this framework achieves higher observation accuracy in heavy quadrupeds without load and body information. Additionally, it enables real-time calculation of load magnitude and position.

In this paper, we propose a lip-inspired passive jamming gripper by mimicking teeth structures from a dog's oral structure. Animal lips are hydrostatic structures. To mimic the features, which are usually soft but rigid when contacted, we used the passive particle jamming effect. To increase the adaptability of our previous gripper to grasp various shaped objects, we focused on the dogs' oral structure and holding behaviors. Dogs use spaces inside their mouths to hold sticks firmly when moving. The grasping force of the upgraded gripper was improved by generating teeth structures with the mimicked lip structures. Experiments were conducted to demonstrate the grasping ability of the gripper with the teeth structures by comparing it to other types of grippers with cylindrical and cuboid objects of various dimensions. We also showed that the proposed gripper could hold daily kitchen objects better than the previous version gripper by using closing lip-pouches.

Data-driven visual odometry (VO) is a critical subroutine for autonomous edge robotics, and recent progress in the field has produced highly accurate point predictions in complex environments. However, emerging autonomous edge robotics devices like insect-scale drones and surgical robots lack a computationally efficient framework to estimate VO's predictive uncertainties. Meanwhile, as edge robotics continue to proliferate into mission-critical application spaces, awareness of the model's predictive uncertainties has become crucial for risk-aware decision-making. This paper addresses this challenge by presenting a novel, lightweight, and statistically robust framework that leverages conformal inference (CI) to extract VO's uncertainty bands. Our approach represents the uncertainties using flexible, adaptable, and adjustable prediction intervals that, on average, guarantee the inclusion of the ground truth across all degrees of freedom (DOF) of pose estimation. We discuss the architectures of generative deep neural networks for estimating multivariate uncertainty bands along with point (mean) prediction. We also present techniques to improve the uncertainty estimation accuracy, such as leveraging Monte Carlo dropout (MC-dropout) for data augmentation. Finally, we propose a novel training loss function that combines interval scoring and calibration loss with traditional training metrics-mean-squared error and KL-divergence-to improve uncertainty-aware learning. Our simulation results demonstrate that the presented framework consistently captures true uncertainty in pose estimations across different datasets, estimation models, and applied noise types, indicating its wide applicability.

Semantic scene understanding is beneficial for mobile robots. Semantic information obtained through onboard cameras can improve robots' navigation performance. However, obtaining semantic information on small mobile robots with constrained power and computation resources is challenging. We propose a new lightweight convolution neural network comparable to previous semantic segmentation algorithms for mobile applications. Our network achieved 73.06% on the Cityscapes validation set and 71.8% on the Cityscapes test set. Our model runs at 116 fps with \mathbf{1024\mathrm{x}2048}\mathbf{1024\mathrm{x}2048}, 172 fps with 1024\mathrm{x}10241024\mathrm{x}1024, and 175 fps with 720\mathrm{x}960720\mathrm{x}960 on NVIDIA GTX 1080. We analyze a model size, which is defined as the summation of the number of floating operations and the number of parameters. The smaller model size enables tiny mobile robot systems that should operate multiple tasks simultaneously to work efficiently. Our model has the smallest model size compared to the real-time semantic segmentation convolution neural networks ranked on Cityscapes real-time benchmark and other high performing, lightweight convolution neural networks. On the Camvid test set, our model achieved a mIoU of 73.29% with Cityscapes pre-training, which outperformed the accuracy of other lightweight convolution neural networks. For mobile applicability, we measured frame-per-second on different low-compute devices. Our model operates 35 fps on Jetson Xavier AGX, 21 FPS on Jetson Xavier NX, and 14 FPS on a ROS ASUS gaming phone. 1024\mathrm{x}20481024\mathrm{x}2048 resolution is used for the Jetson devices, and 512\mathrm{x}512512\mathrm{x}512 size is utilized for the measurement on the phone. Our network did not use extra datasets such as ImageNet, Coarse Cityscapes, and Mapillary. Additionally, we did not use TensorRT to achieve fast inference speed. Compared to other real-time and lightweight CNNs, our model achieved significantly more efficiency while balancing accuracy, inference speed, and model size. Show More

Animal behaviour can reflect the health and physiological stage of the animal. Animal behaviour recognition is a vital part of automated farming systems. Although image-based deep learning algorithms can accurately identify animal behaviour, the lack of data on animal abnormal behaviour makes the practical deployment of models of limited significance. At the same time, the ageing of farm monitoring equipment is also a key factor hindering automated farming. This paper constructs a sheep abnormal behaviour dataset ABSB to address these issues and proposes a lightweight real-time multi-sheep abnormal behaviour detection model YOLOv7-Lrab based on the YOLOv7-tiny network. The abnormal behaviour dataset includes four normal behaviours: standing, lying, eating and drinking, and three abnormal behaviours: lameness, attack and death. In the proposed YOLOv7-Lrab model, the small target detection layer, Coordinate attention module, SPD-Conv and Mobileone module are added compared to YOLOv7-tiny. The experimental results show that with a 7:3 ratio of training data to test data, 96.5% recognition accuracy and 95.5% recall can be achieved, and the model size is only 4.5MB with fps of 156. The model is compressed to a minimum without loss of accuracy, providing a new idea for deploying deep learning model in practical application scenarios.

Learning-based path planning is becoming a promising robot navigation methodology due to its adaptability to various environments. However, the expensive computing and storage associated with networks impose significant challenges for their deployment on low-cost robots. Motivated by this practical challenge, we develop a lightweight neural path planning architecture with a dual input network and a hybrid sampler for resource-constrained robotic systems. Our architecture is designed with efficient task feature extraction and fusion modules to translate the given planning instance into a guidance map. The hybrid sampler is then applied to restrict the planning within the prospective regions indicated by the guide map. To enable the network training, we further construct a publicly available dataset with various successful planning instances. Numerical simulations and physical experiments demonstrate that, compared with baseline approaches, our approach has nearly an order of magnitude fewer model size and five times lower computational while achieving promising performance. Besides, our approach can also accelerate the planning convergence process with fewer planning iterations compared to sample-based methods.