IROS 2024 Papers

Monocular visual odometry (MVO) is vital in autonomous navigation and robotics, providing a cost-effective and flexible motion tracking solution, but the inherent scale ambiguity in monocular setups often leads to cumulative errors over time. In this paper, we present BEV-ODOM, a novel MVO framework leveraging the Bird’s Eye View (BEV) Representation to address scale drift. Unlike existing approaches, BEV-ODOM integrates a depth-based perspective-view (PV) to BEV encoder, a correlation feature extraction neck, and a CNN-MLP-based decoder, enabling it to estimate motion across three degrees of freedom without the need for depth supervision or complex optimization techniques. Our framework reduces scale drift in long-term sequences and achieves accurate motion estimation across various datasets, including NCLT, Oxford, and KITTI. The results indicate that BEV-ODOM outperforms current MVO methods, demonstrating reduced scale drift and higher accuracy.

Cross-view image matching for geo-localisation is a challenging problem due to the significant visual difference between aerial and ground-level viewpoints. The method provides localisation capabilities from geo-referenced images, eliminating the need for external devices or costly equipment. This enhances the capacity of agents to autonomously determine their position, navigate, and operate effectively in GNSS-denied environments. Current research employs a variety of techniques to reduce the domain gap such as applying polar transforms to aerial images or synthesising between perspectives. However, these approaches generally rely on having a 360° field of view, limiting real-world feasibility. We propose BEV-CV, an approach introducing two key novelties with a focus on improving the real-world viability of cross-view geo-localisation. Firstly bringing ground-level images into a semantic Birds-Eye-View before matching embeddings, allowing for direct comparison with aerial image representations. Secondly, we adapt datasets into application realistic format - limited Field-of-View images aligned to vehicle direction. BEV-CV achieves state-of-the-art recall accuracies, improving Top-1 rates of 70° crops of CVUSA and CVACT by 23% and 24% respectively. Also decreasing computational requirements by reducing floating point operations to below previous works, and decreasing embedding dimensionality by 33% - together allowing for faster localisation capabilities.

In this paper, we present a novel study on a BEV (bird’s eye view) image-based lane tracking control system for an autonomous lane repainting robot. Our research introduces a cutting-edge lane detection method based on BEV images, leveraging row-anchor techniques to enhance precision and provide detailed error information for lane tracking algorithms. By utilizing real-time sensor data and advanced deep learning processes, we have successfully implemented a high-performance lane repainting system that minimizes errors and ensures accuracy. Our proposed position-based visual pure pursuit algorithm (PV-PP) plays a crucial role in guiding the lane repainting process with precision and efficiency, ultimately improving the functionality and feasibility of the linear actuator responsible for paint spraying in the real indusrial fields. Through our contributions, including innovative lane detection methods, real-time sensor utilization, and robot control algorithm design, we aim to advance the field of autonomous lane repainting robots and enhance the safety and effectiveness of road maintenance operations.

The quality of a robot’s environmental perception determines whether it can achieve more intelligent applications, such as semantic interaction with humans. SLAM, on the other hand, is one of the crucial capabilities for a robot to perceive its environment. However, when only a monocular image is provided, dynamic objects in the environment significantly impact the accuracy of map construction by the robot, leading to erroneous perception results. To address this issue, we propose a Visual SLAM framework based on BEV perception results, named BE-SLAM. With this framework, we can handle dynamic objects, occlusions, and incompletely observed objects. It can construct a stable static map by strengthening trust in static objects. Considering that object-level semantic maps can enhance a robot’s perception abilities, we also reconstruct static objects in the map and use them to optimize the pose. Through experiments on existing publicly available dataset, we compare BE-SLAM with several existing methods that have shown good performance. The experimental results demonstrate that BE-SLAM performs exceptionally well on high-dynamic sequences and achieves comparable results on static or low-dynamic sequences.

Out-of-distribution (OoD) detection techniques for deep neural networks (DNNs) become crucial thanks to their filtering of abnormal inputs, especially when DNNs are used in safety-critical applications and interact with an open and dynamic environment. Nevertheless, integrating OoD detection into state-of-the-art (SOTA) object detection DNNs poses significant challenges, partly due to the complexity introduced by the SOTA OoD construction methods, which require the modification of DNN architecture and the introduction of complex loss functions. This paper proposes a simple, yet surprisingly effective, method that requires neither retraining nor architectural change in object detection DNN, called Box Abstraction-based Monitors (BAM). The novelty of BAM stems from using a finite union of convex box abstractions to capture the learned features of objects for in-distribution (ID) data, and an important observation that features from OoD data are more likely to fall outside of these boxes. The union of convex regions within the feature space allows the formation of non-convex and interpretable decision boundaries, overcoming the limitations of VOS-like detectors without sacrificing real-time performance. Experiments integrating BAM into Faster R-CNN-based object detection DNNs demonstrate a considerably improved performance against SOTA OoD detection techniques, with a reduction in the false detection rate of over 10% in most cases.

The large-scale deployment of robotic manipulation systems in warehouses has highlighted the rare but costly problem of robot-induced object damage. We present a system that uses a classification model to predict whether an object will get damaged during robotic manipulation. The model uses object attributes retrieved from warehouse information systems as well as attributes available at our robotic workcell. We evaluated different classical machine learning models, as well as a large language model (BERT) and a multimodal-transformer for our task. We show that the multi-modal transformer model that is able to leverage text and image data outperforms models that only rely on categorical and numerical data. Furthermore, our comparative analysis equips the selection the optimal model for an application. We validate our system during an experiment in which the output of the damage prediction system is used to avoid picking objects that are likely to get damaged. In over 50k pick-and-place activities, our system reduces damage rate by 64%.

A novel method for autonomous localization of multiple sources of gamma radiation using a group of Micro Aerial Vehicles (MAVs) is presented in this paper. The method utilizes an extremely lightweight (44 g) Compton camera MiniPIX TPX3. The compact size of the detector allows for deployment onboard safe and agile small-scale Unmanned Aerial Vehicles (UAVs). The proposed radiation mapping approach fuses measurements from multiple distributed Compton camera sensors to accurately estimate the positions of multiple radioactive sources in real time. Unlike commonly used intensity-based detectors, the Compton camera reconstructs the set of possible directions towards a radiation source from just a single ionizing particle. Therefore, the proposed approach can localize radiation sources without having to estimate the gradient of a radiation field or contour lines, which require longer measurements. The instant estimation is able to fully exploit the potential of highly mobile MAVs. The radiation mapping method is combined with an active search strategy, which coordinates the future actions of the MAVs in order to improve the quality of the estimate of the sources’ positions, as well as to explore the area of interest faster. The proposed solution is evaluated in simulation and real-world experiments with multiple Cesium-137 radiation sources.

Social robots are gradually integrating into human’s daily lives. Storytelling by social robots could bring a different experience to users through non-verbal and emotional capabilities compared to text-only one. However, as user needs and preferences over storytelling might change over time during long-term interaction with social robots, it is important for social robots to learn from social interactions with human users in real-time. In this paper, we propose to allow our social robot Haru to learn personalized storytelling styles for different human user’s emotional states via human-centered reinforcement learning using the reward provided and delivered by directly interaction with the user explicitly. Results of our user study show that Haru can learn to adapt its storytelling style for detected human emotional states in a few number of interactions, and was perceived to have a better storytelling performance, experience and impact than a neutral one.

This work proposes a novel drone system designed to autonomously track and follow power lines and reconstruct them in 3D with a point cloud representation based on mmWave radar measurements. The system is composed of a GNSS-enabled quadrotor UAV equipped with a combined mmWave radar sensor and onboard compute module payload and has been designed to be small, lightweight, and low-cost. MmWave radar sensors offer great range and sensitivity in the task of power line detection with a high level of sparsity in the produced data when compared to traditional sensors such as LiDARs. The proposed system overcomes the radar sensor’s shortcomings by building up a point cloud representing the power line environment as the drone moves around in it. The built-up point cloud is analyzed using the onboard computer to detect the cables in the power line environment and to produce pose-estimates of each line. The system has been tested in a variety of scenarios and has been shown to be able to accurately detect and track power lines in varying weather conditions.

Cardiovascular disease treatment involves the challenging task of navigating guidewires and catheters through the vascular anatomy. This often results in prolonged procedures where both the patient and clinician are subjected to X-ray radiation. As a potential solution, Deep Reinforcement Learning methods have demonstrated potential in learning this task, paving the way for automated catheter navigation during robotic interventions. However, current works show a limited ability to generalize to unseen and/or deforming anatomies.In this paper, we extend our previous reinforcement learning approach in two main areas: we improve the training strategy to learn a control of the device even when the vascular anatomy is deforming and we propose a method to estimate the motion of the anatomy from single view fluoroscopy images. The combination of these two contributions makes it possible to automatically navigate across a moving vascular anatomy under fluoroscopic imaging, even without injecting a contrast agent. We validate our method on two scenarios: a simulated beating heart and a liver subjected to breathing motion. Our approach leads to an average success rate of 95% in reaching random targets within these anatomies. Our framework is also computationally efficient, enabling the training of our controller to be completed in about 6 hours.

Enabling humanoid robots to perform autonomously loco-manipulation in unstructured environments is crucial and highly challenging for achieving embodied intelligence. This involves robots being able to plan their actions and behaviors in long-horizon tasks while using multi-modality to perceive deviations between task execution and high-level planning. Recently, large language models (LLMs) have demonstrated powerful planning and reasoning capabilities for comprehension and processing of semantic information through robot control tasks, as well as the usability of analytical judgment and decision-making for multi-modal inputs. To leverage the power of LLMs towards humanoid loco-manipulation, we propose a novel language-model based framework that enables robots to autonomously plan behaviors and low-level execution under given textual instructions, while observing and correcting failures that may occur during task execution. To systematically evaluate this framework in grounding LLMs, we created the robot ’action’ and ’sensing’ behavior library for task planning, and conducted mobile manipulation tasks and experiments in both simulated and real environments using the CENTAURO robot, and verified the effectiveness and application of this approach in robotic tasks with autonomous behavioral planning. Video: https://youtu.be/mmnaxthEX34

Reproductive medicine has recently witnessed significant advancements, particularly in vitro fertilization (IVF). One crucial aspect of IVF involves the extraction of cellular material and its analysis to maximize the chance of successful implantation. This work highlights the development and application of the automated system for Trophectoderm cell (TE) extraction and separation, addressing the need for precision, efficiency, and reduced manual intervention. The presented automated system is equipped with a computer vision algorithm, microliter pump, vacuum system, and micromanipulation tools to consistently and accurately biopsy TE cells. An experimental setup is developed to verify the behavior of the proposed method, in which a holding micropipette is connected to a vacuum system and holds the embryo stationary. Three steps are performed to complete the process and are controlled by a computer vision algorithm. The coordinates of the Zona Pellucida (ZP) perforation (perforated in a previous step) are used as a feedback signal to a simple proportional controller to control the biopsy pipette motion. The computer vision monitors the amount of TE cells aspirated inside the biopsy pipette and controls the microliter pump. The aspirated TE cells were separated away using a laser cutting system. Experimental results demonstrate that the system can relocate the biopsy pipette, TE cell extraction, and separation.

Cybersecurity in mission-critical robotic applications is a necessity to scale deployments securely. ROS2 builds upon DDS-Security specs in ROS Client Library (RCL) to implement its security features. Utilizing SROS2, developers have access to a set of utilities to help set up security in a way RCL can use. Through SROS2, security deployment is eased for developers. However, while access control is handled by DDS and consequently based on the SROS2-generated permission artifacts, the necessary authorization policies are manually generated by developers. This requires an entire system exercise to be sampled via live extraction and, per each node, list all the necessary Topics, Services, and Actions, which is a daunting and laborious process. Developers first have to generate tests. Then, they obtain a ’snapshot’ of the system for each test. Later, these snapshots must be collected and grouped into a policy by a minimum set of rules. All this procedure is quite error-prone. This paper introduces LiSA4ROS2, a tool for automatically extract the ROS2 computational graph via static analysis to derive a minimal correct configuration for ROS2 security policies. Our approach relies on the abstract interpretation theory to statically overapproximate all possible executions to extract a minimal and complete configuration per node. We evaluate our approach with minimal examples covering all the main communication patterns in ROS2 tutorials and all publicly available real-world ROS2 Python systems extracted from GitHub. The results of the minimal examples show that LiSA4ROS2 precisely supports all the main communication patterns. The extensive evaluation underlines that our prototype implementation of the analysis in LiSA4ROS2 is already able to precisely analyze 66% of existing repositories, automatically producing detailed computational graphs and access policies. All the results of the analysis, as well as a Docker artifact to reproduce them, are publicly available. Show More

We investigate the automatic design of robot swarms that perform composite missions—that is, missions specified as the composition of consecutive sub-missions. Automatic design through performance optimization has become a viable and appealing approach to designing robot swarms. First, a user defines a mission by specifying a performance measure: a function indicating to what extent the swarm has attained its goal. An optimization process then generates suitable control software for the robots by maximizing the performance measure. The definition of a performance measure is a challenging task that requires expert input, which hinders the automatic nature of the approach. Recently, inverse reinforcement learning was introduced to minimize the need for human intervention in the automatic design of robot swarms. However, this method was only applied to single-objective missions. In this paper, we extend the method to address composite missions, by formulating and solving the design problem as a multi-objective optimization problem. We conduct simulations with a swarm of twenty e-puck robots that perform twelve composite missions. We compare the performance of the swarm when the robots operate with control software produced manually or using inverse reinforcement learning.

Despite an emerging interest in MIMO radar, the utilization of its complementary strengths in combination with optical depth sensors has so far been limited to far-field applications, due to the challenges that arise from mutual sensor calibration in the near field. In fact, most related approaches in the autonomous industry propose target-based calibration methods using corner reflectors that have proven to be unsuitable for the near field. In contrast, we propose a novel, joint calibration approach for optical RGB-D sensors and MIMO radars that is designed to operate in the radar’s near-field range, within decimeters from the sensors. Our pipeline consists of a bespoke calibration target, allowing for automatic target detection and localization, followed by the spatial calibration of the two sensor coordinate systems through target registration. We validate our approach using two different depth sensing technologies from the optical domain. The experiments show the efficiency and accuracy of our calibration for various target displacements, as well as its robustness of our localization in terms of signal ambiguities.

Detecting road features is a key enabler for autonomous driving and localization. For instance, a reliable detection of poles which are widespread in road environments can improve localization. Modern deep learning-based perception systems need a significant amount of annotated data. Automatic annotation avoids time-consuming and costly manual annotation. Because automatic methods are prone to errors, managing annotation uncertainty is crucial to ensure a proper learning process. Fusing multiple annotation sources on the same dataset can be an efficient way to reduce the errors. This not only improves the quality of annotations, but also improves the learning of perception models. In this paper, we consider the fusion of three automatic annotation methods in images: feature projection from a high accuracy vector map combined with a lidar, image segmentation and lidar segmentation. Our experimental results demonstrate the significant benefits of multi-modal automatic annotation for pole detection through a comparative evaluation on manually annotated images. Finally, the resulting multi-modal fusion is used to fine-tune an object detection model for pole base detection using unlabeled data, showing overall improvements achieved by enhancing network specialization. The dataset is publicly available.

Automatic laparoscopic field of view (FOV) adjustment can effectively assist surgeons in minimally invasive surgery (MIS). However, existing work based on rod-shaped laparoscopes is inevitably constrained by the remote center of motion (RCM) during the process of FOV adjustment. The RCM limits laparoscopic movement and makes modeling and control more complex. This paper proposes a novel tendon-driven continuum laparoscope that is not affected by the RCM constraint. Furthermore, an automatic FOV adjustment method is designed for the proposed laparoscope robot, which considers the surgical instrument position and size in the image, as well as eye-hand consistency. Two simulation platforms are developed using MATLAB and Webots to intuitively study and optimize the proposed adjustment method. The convergence time of surgical tool tracking with a complex 3D trajectory is only 1s, the average tracking error after stabilization is about 9.97 pixels, and the maximum eye-hand error is only 0.04°. A first-generation prototype is built to verify the tracking performance of the proposed tendon-driven continuum laparoscope. The experimental results show that the proposed system can perform real-time laparoscopic FOV adjustment without being constrained by the RCM.

This paper investigates the problem of eating activity detection using motion data from an off-the-shelf smartwatch. The development and integration of the algorithm for detecting eating activity will make it easier for users to monitor their eating habits. For development of a such algorithm about 27500 hours of data were collected from 91 participants. Moreover, a reliable and interpreted approach with adjustable tolerance for model quality estimation in real-world conditions is proposed in this work. The algorithm based on end-to-end neural network (NN) for eating events detection with special postprocessing was developed by our research group. It recognizes eating events with 1 minute delay from the beginning of food intake. For a such tolerance it achieves F1-score of 0.90 in average (at "free-living" scenario test) for users wearing smartwatches either on dominant or on non-dominant hand. To the best of authors’ knowledge, the algorithm provides the best performance of any existing solution or described in the literature.

With the growing use of machine learning algorithms and ubiquitous sensors, many ‘perception-to-control’ systems are being developed and deployed. To ensure their trustworthiness, improving their robustness through adversarial training is one potential approach. We propose a gradient-free adversarial training technique, named AutoJoin, to effectively and efficiently produce robust models for image-based maneuvering. Compared to other state-of-the-art methods with testing on over 5M images, AutoJoin achieves significant performance increases up to the 40% range against perturbations while improving on clean performance up to 300%. AutoJoin is also highly efficient, saving up to 86% time per training epoch and 90% training data over other state-of-the-art techniques. The core idea of AutoJoin is to use a decoder attachment to the original regression model creating a denoising autoencoder within the architecture. This architecture allows the tasks ‘maneuvering’ and ‘denoising sensor input’ to be jointly learnt and reinforce each other’s performance.