IROS 2024 Papers

In the absence of a pre-built indoor map, robot navigation suffers from the limitations of sensors and environments, resulting in decreased efficiency in performing ad-hoc tasks. Given that blueprints are difficult to obtain, an intuitive method is to provide robots with prior knowledge via hand-drawn floor plans. However, due to the inability of robots to directly comprehend hand-drawn styles, the applicability of this method is limited. In this paper, we present a novel framework for hand-drawn floor plan reconstruction that can recognize abstract hand-drawn elements and standardize the reconstruction of hand-drawn floor plans, thereby providing robots with valuable global map information. Specifically, we design a new series of structure descriptors as reconstruction components and employ a deep learning-based model for recognition. Then the standardized results are obtained through the proposed floor plan reconstruction algorithm. To verify the effectiveness of the framework, we conduct experiments on electronic and paper hand-drawn floor plans. Compared with other state-of-the-art methods, our proposed method achieves superior reconstruction results. This work expands the application scenarios for indoor robots, enabling them to quickly comprehend the semantics of complex scenes, thereby enhancing the competitiveness in downstream tasks.

One of the most common cancers among women is breast cancer which can be treated surgically in the early stages with a lumpectomy technique. In the context of breast lumpectomy procedures, accurately tracking tumours presents a critical challenge worsened by various sources of anatomical deformations, including breathing, tissue cutting, and ultrasound probe pressure. To address this, we explore how a realistic tissue deformation simulator can enhance the precision of locating internal targets by accurately assessing the deformation applied to a preoperative model of the breast, considering the distinct mechanical properties of both the breast tissue and the tumour within it. Our method uses advanced artificial intelligence techniques by combining a generative variation autoencoder (GNN-VAE) and an updating method called ensemble smoother with multiple data assimilation (ES-MDA), creating a dynamic model based exclusively on surface node data to update all nodes within the tissue. By leveraging a realistic tissue deformation simulator, our approach uses breast surface tracking to infer full tissue deformations. This makes the method compatible with various simulation tools and suitable for tissues with complex properties. The results indicate that the trained network has an accuracy of 0.014 cm with training data, and 0.026 cm with the testing portion of data, demonstrating precision in tumour localization and significantly improving upon current methods. This innovation can potentially enhance patient outcomes by making breast cancer surgery safer, less invasive, and more efficient.

Addressing safe and efficient navigation in dynamic, realistic, and complex environments stands as a pivotal inquiry within the realm of robotics. Recently, numerous learning-based methods are introduced into the field of navigation, yielding notable outcomes. In this letter, we propose a hierarchical safe reinforcement learning navigation approach (HSRLN) for mapless navigation. It trains mapless navigation policies for non-homogeneous complex scenarios in a hierarchical manner through a kind of three-stage learning, global planning reinforcement learning (RL) + expert imitation learning (IL) + transfer RL (TRL). The innovations of this work are fourfold: a) It effectively reduces the difficulty of training for complex navigation by effectively narrowing the task horizon of RL through a hierarchical framework. b) We designed an imitation learning method based on Relative Driving Safety Index (RDSI) [1] to focus on learning critical expert actions. c) It employs a TRL approach to improve generalization under non-homogeneity assumptions by fine-tuning the policy. d) HSRLN extracts significant features important for navigation decisions from raw observations via velocity obstacle modeling. Experiments indicate that it has performs better than existing hierarchical RL navigation methods (HDRL [2], SRL-ORCA [3]). Relative to SRL-ORCA, it improves navigation success by 12.1% under the non-homogeneity assumption. Videos are available at https://youtu.be/24h9JmcIfMw.

We present a new millimeter-scale bimorph actuator for microrobotic applications, driven by feedforward controlled shape-memory alloy (SMA) wires. The device weighs 10 mg, measures 14 mm in length, and occupies a volume of 4.8 mm3, which makes it the lightest and smallest fully functional SMA-based bimorph actuator for microrobotics developed to date. The experimentally measured operational bandwidth is on the order of 20 Hz, and the unimorph and bimorph maximum low-frequency displacement outputs are on the order of 3.5 and 7 mm, respectively. To test and demonstrate the functionality and suitability of the actuator for microrobotics, we developed the Fish-&-Ribbon–Inspired Small Swimming Harmonic roBot (FRISSHBot). Loosely inspired by carangiformes, the FRISSHBot leverages fluid-structure interaction (FSI) phenomena to propel itself forward, weighs 30 mg, measures 34 mm in length, operates at frequencies of up to 4 Hz, and swims at speeds of up to 3.06 mm • s–1 (0.09 Bl • s–1). This robot is the lightest and smallest swimmer with onboard actuation developed to date.

Autonomous robots, autonomous vehicles, and humans wearing mixed-reality headsets require accurate and reliable tracking services for safety-critical applications in dynamically changing real-world environments. However, the existing tracking approaches, such as Simultaneous Localization and Mapping (SLAM), do not adapt well to environmental changes and boundary conditions despite extensive manual tuning. On the other hand, while deep learning-based approaches can better adapt to environmental changes, they typically demand substantial data for training and often lack flexibility in adapting to new domains. To solve this problem, we propose leveraging the neurosymbolic program synthesis approach to construct adaptable SLAM pipelines that integrate the domain knowledge from traditional SLAM approaches while leveraging data to learn complex relationships. While the approach can synthesize end-to-end SLAM pipelines, we focus on synthesizing the feature extraction module. We first devise a domain-specific language (DSL) that can encapsulate domain knowledge on the essential attributes for feature extraction and the real-world performance of various feature extractors. Our neurosymbolic architecture then undertakes adaptive feature extraction, optimizing parameters via learning while employing symbolic reasoning to select the most suitable feature extractor. Our evaluations demonstrate that our approach, neurosymbolic Feature EXtraction (nFEX), yields higher-quality features. It also reduces the pose error observed for the state-of-the-art baseline feature extractors ORB and SIFT by up to 90% and up to 66%, respectively, thereby enhancing the system’s efficiency and adaptability to novel environments.

This work presents a comprehensive LiDAR-inertial odometry framework featuring robust smoothing and mapping capabilities, effectively correcting LiDAR feature point skewness using an inertial measurement unit (IMU). While the Extended Kalman Filter (EKF) is a common choice for nonlinear motion estimation, its complexity grows when handling maneuvering targets. To overcome this challenge, a new framework that incorporates the Iterated Interactive Multiple Models of Kalman Filter (IMMKF) is given, providing a solution for reliable navigation in dynamic motion and noisy conditions. To ensure map consistency, an ikd-tree that facilitates continuous updates and adaptive rebalance is employed, preserving the map’s integrity. To guarantee the robustness of our approach, it undergoes extensive testing across diverse scales of indoor and outdoor environments. This testing scenario simulates absolute GPS denial. In terms of estimated motion, the new algorithm demonstrates superior accuracy compared to existing approaches. The implementation is openly accessible on GitHub4 for further exploration.

This paper introduces a new formulation that finds the optimum for the Moving-Target Traveling Salesman Problem (MT-TSP), which seeks to find a shortest path for an agent, that starts at a depot, visits a set of moving targets exactly once within their assigned time-windows, and returns to the depot. The formulation relies on the key idea that when the targets move along lines, their trajectories become convex sets within the space-time coordinate system. The problem then reduces to finding the shortest path within a graph of convex sets, subject to some speed constraints. We compare our formulation with the current state-of-the-art Mixed Integer Conic Program (MICP) formulation for the MT-TSP. The experimental results show that our formulation outperforms the MICP for instances with up to 20 targets, with up to two orders of magnitude reduction in runtime, and up to a 60% tighter optimality gap. We also show that the solution cost from the convex relaxation of our formulation provides significantly tighter lower-bounds for the MT-TSP than the ones from the MICP.

In low-texture scenes, Visual Odometry (VO) algorithms often encounter challenges stemming from sparse feature sets and reduced accuracy in feature matching. To overcome this, integrating plane features and vanishing point characteristics can provide additional constraints for refining camera poses. Optical flow-based tracking methods may also offer improved matching precision compared to traditional feature-based approaches. Motivated by these challenges, we present a robust Visual Odometry system tailored for low-texture environments. Our system combines a vanishing point-based approach for camera pose optimization with a Manhattan-aided algorithm for matching line segments using optical flow. By incorporating planes and vanishing points as supplementary features for pose estimation, we enhance overall accuracy without significant time overhead. We utilize detected line features to compute vanishing points, improving accuracy without compromising efficiency. In addition, our Manhattan-aided optical flow technique supplements and refines the results of line feature matching, further enhancing the accuracy of vanishing points. Evaluation on various public datasets demonstrates the superior accuracy and robustness of our system compared to state-of-the-art Simultaneous Localization And Mapping (SLAM) and VO methods. Notably, our method effectively addresses issues of failure in low-texture scenes and improves the accuracy of line feature matching compared to baseline methods. We will release our source code upon paper acceptance.

Bin picking systems in factory automation usually face robustness issues caused by sparse and noisy 3D data of metallic objects. Utilizing multiple views, especially with a one-shot 3D sensor and "sensor on hand" configuration is getting more popularity due to its effectiveness, flexibility, and low cost. While moving the 3D sensor to acquire multiple views for 3D fusion, joint optimization, or active vision suffers from low-speed issues. That is because sensing is taken as a decoupled module from motion tasks and is not intentionally designed for a bin picking system. To address the problems, we designed a bin picking system, which tightly couples a multi-view, active vision scheme with motion tasks in a "sensor on hand" configuration. It not only speeds up the system by parallelizing the high-speed sensing scheme to the robot place action but also decides the next sensing path to maintain the continuity of the whole picking process. Unlike others focusing only on sensing evaluation, we also evaluated our design by picking experiments on 5 different types of objects without human intervention. Our experiments show the whole sensing scheme can be finished within 1.682 seconds (maximum) on CPU and the average picking complete rate is over 97.75%. Due to the parallelization with robot motion, the sensing scheme accounts for only 0.635 seconds in takt time on average.

Image captioning is a multi-modal task that enables the transformation from scene images to natural language, providing valuable insights for visually impaired individuals to understand their environment. Therefore, its application to wearable navigation devices for visually impaired individuals holds immense potential. However, in practical applications, confusion between scene visuals and semantics, coupled with model complexity, often leads to performance degradation, resulting in inaccurate environmental interpretation. In light of this, we introduce a Lightweight De-confounding Transformer Network (LDTNet) for image captioning equipped with a Causal Adjustment module to eliminate confounders. Moreover, we design a Suppression Gate Unit that efficiently integrates fine-grained information from shallow features, while reducing the number of network layers to have a lightweight model. Experimental results demonstrate that our approach not only addresses the visual-semantic confusion issue effectively but also improves the response speed of wearable devices in comparison with the state of the art. Twenty volunteers are recruited to evaluate LDTNet’s efficacy in real-world settings in terms of both response speed and generated outputs by wearing the resulting assistive navigation devices. The outcomes well show its outstanding performance and great potential for visualy impaired individuals to use.

Existing grasp controllers usually either only support finger-tip grasps or need explicit configuration of the inner forces. We propose a novel grasp controller that supports arbitrary grasp types, including power grasps with multi-contacts, while operating self-contained on before unseen objects. No detailed contact information is needed, but only a rough 3D model, e.g., reconstructed from a single depth image. First, the external wrench being applied to the object is estimated by using the measured torques at the joints. Then, the torques necessary to counteract the estimated wrench while keeping the object at its initial pose are predicted. The torques are commanded via desired joint angles to an underlying joint-level impedance controller. To reach real-time performance, we propose a learning-based approach that is based on a wrench estimator- and a torque predictor neural network. Both networks are trained in a supervised fashion using data generated via the analytical formulation of the controller. In an extensive simulation-based evaluation, we show that our controller is able to keep 83.1% of the tested grasps stable when applying external wrenches with up to 10 N. At the same time, we outperform the two tested baselines by being more efficient and inducing less involuntary object movement. Finally, we show that the controller also works on the real DLR-Hand II, reaching a cycle time of 6 ms. Website: aidx-lab.org/grasping

Humans can perceive and interact with their surroundings through multiple senses. For intelligent robots, multimodal sensors are crucial for them to perceive and understand the environment. In this work, we propose a multi-layered flexible multimodal sensor based on laser-induced graphene, capable of detecting both touchless signals (such as the distance from external objects and their material) and tactile signals (three-dimensional force). The sensor has advantages in durability and stability. Under normal force, the sensitivity is –7.449% N–1 in the range of 0 N to 1.5 N and –0.273% N–1 in the range of 1.5 N to 30 N, with fast response (17 ms) and recovery (37 ms). Furthermore, using the Convolutional Neural Networks (CNN) model, we develop an intelligent soft robot system capable of distinguishing objects of different materials and fabric textures with accuracies of 99% and 88.75%, respectively. The proposed flexible multimodal sensor holds a significant effect on the perception and interaction of intelligent robots with the environment.

It is a challenging task for visually impaired people to perceive their surrounding environment due to the complexity of the natural scenes. Their personal and social activities are thus highly limited. This paper introduces a Large Vision-Language Model(LVLM) based environment perception system which helps them to better understand the surrounding environment, by capturing the current scene they face with a wearable device, and then letting them retrieve the analysis results through the device. The visually impaired people could acquire a global description of the scene by long pressing the screen to activate the LVLM output, retrieve the categories of the objects in the scene resulting from a segmentation model by tapping or swiping the screen, and get a detailed description of the objects they are interested in by double-tapping the screen. To help visually impaired people more accurately perceive the world, this paper proposes incorporating the segmentation result of the RGB image as external knowledge into the input of LVLM to reduce the LVLM’s hallucination. Technical experiments on POPE, MME and LLaVA-QA90 show that the system could provide a more accurate description of the scene compared to Qwen-VL-Chat, exploratory experiments show that the system helps visually impaired people to perceive the surrounding environment effectively.

Accurate perception of semantic and spatial information is crucial for robots performing language-driven navigation tasks. Existing approaches utilize visual-language models to extract semantic information from the environment and construct maps. However, constrained by the generalization and accuracy of these models themselves, the constructed maps may not be accurate and comprehensive, thereby affecting the accuracy of navigation tasks. Inspired by foundational models’ outstanding classification and segmentation capabilities, this study introduces a semantic map constructed using foundational models. We leverage a foundational model to semantically segment objects in the robot’s video stream and fuse semantics onto the map. Furthermore, this map is used in conjunction with large language models (LLMs) that receive natural language instructions to complete the navigation task. A substantial number of experiments in a simulated environment demonstrate that our method outperforms existing ones in language-driven navigation tasks.

We introduce a novel hybrid vision/force control strategy for robotic devices designed to obtain clear and consistent images using probe-based confocal laser endomicroscopy (pCLE). Due to the variable nature of tissue characteristics encountered during pCLE imaging, the conventional approach of pre-setting forces or focus metrics for either force or vision control is often impractical and inadequate. To address this, our strategy employs a blur metric called the CR score to assess the level of blur in pCLE images, enabling the attainment of clear and focused images. At the onset of a pCLE scan, the system autonomously determines the target CR score for vision control, in tandem with a real-time peak detection algorithm. Concurrently, force control is applied judiciously to prevent excessive force on the tissue, adjusting to ensure minimal force is applied, thus preserving image focus. This innovative approach facilitates seamless transitions between vision and force control, depending on the imaging conditions, thereby ensuring the acquisition of consistent pCLE images with minimal force. Our method marks a notable improvement over conventional PID force control techniques. By dynamically adjusting target forces and minimizing force application during operation, we not only enhance the precision and quality of pCLE imaging but also eliminate the dependency on manual pre-settings.

Haptic feedback to the surgeon during robotic surgery would enable safer and more immersive surgeries but estimating tissue interaction forces at the tips of robotically controlled surgical instruments has proven challenging. Few existing surgical robots can measure interaction forces directly and the additional sensor may limit the life of instruments. We present a hybrid model and learning-based framework for force estimation for the Patient Side Manipulators (PSM) of a da Vinci Research Kit (dVRK). The model-based component identifies the dynamic parameters of the robot and estimates free-space joint torque, while the learning-based component compensates for environmental factors, such as the additional torque caused by trocar interaction between the PSM instrument and the patient’s body wall. We evaluate our method in an abdominal phantom and achieve an error in force estimation of under 10% normalized root-mean-squared error. We show that by using a model-based method to perform dynamics identification, we reduce reliance on the training data covering the entire workspace. Although originally developed for the dVRK, the proposed method is a generalizable framework for other compliant surgical robots. The code is available at https://github.com/vu-maple-lab/dvrk_force_estimation.

The ability to follow people can benefit the human-robot interaction of mobile robots. This work proposes a hybrid human tracking system for human following robots, integrating sensor fusion of Ultra-Wideband (UWB) and monocular visual positioning to enhance tracking accuracy and precision. At the same time, UWB and the visual positioning system can operate independently, thereby creating a redundancy in the system. Based on our previous study of UWB-positioning, this article elaborates on a visual positioning system that employs human detection using a pre-trained Convolutional Neural Network (CNN), coupled with data fusion process based on experimental assessments. The hybrid human tracking system achieves a 2D Euclidean accuracy RMS of 7.4 cm, demonstrating sufficient accuracy for human following and improving the following performance in real-world experiments compared to our previous study.

Anthropomorphic robotic arms, mimicking the structure and function of human arms, show great potential for helping people in various tedious and repetitive household tasks. However, such arms mostly consist of multiple serial links controlled independently by actuators at joints with high reduction ratios, posing challenges in household services in terms of load capacity, responsiveness, and safety. In this paper, we propose a high-performance anthropomorphic arm called TRX-Arm based on differential cable transmission, characterized by features of high dynamics, high load capacity, and inherent compliance. TRX-Arm is composed of three deferential cable-driven coupling joints and one independent roll joint. Thanks to the cable differential transmission, the joints are capable of achieving doubled torque and stiffness without replacing motors. To enhance safety in human-robot interaction, the actuators including motors, reducer, belt, and pulley are mounted at the shoulder near the base and drive the joints remotely using cables, thereby minimizing the inertia of the whole arm. The workspace of TRX-Arm has a volume of 1.56 m3, much larger than that of the human arm. Real experiments show its capabilities including high repeatability and load capacity as well as high dynamic behavior of a dual-arm robot platform built with TRX-Arms.

The efficient operation of large-scale Cable-Driven Parallel Robots (CDPRs) relies on precise calibration of kinematic parameters and the simplicity of the calibration process. This paper presents a graph-based self-calibration framework that explicitly addresses cable sag effects and facilitates the calibration procedure for large-scale CDPRs by only relying on internal sensors. A unified factor graph is proposed, incorporating a catenary cable model to capture cable sagging. The factor graph iteratively refines kinematic parameters, including anchor point locations and initial cable length, by considering jointly onboard sensor data and the robot’s kineto-static model. The applicability and accuracy of the proposed technique are demonstrated through Finite Element (FE) simulations, on both large and small-scale CDPRs subjected to significant initialization perturbations.