IROS 2023 Papers

This work studies how force measurement/estimation biases affect the force-based cooperative manipulation of a beam-like load suspended with cables by two aerial robots. Indeed, force biases are especially relevant in a force-based manipulation scenario in which direct communication is not relied upon. First, we compute the equilibrium configurations of the system. Then, we show that inducing an internal force in the load augments the robustness of the load attitude error and its sensitivity to force-bias variations. Eventually, we propose a method for zeroing the load position error. The results are validated through numerical simulations and experiments.

When humans see a scene, they can roughly imagine the forces applied to objects based on their expe-rience and use them to handle the objects properly. This paper considers transferring this “force-visualization” ability to robots. We hypothesize that a rough force distribution (named “force map”) can be utilized for object manipulation strategies even if accurate force estimation is impossible. Based on this hypothesis, we propose a training method to predict the force map from vision. To investigate this hypothesis, we generated scenes where objects were stacked in bulk through simulation and trained a model to predict the contact force from a single image. We further applied domain randomization to make the trained model function on real images. The experimental results showed that the model trained using only synthetic images could predict approximate patterns representing the contact areas of the objects even for real images. Then, we designed a simple algorithm to plan a lifting direction using the predicted force distribution. We confirmed that using the predicted force distribution contributes to finding natural lifting directions for typical real-world scenes. Furthermore, the evaluation through simulations showed that the disturbance caused to surrounding objects was reduced by 26 % (translation displacement) and by 39 % (angular displacement) for scenes where objects were overlapping.

The Robot Operating System (ROS2) is the most widely used software platform for building robotics applications. FogROS2 extends ROS2 to allow robots to access cloud computing on demand. We introduce FogROS2-SGC, an extension of FogROS2 that can effectively connect robot systems across different physical locations, networks, and Data Distribution Services (DDS). With globally unique and location-independent identifiers, FogROS2-SGC can securely and efficiently route data between robotics components around the globe. FogROS2-SGC is agnostic to the ROS2 distribution and configuration, is compatible with non-ROS2 software, and seamlessly extends existing ROS2 applications without any code modification. We evaluate FogROS2-SGC with 4 robots and compute nodes that are 3600 km apart. Experiments suggest FogROS2-SGC is 19x faster than rosbridge (a ROS2 package with comparable features, but lacking security). Videos and code are available on the website https://sites.google.com/view/fogros2-sgc.

The future of 3D printing utilizing unmanned aerial vehicles (UAVs) presents a promising capability to revolutionize manufacturing and to enable the creation of large-scale structures in remote and hard-to-reach areas e.g. in other planetary systems. Nevertheless, the limited payload capacity of UAVs and the complexity in the 3D printing of large objects pose significant challenges. In this article we propose a novel chunk-based framework for distributed 3D printing using UAVs that sets the basis for a fully collaborative aerial 3D printing of challenging structures. The presented framework, through a novel proposed optimisation process, is able to divide the 3D model to be printed into small, manageable chunks and to assign them to a UAV for partial printing of the assigned chunk, in a fully autonomous approach. Thus, we establish the algorithms for chunk division, allocation, and printing, and we also introduce a novel algorithm that efficiently partitions the mesh into planar chunks, while accounting for the inter-connectivity constraints of the chunks. The efficiency of the proposed framework is demonstrated through multiple physics based simulations in Gazebo, where a CAD construction mesh is printed via multiple UAVs carrying materials whose volume is proportionate to a fraction of the total mesh volume.

In recent years, there has been a significant effort dedicated to developing efficient, robust, and general human-to-robot handover systems. However, the area of flexible handover in the context of complex and continuous objects' motion remains relatively unexplored. In this work, we propose an approach for effective and robust flexible handover, which enables the robot to grasp moving objects with flexible motion trajectories with a high success rate. The key innovation of our approach is the generation of real-time robust grasp trajec-tories. We also design a future grasp prediction algorithm to enhance the system's adaptability to dynamic handover scenes. We conduct one-motion handover experiments and motion-continuous handover experiments on our novel benchmark that includes 31 diverse household objects. The system we have developed allows users to move and rotate objects in their hands within a relatively large range. The success rate of the robot grasping such moving objects is 78.15 % over the entire household object benchmark.

This paper presents a vision-guided two-stage approach with force feedback to achieve high-precision and flexible gear assembly. The proposed approach integrates YOLO to coarsely localize the target workpiece in a searching phase and deep reinforcement learning (DRL) to complete the insertion. Specifically, DRL addresses the challenge of partial visibility when the on-wrist camera is too close to the workpiece of a small size. Moreover, we use force feedback to improve the robustness of the vision-guided assembly process. To reduce the effort of collecting training data on real robots, we use synthetic RGB images for training YOLO and construct an offline interaction environment leveraging sampled real-world data for training DRL agents. The proposed approach was evaluated in an industrial gear assembly experiment, which requires an assembly clearance of 0.3 mm, demonstrating high robustness and efficiency in gear searching and insertion from arbitrary positions.

Skill transfer from humans to robots is challenging. Presently, many researchers focus on capturing only position or joint angle data from humans to teach the robots. Even though this approach has yielded impressive results for grasping applications, reconstructing motion for object handling or fine manipulation from a human hand to a robot hand has been sparsely explored. Humans use tactile feedback to adjust their motion to various objects, but capturing and reproducing the applied forces is an open research question. In this paper we introduce a wearable fingertip tactile sensor, which captures the distributed 3-axis force vectors on the fingertip. The fingertip tactile sensor is interchangeable between the human hand and the robot hand, meaning that it can also be assembled to fit on a robot hand such as the Allegro hand. This paper presents the structural aspects of the sensor as well as the methodology and approach used to design, manufacture, and calibrate the sensor. The sensor is able to measure forces accurately with a mean absolute error of 0.21, 0.16, and 0.44 Newtons in X, Y, and Z directions, respectively.

There has been increasing awareness of the difficulties in reaching and extracting people from mass casualty scenarios, such as those arising from natural disasters. While platforms have been designed to consider reaching casualties and even carrying them out of harm's way, the challenge of repositioning a casualty from its found configuration to one suitable for extraction has not been explicitly explored. Furthermore, this planning problem needs to incorporate biomechanical safety considerations for the casualty. Thus, we present a first solution to biomechanically safe trajectory generation for repositioning limbs of unconscious human casualties. We describe biomechanical safety as mathematical constraints, mechanical descriptions of the dynamics for the robot-human coupled system, and the planning and trajectory optimization process that considers this coupled and constrained system. We finally evaluate our approach over several variations of the problem and demonstrate it on a real robot and human subject. This work provides a crucial part of search and rescue that can be used in conjunction with past and present works involving robots and vision systems designed for search and rescue.

This paper tackles image segmentation problems for underwater environments. First, we introduce a novel under-water animal-centric dataset with dense pixel-level annotations containing diverse fine-grained animal categories to mitigate the lack of diverse categories in the existing benchmarks. Then, we solve two image segmentation tasks using underwater images in this dataset: (i) few-shot segmentation, and (ii) semantic segmentation. For the segmentation task in a few-shot learning framework, we propose a novel attention-guided deep neural network architecture by infusing attention modules in various stages of our proposed network. We systematically explore how the learned attention maps can improve few-shot segmentation performance for underwater imagery. Finally, we assess the semantic segmentation problem on our proposed dataset by benchmarking it with two state-of-the-art semantic segmentation methods. We believe our new problem setup, i.e., few-shot segmentation for underwater environments, will be a valuable addition to the existing underwater semantic segmentation task. We believe our novel dataset will pave the way for developing better algorithms and exploring new research directions for marine robotics and underwater image understanding. We publicly release our dataset and the code to advance image understanding research in underwater environments: https://github.com/Imran220S/uwsnet.

As a simple and robust mobile robot base, differential drive robots that can be modelled as a kinematic unicycle find significant applications in logistics and service robotics in both industrial and domestic settings. Safe robot navigation around obstacles is an essential skill for such unicycle robots to perform diverse useful tasks in complex cluttered environments, especially around people and other robots. Fast and accurate safety assessment plays a key role in reactive and safe robot motion design. In this paper, as a more accurate and still simple alternative to the standard circular Lyapunov level sets, we introduce novel conic feedback motion prediction methods for bounding the close-loop motion trajectory of the kinematic unicycle robot model under a standard unicycle motion control approach. We present an application of unicycle feedback motion prediction for safe robot navigation around obstacles using reference governors, where the safety of a unicycle robot is continuously monitored based on the predicted future robot motion. We investigate the role of motion prediction on robot behaviour in numerical simulations and conclude that fast and accurate feedback motion prediction is key for fast, reactive, and safe robot navigation around obstacles.

Bronchoscopy is a medical procedure that involves the insertion of a flexible tube with a camera into the airways to survey, diagnose and treat lung diseases. Due to the complex branching anatomical structure of the bronchial tree and the similarity of the inner surfaces of the segmental airways, navigation systems are now being routinely used to guide the operator during procedures to access the lung periphery. Current navigation systems rely on sensor-integrated bronchoscopes to track the position of the bronchoscope in real-time. This approach has limitations, including increased cost and limited use in non-specialized settings. To address this issue, researchers have proposed visual odometry algorithms to track the bronchoscope camera without the need for external sensors. However, due to the lack of publicly available datasets, limited progress is made. To this end, we have developed a database of bronchoscopy videos in a phantom lung model and ex-vivo human lungs. The dataset contains 34 video sequences with over 23,000 frames with odometry ground truth data collected using electromagnetic tracking sensors. With our dataset, we empower the robotics and machine learning community to advance the field. We share our insights on challenges in endoscopic visual odometry. Furthermore, we provide benchmark results for this dataset. State-of-the-art feature extraction algorithms including SIFT, ORB, Superpoint, Shi- Tomasi, and LoFTR are tested on this dataset. The benchmark results demonstrate that the LoFTR algorithm outperforms other approaches, but still has significant errors in the presence of rapid movements and occlusions.

Trajectory prediction of neighboring agents is a critical task for high-speed robotics such as autonomous vehicles. In order to obtain fine-grained and robust scene representations, existing works attempt to consider abundant information that is deemed relevant. The cost, however, is the heavy computational burden and more importantly the inevitable interference brought by redundant information. In this paper, we exploit the explainable AI (XAI) techniques and propose a model in the framework of “Encoder-Decoder” named parallel explainable Transformer (PXT) to identify the contributive features for robust trajectory prediction. A two-branch encoder is designed to disentangle the roadway information and agents' historical trajectories for better feature explanation. Two stages of feature explanation are incorporated into the encoder. In the first stage, an explainable Transformer (XT) comprising a Layer-wise Relevance Propagation (LRP)-based interpretation module is designed and implemented in both branches to score and filter the contextual and motion features. In the second stage of interpretation, the ProbSparse attention mechanism is innovatively adopted to measure the level of interactivity with sparsity, so that the relationships among highly interactive agents are focused on. The results on the Argoverse Benchmark show that our proposal achieves state-of-the-art (SOTA) performance without delicate and tedious network design, demonstrating the effectiveness of tracing and retaining contributive features in enhancing the performance of trajectory prediction.

Unsupervised domain adaptation (UDA) is proposed to better adapt the network trained on labeled synthetic data to unlabeled real-world data for addressing the annotation cost. However, most of these methods pay more attention to domain distributions in input and output stages while ignoring the important differences in semantic expressions and local details in middle feature stages. Therefore, a novel UDA network named FeatDANet is presented to align feature-level domain distributions at each encoder layer. Specifically, two attention-based modules abbreviated as IFAM and DFLM are designed and implemented by mixing queries and keys between domains for advisable domain adaptation. The former realizes Inter-domain Features Alignment by transferring feature style, and the latter achieves Domain-invariant Features Learning robustly for the domain shift. Furthermore, FeatDANet is constructed as a self-training network with three weight-sharing branches, and an improved pseudo-labels learning strategy is suggested by identifying more confident pseudolabels and maximizing the use of pseudo-labels. It increases the participation of unlabeled data and also ensures stability in training. Extensive experiments show that FeatDANet achieves state-of-the-art performances on the tasks of GTA→Cityscapes and Synthia→Cityscapes.

Computing the distance from a point to a triangle mesh is a key computational step in robotics pipelines such as registration and collision detection, with applications to path planning, SLAM, and RGB-D vision. Numerous techniques to accelerate this computation have been developed, many of which use a cheap pre-processing step to construct a hierarchical decomposition of the mesh. If the mesh is fixed and known ahead of time, there is an opportunity to conduct more expensive pre-computations to accelerate the subsequent distance queries. This work presents a voxelization approach, implemented on both CPU and GPU, to compute point to mesh distance that constructs for each voxel a near-minimal set of triangles that is guaranteed to include every triangle that is closest to at least one point in the voxel. Theoretical and numerical comparisons with six alternative distance algorithms demonstrate the speed advantages of the proposed method.

Urban Air Mobility (UAM) promises a new dimension to decongested, safe, and fast travel in urban and suburban hubs. These UAM aircraft are conceived to operate from small airports called vertiports each comprising multiple take-offllanding and battery-recharging spots. Since they might be situated in dense urban areas and need to handle many aircraft landings and take-offs each hour, managing this schedule in real-time becomes challenging for a traditional air-traffic controller but instead calls for an automated solution. This paper provides a novel approach to this problem of Urban Air Mobility - Vertiport Schedule Management (UAM-VSM), which leverages graph reinforcement learning to generate decision-support policies. Here the designated physical spots within the vertiport's airspace and the vehicles being managed are represented as two separate graphs, with feature extraction performed through a graph convolutional network (GCN). Extracted features are passed onto perceptron layers to decide actions such as continue to hover or cruise, continue idling or take-off, or land on an allocated vertiport spot. Performance is measured based on delays, safety (no. of collisions) and battery consumption. Through realistic simulations in AirSim applied to scaled down multi-rotor vehicles, our results demonstrate the suitability of using graph reinforcement learning to solve the UAM-VSM problem and its superiority to basic reinforcement learning (with graph embed dings) or random choice baselines.

Visual navigation has become a standard in robotic applications with the emergence of robust and versatile algorithms. In particular, Visual Odometry (VO) has proven to be the most reliable navigation solution for space missions to estimate an unmanned vehicle's motion and state. Lava Tubes exploration is one of the recent challenges in this field of applied robotics. VO in this scenario requires more robustness to poor lighting conditions while keeping a low computational cost. We propose investigating an indirect bi-monocular VO based on sliding-window optimization in such a context. It focuses on maintaining the sparsity of the problem while keeping the information of the marginalized frames to reduce the computational burden. Different sparse graph topologies are studied to encode information from the past and are evaluated on accuracy and computation load. The best method retained is then compared to state-of-the-art systems on real data under extreme illumination conditions and reaches similar accuracy results at a lower computational cost.

This paper presents Fast Adaptive Tree (FAT), an asymptotically-optimal sampling-based path planner for dynamic and uncertain scenarios. Namely, the solution extracted converges to the optimal solution given the sensor information as the number of samples approaches infinity. The planner maintains an underlying graph, which increasingly approximates the search domain, and a dynamic spanning tree of the graph, which contains the shortest path from the start to the goal state. The planner quickly responds to the availability of new information about the environments or the robot movements by minimally repairing the spanning tree over the navigation. The simulation results show that the proposed path planner achieves higher efficiency of replanning than several state-of-the-art path planners without sacrificing solution quality.

The tradeoff between the field-of-view and resolution of conventional onboard vision systems primarily results from their fixed optical components. We propose a novel active vision system, Falcon, as an optimal solution. This system comprises an electric zoom lens connected to a high-speed camera with a pair of galvanometer mirrors, enabling high-resolution imaging of a moving object across a wide range, from near to far. To ensure accurate calibration of the Falcon system, we introduce a novel mapping-based calibration method using external cameras. We also present a robust and lightweight visual feedback method that utilizes this mapping-based calibration for effective object tracking. The effectiveness of the Falcon system is verified by constructing a prototype and conducting tracking experiments in an indoor setting, which demonstrated the superior performance of our method. Additionally, we successfully achieved continuous and high-resolution imaging of a curved mirror on public roads while the vehicle was moving.