IROS 2024 Papers

Communication constraints can significantly impact robots’ ability to share information, coordinate their movements, and synchronize their actions, thus limiting coordination in Multi-Robot Exploration (MRE) applications. In this work, we address these challenges by modeling the MRE application as a DEC-POMDP and designing a joint policy that follows a rendezvous plan. This policy allows robots to explore unknown environments while intermittently sharing maps opportunistically or at rendezvous locations without being constrained by joint path optimizations. To generate the rendezvous plan, robots represent the MRE task as an instance of the Job Shop Scheduling Problem (JSSP) and minimize JSSP metrics. They aim to reduce waiting times and increase connectivity, which correlates to the DEC-POMDP rewards and time to complete the task. Our simulation results suggest that our method is more efficient than using relays or maintaining intermittent communication with a base station, being a suitable approach for Multi-Robot Exploration. We developed a proof-of-concept using the Robot Operating System (ROS) that is available at: https://github.com/multirobotplayground/Noetic-Multi-Robot-Sandbox.

Planning for contact-rich manipulation involves discontinuous dynamics, which presents challenges to planning methods. Sampling-based planners have higher sample complexity in high-dimensional problems and cannot efficiently handle state constraints such as force limits. Gradient-based solvers can suffer from local optima and their convergence rate is often worse on non-smooth problems. We propose a planning method that is both sampling- and gradient-based, using the Cross-entropy Method to initialize a gradient-based solver, providing better initialization to the gradient-based method and allowing explicit handling of state constraints. The sampling-based planner also allows direct integration of a particle filter, which is here used for online contact mode estimation. The approach is shown to improve performance in MuJoCo environments and the effects of problem stiffness and planing horizon are investigated. The estimator and planner are then applied to an impedance-controlled robot, showing a reduction in solve time in contact transitions to only gradient-based.

Lifestyle support through robotics is an increasingly promising field, with expectations for robots to take over or assist with chores like floor cleaning, table setting and clearing, and fetching items. The growth of AI, particularly foundation models, such as large language models (LLMs) and visual language models (VLMs), is significantly shaping this sector. LLMs, by facilitating natural interactions and providing vast general knowledge, are proving invaluable for robotic tasks. This paper focuses on the benefits of LLMs for "bring-me" tasks, where robots fetch specific items for users, often based on ambiguous instructions. Our previous efforts utilized an ontology extended to handle environmental data to resolve such ambiguities, but faced limitations when unresolvable ambiguities required user intervention for clarity. Here, we enhance our approach by integrating LLMs for providing additional commonsense knowledge, pairing it with ontological data to mitigate the issue of hallucinations and reduce the need for user queries, thus improving system usability. We present a system that merges these knowledge bases and assess its efficacy on "bring-me" tasks, aiming to provide a more seamless and efficient robotic assistance experience.

Navigating robots safely and efficiently in crowded and complex environments remains a significant challenge. However, due to the dynamic and intricate nature of these settings, planning efficient and collision-free paths for robots to track is particularly difficult. In this paper, we uniquely bridge the robot’s perception, decision-making and control processes by utilizing the convex obstacle-free region computed from 2D LiDAR data. The overall pipeline is threefold: (1) We proposes a robot navigation framework that utilizes deep reinforcement learning (DRL), conceptualizing the observation as the convex obstacle-free region, a departure from general reliance on raw sensor inputs. (2) We design the action space, derived from the intersection of the robot’s kinematic limits and the convex region, to enable efficient sampling of inherently collision-free reference points. These actions assists in guiding the robot to move towards the goal and interact with other obstacles during navigation. (3) We employ model predictive control (MPC) to track the trajectory formed by the reference points while satisfying constraints imposed by the convex obstacle-free region and the robot’s kinodynamic limits. The effectiveness of proposed improvements has been validated through two sets of ablation studies and a comparative experiment against the Timed Elastic Band (TEB), demonstrating improved navigation performance in crowded and complex environments.

Collision detection is a fundamental problem across various fields such as robotics, physical simulation, and computer graphics. While numerous studies have provided efficient solutions, based on the well-known Gilbert, Johnson, and Keerthi (GJK) algorithm and Expanding Polytope Algorithm (EPA), existing methods utilizing GJK-EPA often struggle with smooth strictly convex shapes like ellipsoids. This paper proposes a novel approach to the collision detection problem converting it to a problem compatible with an unconstrained Riemannian optimization problem. Moreover, we presents a specific method of solving the problem based on twice differentiable support functions and the Riemannian trust region (RTR) method. The method exhibits fast and robust convergence rate, leveraging the well-established theory of Riemannian optimization. The evaluation studies comparing our method to GJK-EPA method are done with pre-defined primitive shapes. Additionally, a test result with several more complex shapes is demonstrated exhibiting the method’s effectiveness and applicability.

This paper introduces an innovative methodology for object manipulation on the surface of water through the collaboration of an Unmanned Aerial Vehicle (UAV) and an Unmanned Surface Vehicle (USV) connected to the object by tethers. We propose a novel mathematical model of a robotic system that combines the UAV, USV, and the tethered floating object. A novel Model Predictive Control (MPC) framework is designed for using this model to achieve precise control and guidance for this collaborative robotic system. Extensive simulations in the realistic robotic simulator Gazebo demonstrate the system’s readiness for real-world deployment, highlighting its versatility and effectiveness. Our multi-robot system overcomes the state-of-the-art single-robot approach, exhibiting smaller control errors during the tracking of the floating object’s reference. Additionally, our multi-robot system demonstrates a shorter recovery time from a disturbance compared to the single-robot approach.

In the context of Human-Robot Collaboration (HRC), it is crucial that the two actors are able to communicate with each other in a natural and efficient manner. The absence of a communication interface is often a cause of undesired slowdowns. On one hand, this is because unforeseen events may occur, leading to errors. On the other hand, due to the close contact between humans and robots, the speed must be reduced significantly to comply with safety standard ISO/TS 15066. In this paper, we propose a novel architecture that enables operators and robots to communicate efficiently, emulating human-to-human dialogue, while addressing safety concerns. This approach aims to establish a communication framework that not only facilitates collaboration but also reduces undesired speed reduction. Through the use of a predictive simulator, we can anticipate safety-related limitations, ensuring smoother workflows, minimizing risks, and optimizing efficiency. The overall architecture has been validated with a UR10e and compared with a state of the art technique. The results show a significant improvement in user experience, with a corresponding 23% reduction in execution times and a 50% decrease in robot downtime.

We investigate a pursuit-evasion game taking place in an unbounded three-dimensional space, where a flexible pursuer with hybrid dynamics collaborates with a fast pursuer and aims to capture a flexible evader within a finite time. The key feature of this problem lies in the hybrid dynamics of the flexible pursuer, which can change its dynamics once during the game and switch to a fast pursuer with increased speed but lower maneuverability. To address this challenge, we devise a hybrid strategy based on the soft actor-critic framework, tailored specifically for the flexible pursuer, which encompasses both maneuvering and switch tactics. We introduce a switch factor to the input of the actor network and incorporate switch actions to further expand the action space. These additions enable the flexible pursuer to execute maneuvering actions and determine a moment to switch to a fast pursuer. The reward function is designed to account for related angle, altitude, speed, and sparse reward. Through extensive ablation experiments conducted in a simulated environment, we demonstrate the efficacy of our algorithm in facilitating the learning of hybrid strategies for the flexible pursuer, resulting in significantly improved capture rates compared to alternative methods.

This paper presents CollabLoc, a novel approach for real-time multi-user visual localization. Typically, localization systems employ a client-server design for locating cameras. In these systems, lightweight simultaneous localization and mapping computations are performed on the client side, while the server handles intensive localization tasks. This approach harnesses the complementary capabilities of the client and server, resulting in accurate, real-time localization results. However, existing architectures primarily operate on a one-to-one client-server structure, limiting their scalability and multi-user capabilities. Therefore, CollabLoc is designed to accommodate multiple clients through collaborative information sharing to considerably reduce computational overhead and enhance overall efficiency and accuracy. We propose a tracking confidence module that evaluates the tracking quality of individual clients and plays a pivotal role in prioritizing client requests by the server-side algorithm. On the server, we utilize fused poses to accelerate image retrieval. Moreover, we enhance the efficiency of optical flow estimation by employing a simplified feature extraction module and leveraging spatial similarities among neighboring clients to improve its performance. Finally, via the Pose Fusion Module, the server can periodically adjust fused poses to mitigate accumulated errors. Experimental results indicate that compared with a baseline method, CollabLoc improves positioning efficiency by nearly twice and achieves higher accuracy in multi-user scenarios.

Efficiently detecting target weld seams while ensuring sub-millimeter accuracy has always been an important challenge in autonomous welding, which has significant application in industrial practice. Previous works mostly focused on recognizing and localizing welding seams one by one, leading to inferior efficiency in modeling the workpiece. This paper proposes a novel framework capable of multiple weld seams extraction using both RGB images and 3D point clouds. The RGB image is used to obtain the region of interest by approximately localizing the weld seams, and the point cloud is used to achieve the fine-edge extraction of the weld seams within the region of interest using region growth. Our method is further accelerated by using a pre-trained deep learning model to ensure both efficiency and generalization ability. The proposed method was comprehensively tested on various workpieces featuring both linear and curved weld seams, as well as in physical experiment systems. The results showcase considerable potential for real-world industrial applications, emphasizing the method’s efficiency and effectiveness. Videos of the real-world experiments can be found at https://youtu.be/pq162HSP2D4.

This paper studies a case of the multi-robot task allocation (MRTA) problem, where each unmanned aerial vehicle (UAV) is endowed with multiple but limited resources. Completing each task necessitates UAVs to combine different resources through coalition formation, which will incur various costs including flight cost, execution cost, and cooperation cost. To minimize the total cost while maximizing both task completion rate and resource utilization rate, we model the MRTA problem of the UAVs as a leader-follower coalition formation game. In this game, leader UAVs coordinate follower UAVs to fulfill task resource requisites. Meanwhile, follower UAVs select suitable coalitions to join based on the altruistic preference. Theoretical analysis confirms the existence of a Nash stable partition in the coalition formation game. To achieve this stable partition, we propose a coalition formation algorithm. Simulation experiments validate that the proposed algorithm outperforms existing methods for the MRTA problem under resource constraints in terms of both task completion rate and resource utilization rate.

Autonomous agents often face the challenge of interpreting uncertain natural language instructions for planning tasks. Representing these instructions as Linear Temporal Logic (LTL) enables planners to synthesize actionable plans. We introduce CoT-TL, a data-efficient in-context learning framework for translating natural language specifications into LTL representations. CoT-TL addresses the limitations of large language models, which typically rely on extensive fine-tuning data, by extending chain-of-thought reasoning and semantic roles to align with the requirements of formal logic creation. This approach enhances the transparency and rationale behind LTL generation, fostering user trust. CoT-TL achieves state-of-the-art accuracy across three diverse datasets in low-data scenarios, outperforming existing methods without fine-tuning or intermediate translations. To improve reliability and minimize hallucinations, we incorporate model checking to validate the syntax of the generated LTL output. We further demonstrate CoT-TL’s effectiveness through ablation studies and evaluations on unseen LTL structures and formulas in a new dataset. Finally, we validate CoT-TL’s practicality by integrating it into a QuadCopter for multi-step drone planning based on natural language instructions.

Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object’s grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how CoPa can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that CoPa possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training. Project page: copa-2024.github.io

Assembly processes involving humans and robots are challenging scenarios because the individual activities and access to shared workspace have to be coordinated. Fixed robot programs leave no room to diverge from a fixed protocol. Working on such a process can be stressful for the user and lead to ineffective behavior or failure. We propose a novel approach of online constraint-based scheduling in a reactive execution control framework facilitating behavior trees called CoBOS. This allows the robot to adapt to uncertain events such as delayed activity completions and activity selection (by the human). The user will experience less stress as the robotic coworkers adapt their behavior to best complement the human-selected activities to complete the common task. In addition to the improved working conditions, our algorithm leads to increased efficiency, even in highly uncertain scenarios. We evaluate our algorithm using a probabilistic simulation study with 56000 experiments. We outperform all other compared methods by a margin of 4−10%. Initial real robot experiments using a Franka Emika Panda robot and human tracking based on HTC Vive VR gloves look promising.

In this work, we present a novel algorithm to perform spill-free handling of open-top liquid-filled containers that operates in cluttered environments. By allowing liquid-filled containers to be tilted at higher angles and enabling motion along all axes of end-effector orientation, our work extends the reachable space and enhances maneuverability around obstacles, broadening the range of feasible scenarios. Our key contributions include: i) generating spill-free paths through the use of RRT* with an informed sampler that leverages container properties to avoid spill-inducing states (such as an upside-down container), ii) parameterizing the resulting path to generate spill-free trajectories through the implementation of a time parameterization algorithm, coupled with a transformer-based machine-learning model capable of classifying trajectories as spill-free or not. We validate our approach in real-world, obstacle-rich task settings using containers of various shapes and fill levels and demonstrate an extended solution space that is at least 3x larger than an existing approach.

This paper introduces climbing gait for the snake robot by adapting to a flexible net. A net deforms in various ways due to external forces. Therefore, we tilt snake robot’s body to adapt to the deformation of the net. In addition, we can adjust the position of the head of a snake robot passing through the mesh. A snake robot can move not only vertically but also horizontally. We demonstrated the validity of the proposed method through experiments in vertical movement and movement in diagonal direction.

Precise manipulation that is generalizable across scenes and objects remains a persistent challenge in robotics. Current approaches for this task heavily depend on having a significant number of training instances to handle objects with pronounced visual and/or geometric part ambiguities. Our work explores the grounding of fine-grained part descriptors for precise manipulation in a zero-shot setting by utilizing web-trained text-to-image diffusion-based generative models. We tackle the problem by framing it as a dense semantic part correspondence task. Our model returns a gripper pose for manipulating a specific part, using as reference a user-defined click from a source image of a visually different instance of the same object. We require no manual grasping demonstrations as we leverage the intrinsic object geometry and features. Practical experiments in a real-world tabletop scenario validate the efficacy of our approach, demonstrating its potential for advancing semantic-aware robotics manipulation.Web page: https://tsagkas.github.io/click2grasp

Spatial cognition refers to the ability to gain knowledge about their surroundings and utilize this information to identify their location, acquire resources, and navigate their way back to familiar places. People with blindness and low vision (pBLV) face significant challenges with spatial cognition due to the reliance on visual input. Without the full range of visual cues, pBLV individuals often find it difficult to grasp a comprehensive understanding of their environment, leading to obstacles in scene recognition and precise object localization, especially in unfamiliar environments. This limitation extends to their ability to independently detect and avoid potential tripping hazards, making navigation and interaction with their environment more challenging. In this paper, we present a pioneering wearable platform tailored to enhance the spatial cognition of pBLV through the integration of multi-modal foundation model. The proposed platform integrates a wearable camera with audio module and leverages the advanced capabilities of vision language foundation model (i.e., GPT-4 and GPT-4V), for the nuanced processing of visual and textual data. Specifically, we employ vision language models to bridge the gap between visual information and the proprioception of visually impaired users, offering more intelligible guidance by aligning visual data with the natural perception of space and movement. Then we apply prompt engineering to guide the large language model to act as an assistant tailored specifically for pBLV users to produce accurate answers. Another innovation in our model is the incorporation of a chain of thought reasoning process, which enhances the accuracy and interpretability of the model, facilitating the generation of more precise responses to complex user inquiries across diverse environmental contexts. To assess the practical impact of our proposed wearable platform, we carried out a series of real-world experiments across three tasks that are commonly... Show More

For safety-critical robotics applications such as autonomous driving, it is important to detect all required objects accurately in real-time. Motion segmentation offers a solution by identifying dynamic objects from the scene in a class-agnostic manner. Recently, various motion segmentation models have been proposed, most of which jointly use subnetworks to estimate Depth, Pose, Optical Flow, and Scene Flow. As a result, the overall computational cost of the model increases, hindering real-time performance.In this paper, we propose a novel cost-volume-based motion feature representation, Channel-wise Motion Features. By extracting depth features of each instance in the feature map and capturing the scene’s 3D motion information, it offers enhanced efficiency. The only subnetwork used to build Channel-wise Motion Features is the Pose Network, and no others are required. Our method not only achieves about 4 times the FPS of state-of-the-art models in the KITTI Dataset and Cityscapes of the VCAS-Motion Dataset, but also demonstrates equivalent accuracy while reducing the parameters to about 25%.