IROS 2024 论文列表

A global modeling methodology based on Koopman operator theory for the dynamics of rigid bodies that make and break contact is presented. Traditionally, robotic systems that contact with their environment are represented as a system comprised of multiple dynamic equations that are switched depending on the contact state. This switching of governing dynamics has been a challenge in both task planning and control. Here, a Koopman lifting linearization approach is presented to subsume multiple dynamics such that no explicit switching is required for examining the dynamic behaviors across diverse contact states. First, it is shown that contact/non-contact transitions are continuous at a microscopic level. This allows for the application of Koopman operator theory to the class of robotic systems that repeat contact/non-contact transitions. Second, an effective method for finding Koopman operator observables for capturing rapid changes to contact forces is presented. The method is applied to the modeling of dynamic peg insertion where a peg collides against and bounces on the chamfer of the hole. Furthermore, the method is applied to the dynamic modeling of a sliding object subject to complex friction and damping properties. Segmented dynamic equations are unified with the Koopman modeling method.

In this paper, we introduce a knowledge-based Programming by Demonstration (kb-PbD) paradigm to facilitate robot programming in small and medium-sized enterprises (SMEs). PbD in production scenarios requires the recognition of product-specific actions but faces challenges in the lack of suitable and comprehensive datasets, due to the large variety of involved hand actions across different production scenarios. To address this issue, we utilize standardized grasp types as the fundamental feature to recognize basic hand movements, where a Long Short-Term Memory (LSTM) network is employed to recognize grasp types from hand landmarks. The product-specific actions, aggregated from the basic hand movements, are formally modeled in a semantic description language based on the Web Ontology Language (OWL). Description Logic (DL) is used to define the actions with their characteristic properties, which enables the efficient classification of new action instances by an OWL reasoner.The semantic models of hand actions, robot tasks, and work-cell resources are interconnected and stored in a Knowledge Base (KB), which enables the efficient pair-wise translation between hand actions and robot tasks. For the reproduction of human assembly processes, actions are converted to robot tasks via skill descriptions, while reusing the action parameters of involved objects to ensure product integrity. We showcase and evaluate our method in an industrial production setting for control cabinet assembly. Demonstration video available at: https://kb-pbd.github.io/.

Assistive robot arms have the potential to help disabled or elderly adults eat everyday meals without relying on a caregiver. To provide meaningful assistance, these robots must reach for food items, pick them up, and then carry them to the human’s mouth. Current work equips robot arms with standard utensils (e.g., forks and spoons). But — although these utensils are intuitive for humans — they are not easy for robots to control. If the robot arm does not carefully and precisely orchestrate its motion, food items may fall out of a spoon or slide off of the fork. Accordingly, in this paper we design, model, and test Kiri-Spoon, a novel utensil specifically intended for robot-assisted feeding. Kiri-Spoon combines the familiar shape of traditional utensils with the capabilities of soft grippers. By actuating a kirigami structure the robot can rapidly adjust the curvature of Kiri-Spoon: at one extreme the utensil wraps around food items to make them easier for the robot to pick up and carry, and at the other extreme the utensil returns to a typical spoon shape so that human users can easily take a bite of food. Our studies with able-bodied human operators suggest that robot arms equipped with Kiri-Spoon carry foods more robustly than when leveraging traditional utensils. See videos here: https://youtu.be/nddAniZLFPk

We propose a motion planner for cable-driven payload transportation using multiple unmanned aerial vehicles (UAVs) in an environment cluttered with obstacles. Our planner is kinodynamic, i.e., it considers the full dynamics model of the transporting system including actuation constraints. Due to the high dimensionality of the planning problem, we use a hierarchical approach where we first solve for the geometric motion using a sampling-based method with a novel sampler, followed by constrained trajectory optimization that considers the full dynamics of the system. Both planning stages consider inter-robot and robot/obstacle collisions. We demonstrate in a software-in-the-loop simulation and real flight experiments that there is a significant benefit in kinodynamic motion planning for such payload transport systems with respect to payload tracking error and energy consumption compared to the standard methods of planning for the payload alone. Notably, we observe a significantly higher success rate in scenarios where the team formation changes are needed to move through tight spaces.

Bridge inspections are essential for maintaining key infrastructure and preventing structural and functional failures. Nevertheless, traditional manual inspection techniques are plagued by laboriousness, high risk, and low efficiency. Although numerous automation inspection methods have been studied, inspection performance remains challenging. The main difficulties are redundant mechanisms, complex control, high energy consumption, and limited autonomy and safety. To address these problems, we are developing a small, lightweight, electrically-driven robotic manipulator for bridge inspection named the BIRM. Here, we propose a kinetic-energy-optimal and safety-guaranteed trajectory planning for BIRM. Compared with existing methods, it simultaneously addresses energy consumption and safety. The approach formulates a quadratic programming (QP) problem by considering the robot’s kinetic energy as the objective function, and the augmented Lagrange multiplier (ALM) is applied to find the solution of the QP. The proposed method completely satisfies the joint position, velocity, and acceleration limits at the speed level while considering collision avoidance. In this paper, the collision detection strategy can achieve low-complexity computation through several structural parameters of the bridge, thereby quickly adapting to environmental changes. Through simulation experiments, we validate the effectiveness and superiority of the proposed method. Through physical experiments, we demonstrate the sustainability and safety of bridge inspections in the field.

Trajectory generation and trajectory prediction are two critical tasks in autonomous driving, which generate various trajectories for testing during development and predict the trajectories of surrounding vehicles during operation, respectively. In recent years, emerging data-driven deep learning-based methods have shown great promise for these two tasks in learning various traffic scenarios and improving average performance without assuming physical models. However, it remains a challenging problem for these methods to ensure that the generated/predicted trajectories are physically realistic. This challenge arises because learning-based approaches often function as opaque black boxes and do not adhere to physical laws. Conversely, existing model-based methods provide physically feasible results but are constrained by predefined model structures, limiting their capabilities to address complex scenarios. To address the limitations of these two types of approaches, we propose a new method that integrates kinematic knowledge into neural stochastic differential equations (SDE) and designs a variational autoencoder based on this latent kinematics-aware SDE (LK-SDE) to generate vehicle motions. Experimental results demonstrate that our method significantly outperforms both model-based and learning-based baselines in producing physically realistic and precisely controllable vehicle trajectories. Additionally, it performs well in predicting unobservable physical variables in the latent space.

Twisted String Actuators (TSAs) exhibit several advantages, including lightweight, compact, and having a high power-to-weight ratio. However, current research on kinematic models of TSAs is limited to deriving the relationship between motor input and output through idealized geometric calculations. Therefore, the accuracy of these models does not meet the requirements for practical applications. Previous studies on the kinematic modeling of TSA have not considered the impact of material plastic deformation and stroke times on TSA kinematics. Accumulation of plastic deformation over multiple strokes leads to changes in the output displacement of TSA, significantly affecting the accuracy of the kinematic model. This study aims to address the limitations of previous research by investigating the use of Invertible Neural Networks (INNs) in kinematic modeling of TSAs, taking into account material plastic deformation and stroke times. Through a series of TSA experiments, a kinematic model of TSA was established using an INN that considers stroke times. The INN model proves to be superior in both forward and inverse kinematic modeling by effectively compensating for the effects of plastic deformation during TSA operation. The experimental results demonstrate that the kinematic model established by the proposed INN is more aligned with the actual conditions when compared to traditional kinematic models. This insight can aid in predicting the lifespan of TSA in the future.

Tuning on instruction-following data has been shown to enhance the capabilities and controllability of language models, but the idea is less explored in the robotic field. In this work, we introduce KOSMOS-E, a Multimodal Large Language Model (MLLM) that leverages instruction-following robotic grasping data to enhance capabilities for precise and intricate robotic grasping maneuvers. To achieve this, we craft a large-scale instruction-following robotic grasping dataset, termed INSTRUCT-GRASP, primarily comprising two aspects: (i) grasp a single object following varying levels of granularity descriptions, e.g., different angles and aspects, and (ii) grasp a specific object within a multi-object environment following specific attributes, e.g., color and shape. Extensive experiments show the effectiveness of KOSMOS-E on robotic grasping tasks across a variety of environments.

This paper presents a Kalman filter based LiDAR-Inertial-Leg Odometry (KLILO) system for legged robots to navigate in challenging environments. In particular, we employ the iterated error-state extended Kalman filter framework on manifolds to fuse measurements from the inertial measurement unit (IMU), LiDAR, joint encoders, and contact force sensors in a tightly coupled manner. To assess the performance of KLILO, we build a dataset that encompasses intricate environments with challenging conditions such as dynamic objects and deformable terrains. The results demonstrate that our algorithm can provide efficient and reliable localization in all tests. It exhibits an average improvement of around 40% in positioning accuracy compared to the baselines. Furthermore, we validate KLILO in a challenging navigation task on a real robot, where the LiDAR encounters ineffective measurements.

With the advent of Neural Radiance Field (NeRF), representing 3D scenes through multiple observations has shown significant improvements. Since this cutting-edge technique can obtain high-resolution renderings by interpolating dense 3D environments, various approaches have been proposed to apply NeRF for the spatial understanding of robot perception. However, previous works are challenging to represent unobserved scenes or views on the unexplored robot trajectory, as these works do not take into account 3D reconstruction without observation information. To overcome this problem, we propose a method to generate flipped observation in order to cover absent observation for unexplored robot trajectory. Our approach involves a data augmentation technique for 3D reconstruction using NeRF, by flipping observed images and estimating the 6DOF poses of the flipped cameras. Furthermore, to ensure the NeRF model operates robustly in general scenarios, we also propose a training method that adjusts the flipped pose and considers the uncertainty in flipped images accordingly. Our technique does not utilize an additional network, making it simple and fast, thus ensuring its suitability for robotic applications where real-time performance is crucial.

Learning from Demonstration (LfD) allows robots to mimic human actions. However, these methods do not model constraints crucial to ensure safety of the learned skill. Moreover, even when explicitly modelling constraints, they rely on the assumption of a known cost function, which limits their practical usability for task with unknown cost. In this work we propose a two-step optimization process that allow to estimate cost and constraints by decoupling the learning of cost functions from the identification of unknown constraints within the demonstrated trajectories. Initially, we identify the cost function by isolating the effect of constraints on parts of the demonstrations. Subsequently, a constraint leaning method is used to identify the unknown constraints. Our approach is validated both on simulated trajectories and a real robotic manipulation task. Our experiments show the impact that incorrect cost estimation has on the learned constraints and illustrate how the proposed method is able to infer unknown constraints, such as obstacles, from demonstrated trajectories without any initial knowledge of the cost.

Unmanned aerial vehicles (UAVs) visual localization in planetary aims to estimate the absolute pose of the UAV in the world coordinate system through satellite maps and images captured by on-board cameras. However, since planetary scenes often lack significant landmarks and there are modal differences between satellite maps and UAV images, the accuracy and real-time performance of UAV positioning will be reduced. In order to accurately determine the position of the UAV in a planetary scene in the absence of the global navigation satellite system (GNSS), this paper proposes JointLoc, which estimates the real-time UAV position in the world coordinate system by adaptively fusing the absolute 2-degree-of-freedom (2-DoF) pose and the relative 6-degree-of-freedom (6-DoF) pose. Extensive comparative experiments were conducted on a proposed planetary UAV image cross-modal localization dataset, which contains three types of typical Martian topography generated via a simulation engine as well as real Martian UAV images from the Ingenuity helicopter. JointLoc achieved a root-mean-square error of 0.237m in the trajectories of up to 1,000m, compared to 0.594m and 0.557m for ORB-SLAM2 and ORB-SLAM3 respectively. The source code will be available at https://github.com/LuoXubo/JointLoc.

We propose an effective whole-body MPC controller for locomotion of humanoid robots. Our method generates motions using the full kinematics, allowing it to account for joint limits and to exploit upper-body motions to reject disturbances. Each MPC iteration solves a single QP that considers the interplay between dynamic and kinematic features of the robot. Thanks to our special formulation, we are able to perform a feasibility analysis, which opens the door to future enhancements of functionality and performance, e.g., step adaptation in complex environments. We demonstrate its effectiveness through a campaign of dynamic simulations aimed at highlighting how the joint limits and the use of the angular momentum through upper-body motions are fundamental for maximizing performance, robustness, and ultimately make the robot able to execute more challenging gaits.

Joint pedestrian trajectory prediction has long grappled with the inherent unpredictability of human behaviors. Recent works employing conditional diffusion models in trajectory prediction have exhibited notable success. Nevertheless, the heavy dependence on accurate historical data results in their vulnerability to noise disturbances and data incompleteness. To improve the robustness and reliability, we introduce the Guided Full Trajectory Diffuser (GFTD), a novel diffusion-based framework that translates prediction as the inverse problem of spatial-temporal inpainting and models the full joint trajectory distribution which includes both history and the future. By learning from the full trajectory and leveraging flexible posterior sampling methods, GFTD can produce accurate predictions while improving the robustness that can generalize to scenarios with noise perturbation or incomplete historical data. Moreover, the pre-trained model enables controllable generation without an additional training budget. Through rigorous experimental evaluation, GFTD exhibits superior performance in joint trajectory prediction with different data quality and in controllable generation tasks. See more results at https://sites.google.com/andrew.cmu.edu/posterior-sampling-prediction.

While learning from demonstrations is powerful for acquiring visuomotor policies, high-performance imitation without large demonstration datasets remains challenging for tasks requiring precise, long-horizon manipulation. This paper proposes a pipeline for improving imitation learning performance with a small human demonstration budget. We apply our approach to assembly tasks that require precisely grasping, reorienting, and inserting multiple parts over long horizons and multiple task phases. Our pipeline combines expressive policy architectures and various techniques for dataset expansion and simulation-based data augmentation. These help expand dataset support and supervise the model with locally corrective actions near bottleneck regions requiring high precision. We demonstrate our pipeline on four furniture assembly tasks in simulation, enabling a manipulator to assemble up to five parts over nearly 2500 time steps directly from RGB images, outperforming imitation and data augmentation baselines. Project website: https://imitation-juicer.github.io/.

In this paper, an iterative learning strategy was developed to improve trajectory tracking for an impedance-controlled robot manipulator. In this learning strategy, an update law was proposed to modify the Cartesian reference of an impedance controller. Also, the conditions that ensure its convergence considering the dynamics of the robot were derived. Finally, an experimental evaluation was performed using a Franka Emika Panda robot in two different robot tasks, and its results showed that robot task completion was achieved in a lower number of iterations, while maintaining a smooth physical interaction between the robot and its surroundings.

This study explores the feasibility of using simulations as a better interface to collect human object manipulation skills for learning from demonstrations (LfD). Recently, numerous researchers have started introducing teleoperation systems to acquire human manipulation skills. However, capturing the subtle, force-involved interaction skills of humans in teleoperation is still challenging due to its inherent dynamic delays and feedback transparency. This research evaluates the effectiveness of demonstration data obtained through simulation versus teleoperation. To evaluate the efficacy of this approach, tasks such as plane cutting, tight peg-in-hole, and deformable pipe plugging were performed to assess the quality of demonstrations acquired. The experimental results highlight the effectiveness of demonstration through simulation in capturing the operator’s force-involved interaction skills. Simulation creates an environment similar to performing tasks with bare hands by minimising dynamic delays due to the exclusion of physical robots and effectively rendering high stiffness. As a result, the demonstration through simulation method has proven effective in extracting interaction data and capturing physical task performance skills.

Autonomous delivery robots (ADRs) encounter incidentally copresent persons (InCoPs) during their delivery journeys. Despite the potential for ADRs' behavior to influence the behavior and cognition of InCoPs, there is limited research on the interaction between ADRs and InCoPs. Therefore, in this study, we conducted a within-participants experiment (N=30) with a 3 (confederate types: humans vs. high anthropomorphism robots vs. low anthropomorphism robots) x 2 (jaywalking status: jaywalking vs. not jaywalking) design to investigate the impact of ADRs on InCoPs’ behavioral and cognitive changes induced by the social influence of ADRs. During the experiment, participants watched a video depicting interactions between ADRs and InCoPs at a crosswalk. Each participant was immersed in the video as an InCoP, instructed to make jaywalking decisions, and subsequently completed questionnaires. Results indicated that, behaviorally, participants displayed similar levels of conformity towards jaywalking behaviors across both the human and robot confederates. Cognitively, there were significant differences in morality based on the confederate types. Additionally, robots that refrained from jaywalking received more positive ratings in terms of morality and intention to use. This study confirms that ADRs have the capacity to induce conformity similar to humans and that the ethical behavior of ADRs can positively influence InCoPs' impressions and intention to use toward ADRs.

This paper introduces InverseMatrixVT3D, an efficient method for transforming multi-view image features into 3D feature volumes for 3D semantic occupancy prediction. Existing methods for constructing 3D volumes often rely on depth estimation, device-specific operators, or transformer queries, which hinders the widespread adoption of 3D occupancy models. In contrast, our approach leverages two projection matrices to store the static mapping relationships and matrix multiplications to efficiently generate global Bird’s Eye View (BEV) features and local 3D feature volumes. Specifically, we achieve this by performing matrix multiplications between multi-view image feature maps and two sparse projection matrices. We introduce a sparse matrix handling technique for the projection matrices to optimize GPU memory usage. Moreover, a global-local attention fusion module is proposed to integrate the global BEV features with the local 3D feature volumes to obtain the final 3D volume. We also employ a multi-scale supervision mechanism to enhance performance further. Extensive experiments performed on the nuScenes and SemanticKITTI datasets reveal that our approach not only stands out for its simplicity and effectiveness but also achieves the top performance in detecting vulnerable road users (VRU), crucial for autonomous driving and road safety. The code has been made available at: https://github.com/DanielMing123/InverseMatrixVT3D