IROS 2023 Papers

For socially assistive robots to achieve widespread adoption, the ability to learn new tasks in the wild is critical. Learning from Demonstration (LfD) approaches are a popular method for learning in the wild, but current methods require significant amounts of data and can be difficult to interpret. Interactive Task Learning (ITL) is an emerging learning paradigm that aims to teach tasks in a structured manner, minimizing the need for data and increasing transparency. However, to date ITL has only been explored for physical robotics applications. Additionally, minimal research has explored how usable existing ITL systems are for non-expert users. In this work, we propose a novel approach to learn social tasks via ITL. This system utilizes recent advances in Natural Language Understanding (NLU) to learn from natural dialogue. We conducted a pilot study to compare the ITL system against an LfD approach to investigate differences in teaching performance as well as teachers' perceptions of trust and workload towards these systems. Additionally, we analyzed the teaching behavior of participants to identify successful and unsuccessful teaching strategies. Our findings suggest ITL could provide more transparency to users and improve performance by correcting speech recognition errors. However, participants generally preferred LfD and found it an easier teaching method. From the observed teaching behavior, we identify existing challenges in ITL for non-experts to teach social tasks. Using this, we propose areas of improvement toward future ITL learning paradigms that are intuitive, transparent, and performant.

Recognizing interactive action plays an important role in human-robot interaction and collaboration. Previous methods use late fusion and co-attention mechanism to capture interactive relations, which have limited learning capability or inefficiency to adapt to more interacting entities. With assumption that priors of each entity are already known, they also lack evaluations on a more general setting addressing the diversity of subjects. To address these problems, we propose an Interactive Spatiotemporal Token Attention Network (ISTA-Net), which simultaneously model spatial, temporal, and interactive relations. Specifically, our network contains a tokenizer to partition Interactive Spatiotemporal Tokens (ISTs), which is a unified way to represent motions of multiple diverse entities. By extending the entity dimension, ISTs provide better interactive representations. To jointly learn along three dimensions in ISTs, multi-head self-attention blocks integrated with 3D convolutions are designed to capture inter-token correlations. When modeling correlations, a strict entity ordering is usually irrelevant for recognizing interactive actions. To this end, Entity Rearrangement is proposed to eliminate the orderliness in ISTs for interchangeable entities. Extensive experiments on four datasets verify the effectiveness of ISTA-Net by outperforming state-of-the-art methods. Our code is publicly available at https://github.com/Necolizer/ISTA-Net.

Planning and prediction are two important modules of autonomous driving and have experienced tremendous advancement recently. Nevertheless, most existing methods regard planning and prediction as independent and ignore the correlation between them, leading to the lack of consideration for interaction and dynamic changes of traffic scenarios. To address this challenge, we propose InteractionNet, which leverages transformer to share global contextual reasoning among all traffic participants to capture interaction and interconnect planning and prediction to achieve joint. Besides, InteractionNet deploys another transformer to help the model pay extra attention to the perceived region containing critical or unseen vehicles. InteractionNet outperforms other baselines in several benchmarks, especially in terms of safety, which benefits from the joint consideration of planning and forecasting. The code will be available at https://github.com/fujiawei0724/InteractionNet.

Understanding and learning the actor-to-X inter-actions (AXIs), such as those between the focal vehicles (actor) and other traffic participants (e.g., other vehicles, pedestrians) as well as traffic environments (e.g., city/road map), is essential for the development of a decision-making model and simulation of autonomous driving (AD). Existing practices on imitation learning (IL) for AD simulation, despite the advances in the model learnability, have not accounted for fusing and differentiating the heterogeneous AXIs in complex road environments. Furthermore, how to further explain the hierarchical structures within the complex AXIs remains largely under-explored. To overcome these challenges, we propose HGIL, an interaction- aware and hierarchically-explainable Heterogeneous _Graph- based Imitation Learning approach for AD simulation. We have designed a novel heterogeneous interaction graph (HIG) to provide local and global representation as well as awareness of the AXIs. Integrating the HIG as the state embeddings, we have designed a hierarchically-explainable generative adversarial imitation learning approach, with local sub-graph and global cross-graph attention, to capture the interaction behaviors and driving decision-making processes. Our data-driven simulation and explanation studies have corroborated the accuracy and explainability of HGIL in learning and capturing the complex AXIs.

Understanding human interaction with objects is an important research topic for embodied Artificial Intelligence and identifying the objects that humans are interacting with is a primary problem for interaction understanding. Existing methods rely on frame-based detectors to locate interacting objects. However, this approach is subjected to heavy occlusions, background clutter, and distracting objects. To address the limitations, in this paper, we propose to leverage spatio-temporal information of hand-object interaction to track interactive objects under these challenging cases. Without prior knowledge of the general objects to be tracked like object tracking problems, we first utilize the spatial relation between hands and objects to adaptively discover the interacting objects from the scene. Second, the consistency and continuity of the appearance of objects between successive frames are exploited to track the objects. With this tracking formulation, our method also benefits from training on large-scale general object-tracking datasets. We further curate a video-level hand-object interaction dataset for testing and evaluation from 100DOH. The quantitative results demonstrate that our proposed method outperforms the state-of-the-art methods. Specifically, in scenes with continuous interaction with different objects, we achieve an impressive improvement of about 10% as evaluated using the Average Precision (AP) metric. Our qualitative findings also illustrate that our method can produce more continuous trajectories for interacting objects.

Soft-Continuum Manipulators are of increasing interest to researchers for various non-destructive applications (minimally invasive surgery, fibroscopy, oncology, pipe exploration and many others). They are made with soft material or special arrangement of actuators allowing them to exhibit resilience and dexterity. The concept of Proprioceptive Soft-Continuum Manipulators still remains a major challenge for soft roboticists due to the big issues related to manufacturing process which becomes very expensive including sophisticated or experimental tools, highly skilled technicians and time. However, the manipulator proprioception is very usefull for enhancing the dexterity during their manipulation. Henceforth, this paper investigates a quick and simple approach for the integrated design of a proprioceptive Soft Robotic Bio- Inspired Trunk made with dragon skin 30 Material. This soft manipulator is made up of two segments composed of three independent physical control inputs each. It has an embedded electronics mainly composed of IMUs. The latter have allowed controlling the shape kinematics using a control-oriented modeling approach inspired from the kinematics control of a puppet toy. The developed modeling approach is a Reduced Order Modeling (ROM) which uses Pythagorean Hodograph (PH) curves which lowers in real time, the control dimension of the robot to virtual control points of its representative PH curve. The proposed investigation presents also a comprehensive approach for the manufacturing process of Soft-Continuum Manipulators with complex geometry.

Complex multibody legged robots can have complex rotational control challenges. In this paper, we propose a concise way to understand and formulate a whole-body orientation that (i) depends on system configuration only and not a history of motion, (ii) can be representative of the orientation of the entire system while not being attached to any specific link, and (iii) has a rate of change that approximates total system angular momentum. We relate this orientation coordinate to past work, and discuss and demonstrate, including on hardware, several different uses for it.

Fabricating existing and popular open-source adaptive robotic grippers commonly involves using multiple professional machines, purchasing a wide range of parts, and tedious, time-consuming assembly processes. This poses a significant barrier to entry for some robotics researchers and drives others to opt for expensive commercial alternatives. To provide both parties with an easier and cheaper (under £100) solution, we propose a novel adaptive gripper design where every component (with the exception of actuators and the screws that come packaged with them) can be fabricated on a hobby-grade 3D printer, via a combination of inexpensive and readily available PLA and TPU filaments. This approach means that the gripper's tendons, flexure joints and finger pads are now printed, as a replacement for traditional string-tendons and molded urethane flexures / pads. A push-fit systems results in an assembly time of under 10 minutes. The gripper design is also highly modular and requires only a few minutes to replace any part, leading to extremely user-friendly maintenance and part modifications. An extensive stress test has shown a level of durability more than suitable for research, whilst grasping experiments (with perturbations) using items from the YCB object set has also proven its mechanical adaptability to be highly satisfactory.

In recent years, the aging of society has become a serious problem, especially in developed countries. Walking is an important element in extending healthy life expectancy in old age. In particular, induction of proper ankle joint alignment at heel contact is important during the gait cycle from the perspective of smooth weight transfer and reduction of burden on the knees and hip. In this study, we focus on the behavior of the ankle joint at heel contact and propose an insole-type assist device that can induce the ankle angle inversion/eversion rotation. The proposed device has tilting of the heel part from left to right in response to the rotation of a stepping motor, and an inertial sensor mounted inside controls the heel part to always maintain a horizontal position. The effectiveness of the proposed device is verified by evaluating the amount of lateral thrust of the knee joint of six healthy male subjects during a foot-stepping motion using motion capture system. The results showed that the amount of lateral thrust is significantly reduced by wearing the device with control.

Miniature Magnetic Rotating Swimmers (MMRSs) are untethered machines containing magnetic materials. An external rotating magnetic field produces a torque on the swimmers to make them rotate. MMRSs have propeller fins that convert the rotating motion into forward propulsion. This type of robot has been shown to have potential applications in the medical realm. This paper presents new MMRS designs with (1) an increased permanent magnet volume to increase the available torque and prevent the MMRS from becoming stuck inside a thrombus; (2) new helix designs that produce an increased force to compensate for the weight added by the larger permanent magnet volume; (3) different head drill shape designs that have different interactions with thrombi. The two best MMRS designs were tested experimentally by removing a partially dried 1-hour-old thrombus with flow in a bifurcating artery model. The first MMRS disrupted a large portion of the thrombus. The second MMRS retrieved a small remaining piece of the thrombus. In addition, a tool for inserting, retrieving, and switching MMRSs during an experiment is presented and demonstrated. Finally, this paper shows that the two selected MMRS designs can perform accurate 3D path-following.

Identifying moving objects is a crucial capability for autonomous navigation, consistent map generation, and future trajectory prediction of objects. In this paper, we propose a novel network that addresses the challenge of segmenting moving objects in 3D LiDAR scans. Our approach not only predicts point-wise moving labels but also detects instance information of main traffic participants. Such a design helps determine which instances are actually moving and which ones are temporarily static in the current scene. Our method exploits a sequence of point clouds as input and quantifies them into 4D voxels. We use 4D sparse convolutions to extract motion features from the 4D voxels and inject them into the current scan. Then, we extract spatio-temporal features from the current scan for instance detection and feature fusion. Finally, we design an upsample fusion module to output point-wise labels by fusing the spatio-temporal features and predicted instance information. We evaluated our approach on the LiDAR-MOS benchmark based on SemanticKITTI and achieved better moving object segmentation performance compared to state-of-the-art methods, demonstrating the effectiveness of our approach in integrating instance information for moving object segmentation. Furthermore, our method shows superior performance on the Apollo dataset with a pre-trained model on SemanticKITTI, indicating that our method generalizes well in different scenes. The code and pre-trained models of our method will be released at https://github.com/nubot-nudt/InsMOS.

Multi-human multi-robot teams have great potential for complex and large-scale tasks through the collaboration of humans and robots with diverse capabilities and expertise. To efficiently operate such highly heterogeneous teams and maximize team performance timely, sophisticated initial task allocation strategies that consider individual differences across team members and tasks are required. While existing works have shown promising results in reallocating tasks based on agent state and performance, the neglect of the inherent heterogeneity of the team hinders their effectiveness in realistic scenarios. In this paper, we present a novel formulation of the initial task allocation problem in multi-human multi-robot teams as a contextual multi-attribute decision-make process and propose an attention-based deep reinforcement learning approach. We introduce a cross-attribute attention module to encode the latent and complex dependencies of multiple attributes in the state representation. We conduct a case study in a massive threat surveillance scenario and demonstrate the strengths of our model.

It is easier to program effective robots when they inhabit highly structured environments. The growing literature on methods to aid robot design has given comparatively little consideration to elements external to the robot itself, yet such elements can encode or enhance information (to improve perception), can alter the effects or costs of actions (to help control), and can provide regularity by imposing constraints. External elements have the potential to be shared, to scale elastically, and to spread both benefits and installation/operating costs. These are traits of infrastructure in support of robots. We introduce a basic but flexible mathematical model –via the MDP framework– for rational evaluation of proposed additions and changes to environments, including where infrastructure may improve precision or performance of either perception or actuation. Through it, one can assess the numbers of agents needed for infrastructure investment to be economical, determine when installation costs would be recouped, and evaluate the effect of behavior changes as responses to environmental modifications. To demonstrate how the model can be instantiated, four simple but practical case studies are presented.

The proliferation of unmanned vehicles offers many opportunities for solving environmental sampling tasks with applications in resource monitoring and precision agriculture. Informative path planning (IPP) includes a family of methods which offer improvements over traditional surveying techniques for suggesting locations for observation collection. In this work, we present a novel solution to the IPP problem by using a coregionalized Gaussian processes to estimate a dynamic scalar field that varies in space and time. Our method improves previous approaches by using a composite kernel accounting for spatiotemporal correlations and at the same time, can be readily incorporated in existing IPP algorithms. Through extensive simulations, we show that our novel modeling approach leads to more accurate estimations when compared with formerly proposed methods that do not account for the temporal dimension.

Magnetic nanoparticles can be electrostatically assembled around sperm cells to form biohybrid micro robots. These biohybrid microrobots possess sufficient magnetic material to potentially allow for pulse-echo localization and wireless actuation. Alternatively, magnetic excitation of these nanoparticles can be used for localization based on Faraday's law of induction using a detection coil. Here, we investigate the influence of the electrostatic attraction between positively charged nanoparticles and negatively charged sperm cells on the activation of the nanoparticles during nonlinear differential magnetometry and wireless magnetic actuation. Activation of clusters of free nanoparticles and nanoparticles bound to the body of sperm cells is achieved by a combination of a high- frequency alternating field and a pulsating static field. The nonlinear response in both cases indicates that constraining the nanoparticles is likely to yield significant decreases in the magnetometry sensitivity. While the attachment of particles to the cells enables wireless actuation (rolling locomotion), the rate of change of the magnetization of the nanoparticles decreases one order of magnitude compared to free nanoparticles.

We have been investigating a crawling-like loco-motion robot to make it efficiently slide forward based on a simple system and control mechanisms on a slippery level surface, where the motion of the center of mass plays an important role. In this paper, we induce an effective motion of the center of mass considering a streamlined body shape of a locomotion robot in which a pendulum is installed. First, we derive the equation of motion and a control input to achieve a desired motion of the inner pendulum. Second, we formulate constraint conditions between the streamlined body and a slippery floor. Third, we demonstrate the numerical simulation, and the robot steadily slides forward by adopting the streamlined shape as the body frame. Fourth, we verify the numerical results through experiments, and the experimental results exhibit a similar tendency compared with the numerical results. Fifth, we find a local minimum value of locomotion efficiency based on Bayesian optimization which is a class of machine-learning-based optimization, and we achieve exceedingly efficient locomotion of the robot on the slippery floor at the local minimum in both the simulation and experiment.

Modeling a human-driven vehicle is a difficult subject since human drivers have a variety of stochastic behavioral components that influence their driving styles. We develop a cooperative driving framework to incorporate dif-ferent human behavior aspects, including the attentiveness of a driver and the tendency of the driver following advising commands. To demonstrate the framework, we consider the merging coordination between a human-driven vehicle and an autonomous vehicle (AV) in a connected environment. We propose a stochastic model predictive controller (sMPC) to address the stochasticity in human driving behavior and design coordinated merging actions to optimize the AV input and influence human driving behavior through advising commands. Simulation and human-in-the-loop (HITL) experimental results show that our formulation is capable of accommodating a distracted driver and optimizing AV inputs based on human driving behavior recognition.

Incipient slip is defined as the slippage of part, but not all, of the contact surface between a sensor and an object. Reliably detecting incipient slip in artificial tactile sensors would benefit autonomous robot handling capabilities by helping prevent object slippage during manipulation. Here, we present a biomimetic skin morphology based on the human fingerprint with application to marker-based tactile sensors such as the TacTip biomimetic optical tactile sensor. We modify the 3D-printed outer membrane of the TacTip to mimic glabrous skin morphology with the inclusion of external ridges (fingerprint) and internal markers (intermediate ridges), allowing localised shear deformation of the sensor's skin prior to the onset of gross slip. To validate the performance of this skin morphology, we train a random forest classifier (RFC) to identify incipient slip based on the extracted marker displacements from the sensor when it is compressed against an acrylic plate and moved laterally. The RFC model achieves 97.46% accuracy on incipient slip prediction, and is then validated on an unseen pouring task, in which gravity-induced incipient slip is detected on average within 418\pm 753\ \text{ms}418\pm 753\ \text{ms} of its onset, and before gross slip in all trials. This accurate detection of incipient slip enables corrective actions prior to the onset of gross slip, a key capability in robotic manipulation and upper-limb prosthetics.

Measuring geometry of the printing road is key for detection of anomalies in 3D printing processes. Although commercial 3D printers can measure the extrusion height using various distance sensors, measuring of the width in real-time remains a challenge. This paper presents a visual in-situ monitoring system to measure width of the printing filament road in 2D patterns. The proposed system is composed of a printable shroud with embedded camera setup and a visual detection approach based on a two-stage instance segmentation method. Each of the segmentation and localization stages can use multiple computational approaches including Gaussian mixture model, color filter, and deep neural network models. The visual monitoring system is mounted on a standard 3D printer and validated with the measurement of printed filament roads of sub-millimeter widths. The results on accuracy and robustness reveal that combinations of deep models for both segmentation and localization stages have better performance. Particularly, fully connected CNN segmentation model combined with YOLO object detector can measure sub-millimeter extrusion width with 90 μm accuracy at 125 ms speed. This visual monitoring system has potential to improve the control of printing processes by the real-time measurement of printed filament geometry.