IROS 2023 Papers

While federated learning (FL) improves the generalization of end-to-end autonomous driving by model aggregation, the conventional single-hop FL (SFL) suffers from slow convergence rate due to long-range communications among vehicles and cloud server. Hierarchical federated learning (HFL) overcomes such drawbacks via introduction of mid-point edge servers. However, the orchestration between constrained communication resources and HFL performance becomes an urgent problem. This paper proposes an optimization-based Communication Resource Constrained Hierarchical Federated Learning (CRCHFL) framework to minimize the generalization error of the autonomous driving model using hybrid data and model aggregation. The effectiveness of the proposed CRCHFL is evaluated in the Car Learning to Act (CARLA) simulation platform. Results show that the proposed CRCHFL both accelerates the convergence rate and enhances the generalization of federated learning autonomous driving model. Moreover, under the same communication resource budget, it outperforms the HFL by 10.33% and the SFL by 12.44%.

Human-Robot collaboration in home and industrial workspaces is on the rise. However, the communication between robots and humans is a bottleneck. Although people use a combination of different types of gestures to complement speech, only a few robotic systems utilize gestures for communication. In this paper, we propose a gesture pseudo-language and show how multiple types of gestures can be combined to express human intent to a robot (i.e., expressing both the desired action and its parameters - e.g., pointing to an object and showing that the object should be emptied into a bowl). The demonstrated gestures and the perceived tabletop scene (object poses detected by CosyPose) are processed in real-time) to extract the human's intent. We utilize behavior trees to generate reactive robot behavior that handles various possible states of the world (e.g., a drawer has to be opened before an object is placed into it) and recovers from errors (e.g., when the scene changes). Furthermore, our system enables switching between direct teleoperation of the end-effector and high-level operation using the proposed gesture sentences. The system is evaluated on increasingly complex tasks using a real 7-DoF Franka Emika Panda manipulator. Controlling the robot via action gestures lowered the execution time by up to 60%, compared to direct teleoperation.

In this paper, we present a method to engage measurement uncertainties with the probabilistic robustness to one system uncertainty measure. Providing a metric indicating the potential occurrence of dangerous situations is highly essential for safety-critical robot applications. Due to the difficulty of finding a quantifiable, unambiguous representation however, such a metric has not been derived to date. In case of sensory devices, measurement uncertainties are usually provided by manufacturer specifications. Apart from that, several contributions demonstrate that the accuracy of neural networks is verifiable via the robustness. However, state-of-the-art literature is mainly concerned with theoretical investigations such that scarce attention has been devoted to the transfer of the robustness to real-world applications. To fill this gap, we show how the probabilistic robustness can be made useful for evaluating quantitative safety limits. Our key idea is to exploit the analogy between measurement uncertainties and the probabilistic robustness: While measurement uncertainties reflect possible shifts due to technical limitations, the robustness refers to the tolerated amount of distortions in the input data for an unaltered output. Inspired by this analogy, we combine both measures to quantify the system uncertainty online. We validate our method in different settings under real-world conditions. Our findings exemplify that incorporating the novel uncertainty metric effectively prevents the rate of dangerous situations in Human-Robot Collaboration.

We present a new approach for modeling avoidance constraints in 2D environments, in which waypoints are assigned to obstacle-free polyhedral regions. Constraints of this form are often formulated as mixed-integer programming (MIP) problems employing big-M techniques-however, these are generally not the strongest formulations possible with respect to the MIP's convex relaxation (so called ideal formulations), potentially resulting in larger computational burden. We instead model obstacle avoidance as combinatorial disjunctive constraints and leverage the independent branching scheme to construct small, ideal formulations. As our approach requires a biclique cover for an associated graph, we exploit the structure of this class of graphs to develop a fast subroutine for obtaining biclique covers in polynomial time. We also contribute an open-source Julia library named ClutteredEnvPathOpt to facilitate computational experiments of MIP formulations for obstacle avoidance. Experiments have shown our formulation is more compact and remains competitive on a number of instances compared with standard big-M techniques, for which solvers possess highly optimized procedures.

We introduce a multi-modal reconfiguration planner for the Variable Topology Truss (VTT) modular robot system. The VTT system is a truss-architecture modular self-reconfigurable robot. When a VTT is restricted to a single topology, the collision constraints between the truss members divide the configuration space into many connected components, which makes collision-free planning difficult. This new planner leverages a mathematical invariant based on link theory to find topological reconfiguration actions that can connect these different regions and make progress towards a goal. We show that this planner is effective at finding paths between configurations with different truss topologies.

This paper proposes a collision prevention strategy for indoor mobile robots that utilizes a low-cost two-dimensional (2D) range scanner. The use of indoor mobile robots for versatile tasks, such as serving, cleaning, and delivery, has been increasing. As these robots navigate in environments with dynamic obstacles of arbitrary sizes and shapes that move at irregular speeds and directions, sophisticated motion planning is essential to ensure safe and successful task completion. Previous works have relied on object-wise detection and tracking methods that demand significant computing resources and rich information. However, these approaches are challenging to apply for commercial robots that require light computation and guaranteed safety. In this paper, we propose a novel and straightforward strategy that predicts the behavior of dynamic obstacles using agglomerated dynamic points-based trajectory envelope estimation (ADTE), instead of the conventional object-wise approach. We also introduce the stop trigger as a practical and safe strategy that uses the predicted information. The stop trigger operates independently of the primary motion planning module to isolate prediction uncertainty. The proposed strategy was verified through robot testing and simulation with standardized test scenarios defined in ISO 18646–2. The robot testing was officially conducted by an authorized organization. The results demonstrate the effectiveness of the proposed strategy in preventing collisions with moving obstacles.

Parallel robots (PRs) allow for higher speeds in human-robot collaboration due to their lower moving masses but are more prone to unintended contact. For a safe reaction, knowledge of the location and force of a collision is useful. A novel algorithm for collision isolation and identification with proprioceptive information for a real PR is the scope of this work. To classify the collided body, the effects of contact forces at the links and platform of the PR are analyzed using a kinetostatic projection. This insight enables the derivation of features from the line of action of the estimated external force. The significance of these features is confirmed in experiments for various load cases. A feedforward neural network (FNN) classifies the collided body based on these physically modeled features. Generalization with the FNN to 300k load cases on the whole robot structure in other joint angle configurations is successfully performed with a collision-body classification accuracy of 84% in the experiments. Platform collisions are isolated and identified with an explicit solution, while a particle filter estimates the location and force of a contact on a kinematic chain. Updating the particle filter with estimated external joint torques leads to an isolation error of less than 3 cm and an identification error of 4 N in a real-world experiment.

Solving complex problems collectively with simple entities is a challenging task for swarm robotics. For the task of collective decision-making, robots decide based on local observations on the microscopic level to achieve consensus on the macroscopic level. We study this problem for a common benchmark of classifying distributed features in a binary dynamic environment. Our special focus is on environmental features that are dynamic as they change during the experiment. We present a control algorithm that uses sophisticated statistical change detection in combination with Bayesian robots to classify dynamic environments. The main profit is to reduce false positives allowing for improved speed and accuracy in decision-making. Supported by results from various simulated experiments, we introduce three feedback loops to balance speed and accuracy. In our benchmarks, we show the superiority of our new approach over previous works on Bayesian robots. Our approach of using change detection shows a more reliable detection of environmental changes. This enables the swarm to successfully classify even difficult environments (i.e., hard to detect differences between the binary features), while achieving faster and more accurate results in simpler environments.

Cooperative object transportation using multiple robots has been intensively studied in the control and robotics literature, but most approaches are either only applicable to omnidirectional robots or lack a complete navigation and decision-making framework that operates in real time. This paper presents an autonomous nonholonomic multi-robot system and an end-to-end hierarchical autonomy framework for collaborative luggage trolley transportation. This framework finds kinematic-feasible paths, computes online motion plans, and provides feedback that enables the multi-robot system to handle long lines of luggage trolleys and navigate obstacles and pedestrians while dealing with multiple inherently complex and coupled constraints. We demonstrate the designed collaborative trolley transportation system through practical transportation tasks, and the experiment results reveal their effectiveness and reliability in complex and dynamic environments. (Video11Video demonstration: https://youtu.be/efnPERm0Rco.)

In an efficient and flexible human-robot collaborative work environment, a robot team member must be able to recognize both explicit requests and implied actions from human users. Identifying “what to do” in such cases requires an agent to have the ability to construct associations between objects, their actions, and the effect of actions on the environment. In this regard, semantic memory is being introduced to understand the explicit cues and their relationships with available objects and required skills to make “tea” and “sandwich”. We have extended our previous hierarchical robot control architecture to add the capability to execute the most appropriate task based on both feedback from the user and the environmental context. To validate this system, two types of skills were implemented in the hierarchical task tree: 1) Tea making skills and 2) Sandwich making skills. During the conversation between the robot and the human, the robot was able to determine the hidden context using ontology and began to act accordingly. For instance, if the person says “I am thirsty” or “It is cold outside” the robot will start to perform the tea-making skill. In contrast, if the person says, “I am hungry” or “I need something to eat”, the robot will make the sandwich. A humanoid robot Baxter was used for this experiment. We tested three scenarios with objects at different positions on the table for each skill. We observed that in all cases, the robot used only objects that were relevant to the skill.

Distributed Data Fusion (DDF), as a prevalent technique that empowers scalable, flexible, and robust information fusing, has been employed in various multi-sensor networks operating in uncertain and dynamic environments. This paper proposes a cognition difference-based mechanism to construct a dynamic trust network for real-time DDF, where the cognition difference is defined as the statistical difference between the sensors' estimated probability distributions. Distinguished by the mutual correlation between trust and cognition difference, two principles of determining trust are investigated, and their performances are analyzed by conducting simulations in the scenarios of source seeking. Our simulation and experiment results show that the proposed approach is effective in providing comprehensive and robust performance in general and unstructured environments.

Swarm drones flying is a very attractive field of robotics research, motivated by natural bird flocking or other animal collective behaviors. In this paper, we propose and develop an open-source11https://github.com/micros-uav/CoFlyers universal platform CoFlyers for end-to-end whole-chain development from flocking-inspired models to real-drone swarm flying. In particular, CoFlyers is more user-friendly with only a unified programming language of MATLAB&Simulink, rather than several existing platforms with mixed programming languages or more efforts on raw functional modules. The prototype simulator of CoFlyers is implemented in MATLAB, allowing users to quickly develop and prototype swarm flying algorithms, and to conduct task-oriented parameter auto-tuning and batch processing within reproducible scenarios. Moreover, a real-world verification module of swarm drones is developed in Simulink as well, which directly calls the prototype simulator modules for code reuse. It connects the external platforms via a standardized user-datagram-protocol communication in-terface. As a case study, CoFlyers is utilized into a multi-drone collective flying scenario in confined environments, by implementing ROS&PX4&Gazebo for high-fidelity simulation and Optitrack&Tello-drones for experiments. Eventually, both simulation and experimental results have demonstrated and validated the user-friendly practicability of CoFlyers.

Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motions. We proposed a two-stage model to address this uncertainty issue in gesture synthesis by modeling the gesture segments as discrete latent codes. Our method utilizes RQ- VAE in the first stage to learn a discrete codebook consisting of gesture tokens from training data. In the second stage, a two-level autoregressive transformer model is used to learn the prior distribution of residual codes conditioned on input speech context. Since the inference is formulated as token sampling, multiple gesture sequences could be generated given the same speech input using top-k sampling. The quantitative results and the user study showed the proposed method outperforms the previous methods and is able to generate realistic and diverse gesture motions.

Maintaining balance during gait in the mediolateral direction requires more active motor control than in the anteroposterior direction. Step width modulation is a key strategy used by healthy individuals to achieve mediolateral walking balance, but it can be disrupted in populations with poor sensorimotor integration and weak hip abductors, such as the elderly, stroke patients, and people with lower limb amputation. Wearable hip exoskeletons have the potential to serve as assistive or rehabilitation devices for these populations, but there has been limited research on their appropriate usage. In this study, we successfully demonstrated the feasibility of controlling step width using a mediolaterally acting robotic hip exoskeleton. We were able to effectively adjust the user's step width by increasing or decreasing it to predefined targets through the regulation of admittance control parameters governing the device. The maximum average error to increase or decrease the step width was 1.2 cm. This research has the potential to facilitate the development of assistive and rehabilitation applications focused on enhancing the mediolateral gait balance of individuals with neurological impairments, elderly individuals, and amputees via the control of step width.

In this paper, we present a new approach to the control of continuum robot sections using IMU quaternion feedback. We use a discrete time root finding algorithm to steer a continuum section in the desired shape space direction. We found that the approach lacks end effector positioning accuracy when used by itself, however, when used in conjunction with a feedforward model it actively counters the influence of unmodeled factors. The approach is implemented on a single section of a continuum hose robot developed for 3D printing of concrete in construction applications. The results demonstrate significant improvements in positioning accuracy compared to standalone kinematics/mechanics-based position control of tendon lengths. Additionally, this approach can be implemented using low cost sensing and control hardware.

This work presents CineTransfer, an algorithmic framework that drives a robot to record a video sequence that mimics the cinematographic style of an input video. We propose features that abstract the aesthetic style of the input video, so the robot can transfer this style to a scene with visual details that are significantly different from the input video. The framework builds upon CineMPC, a tool that allows users to control cinematographic features, like subjects' position on the image and the depth of field, by manipulating the intrinsics and extrinsics of a cinematographic camera. However, CineMPC requires a human expert to specify the desired style of the shot (composition, camera motion, zoom, focus, etc). CineTransfer bridges this gap, aiming a fully autonomous cinematographic platform. The user chooses a single input video as a style guide. CineTransfer extracts and optimizes two important style features, the composition of the subject in the image and the scene depth of field, and provides instructions for CineMPC to control the robot to record an output sequence that matches these features as closely as possible. In contrast with other style transfer methods, our approach is a lightweight and portable framework which does not require deep network training or extensive datasets. Experiments with real and simulated videos demonstrate the system's ability to analyze and transfer style between recordings, and are available in the supplementary video11https://youtu.be/_QzNz5WUtpk

Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability. The project website can be found at https://matcha-model.github.io/.

Two distinct technologies have gained attention lately due to their prospects for motor rehabilitation: robotics and brain-machine interfaces (BMIs). Harnessing their combined efforts is a largely uncharted and promising direction that has immense clinical potential. However, a significant challenge is whether motor intentions from the user can be accurately detected using non-invasive BMIs in the presence of instrumental noise and passive movements induced by the rehabilitation exoskeleton. As an alternative to the straight-forward continuous control approach, this study instead aims to characterize the onset and offset of motor imagery during passive arm movements induced by an upper-body exoskeleton to allow for the natural control (initiation and termination) of functional movements. Ten participants were recruited to perform kinesthetic motor imagery (MI) of the right arm while attached to the robot, simultaneously cued with LEDs indicating the initiation and termination of a goal-oriented reaching task. Using electroencephalogram signals, we built a decoder to detect the transition between i) rest and beginning MI and ii) maintaining and ending MI. Offline decoder evaluation achieved group average onset accuracy of 60.7% and 66.6% for offset accuracy, revealing that the start and stop of MI could be identified while attached to the robot. Furthermore, pseudo-online evaluation could replicate this performance, forecasting reliable online exoskeleton control in the future. Our approach showed that participants could produce quality and reliable sensorimotor rhythms regardless of noise or passive arm movements induced by wearing the exoskeleton, which opens new possibilities for BMI control of assistive devices.

Characteristics of wireless manipulation based on permanent magnet coupling, including anchoring distance, panning torque, and translational force, are assessed in this paper. The study focuses on a typical scenario where a slave robot embedded with a small permanent magnet can be remotely controlled within a constrained area by a master large permanent magnet placed outside the environment at a certain distance from it. The key parameters (force and torque) acting on the slave robot are quantified and evaluated. In this article, several combinations of permanent magnets with various dimensions and configurations are studied using finite element methods. Based on the obtained results, we create a lookup table for each parameter, serving as a guideline to help interested researchers choose suitable magnetic combinations for their applications.