ICRA 2024 Papers

Rehabilitative robotics, particularly lower-limb exoskeletons (LLEs), have gained increasing importance in aiding patients regain ambulatory functions. One of the challenges in making these systems effective is the implementation of an assist-as-needed (AAN) control strategy that intervenes only when the patient deviates from the correct movement pattern. Equally crucial is the need for the LLE to exhibit "transparency" — minimising its interaction forces with the wearer to feel as natural as possible. This paper introduces a novel approach to transparency control based on a human-in-the-loop velocity optimisation framework. The proposed method employs torque data captured from past steps through a Series Elastic Actuator (SEA) to approximate the wearer’s intended future movements and computes a corresponding transparent velocity trajectory. The velocity commands are complemented by an Adaptive Frequency Oscillator (AFO) based position controller that leverages the periodic nature of human gait and is modified with a force sensor for increased reactiveness to human gait variations. This approach is experimentally evaluated against a standard zero-torque controller with a stationary single-degree-of-freedom knee exoskeleton test platform in a proof-of-concept study. Preliminary results indicate that combining adaptive oscillators with interaction force sensing can improve transparency compared to the conventional zero-torque controller, using force readings for position control and torque measurements for velocity optimisation and control.

Robots performing human-scale manipulation tasks require an extensive amount of knowledge about their surroundings in order to perform their actions competently and human-like. In this work, we investigate the use of virtual reality technology as an implementation for robot environment modeling, and present a technique for translating scene graphs into knowledge bases. To this end, we take advantage of the Universal Scene Description (USD) format which is an emerging standard for the authoring, visualization and simulation of complex environments. We investigate the conversion of USD-based environment models into Knowledge Graph (KG) representations that facilitate semantic querying and integration with additional knowledge sources. The contributions of the paper are validated through an application scenario in the service robotics domain.

The popular methods for semi-supervised semantic segmentation mostly adopt a unitary network model using convolutional neural networks (CNNs) and enforce consistency of the model’s predictions over perturbations applied to the inputs or model. However, such a learning paradigm suffers from two critical limitations: a) learning the discriminative features for the unlabeled data; b) learning both global and local information from the whole image. In this paper, we propose a novel Semi-supervised Learning (SSL) approach, called Transformer-CNN Cohort (TCC), that consists of two students with one based on the vision transformer (ViT) and the other based on the CNN. Our method subtly incorporates the multi-level consistency regularization on the predictions and the heterogeneous feature spaces via pseudo-labeling for the unlabeled data. First, as the inputs of the ViT student are image patches, the feature maps extracted encode crucial class-wise statistics. To this end, we propose class-aware feature consistency distillation (CFCD) that first leverages the outputs of each student as the pseudo labels and generates class-aware feature (CF) maps for knowledge transfer between the two students. Second, as the ViT student has more uniform representations for all layers, we propose consistency-aware cross distillation (CCD) to transfer knowledge between the pixel-wise predictions from the cohort. We validate the TCC framework on Cityscapes and Pascal VOC 2012 datasets, which outperforms existing SSL methods by a large margin. Project page: https://vlislab22.github.io/TCC/.

In this paper, we propose a transformer-based architecture for predicting contact forces during a physical human-robot interaction. Our Neural Network is composed of two main parts: a Multi-Layer Perceptron called Transducer and a Transformer. The former estimates, based on the kinematic data from a motion capture suit, the current contact forces. The latter predicts – taking as input the same kinematic data and the output of the Transducer – the human motions and the contact forces over a time window in the future. We validated our approach by testing the network on directions of motions that were not provided in the training set. We also compared our approach to a purely Transformer-based network, showing a better prediction accuracy of the contact forces.

Tracking dynamic targets poses a significant challenge for Unmanned Ground Vehicles (UGVs). Existing methods often lack research on multi-disturbed conditions. To address this issue, we propose a trajectory-prediction-based dynamic tracking scheme, which includes target localization, trajectory prediction, and UGV control. Firstly, an estimation algorithm based on the Extended Kalman Filter (EKF) is employed to mitigate noise and estimate the absolute states of the target accurately. To enhance robustness, we present an Adaptive Trajectory Prediction (ATP) algorithm based on prediction anchors. In this method, a quantization standard for trajectory disturbance is designed for adaptive control. Subsequently, we iteratively solve prediction anchor points based on two motion models to robustly predict the target trajectory even in the presence of unknown disturbances. Finally, the Linear Time-Varying Model Predictive Control (LTV-MPC) is utilized in the UGV controller for dynamic tracking. Experimental results demonstrate that the ATP exhibits superior prediction robustness and accuracy in perturbed environments compared to other prediction algorithms. In addition, the proposed scheme effectively achieves dynamic tracking of the Unmanned Aerial Vehicle (UAV) by the UGV under multi-disturbed conditions. Specifically, when the target moves at a speed of 1.0 m/s, the UGV can maintain a tracking error within 0.346 m.

We consider the problem of tracking a reference trajectory for dynamical systems subject to a priori unknown state-dependent disturbance behavior. We propose a formulation that embeds the uncertain system into a higher dimensional deterministic system that accounts for worst case disturbances. Our main insight is that a single controlled trajectory of this embedding system corresponds to a controlled forward invariant interval tube around the reference trajectory. By taking observations of the system, we then propose to estimate the state-dependent uncertainty with Gaussian Process regression, which improves the accuracy of the forward invariant tube as data is collected. Given a safety objective, we also provide conditions on when an additional observation of the unknown disturbance behavior needs to be collected to maintain safety. We demonstrate our formulation on a case study of a planar multirotor attempting a safe landing in an unknown wind field.

Predicting pedestrian movements remains a complex and persistent challenge in robot navigation research. We must evaluate several factors to achieve accurate predictions, such as pedestrian interactions, the environment, crowd density, and social and cultural norms. Accurate prediction of pedestrian paths is vital for ensuring safe human-robot interaction, especially in robot navigation. Furthermore, this research has potential applications in autonomous vehicles, pedestrian tracking, and human-robot collaboration. Therefore, in this paper, we introduce FlowMNO, an Optical Flow-Integrated Markov Neural Operator designed to capture pedestrian behavior across diverse scenarios. Our paper models trajectory prediction as a Markovian process, where future pedestrian coordinates depend solely on the current state. This problem formulation eliminates the need to store previous states. We conducted experiments using standard benchmark datasets like ETH, HOTEL, ZARA1, ZARA2, UCY, and RGB-D pedestrian datasets. Our study demonstrates that FlowMNO outperforms some of the state-of-the-art deep learning methods like LSTM, GAN, and CNN-based approaches, by approximately 86.46% when predicting pedestrian trajectories. Thus, we show that FlowMNO can seamlessly integrate into robot navigation systems, enhancing their ability to navigate crowded areas smoothly.

In this paper, an Active Cooperative Localization system for Quadrotor Unmanned Aerial Vehicles is developed. The optimal trajectories are determined by minimizing the uncertainty in position estimation by Extended Kalman Filter. In this system, a piecewise polynomial parameterization of trajectories is adopted for the optimizer, and the underlying state estimator is updated with appropriate models of sensors and quadrotor dynamics. This system is verified in extensive simulations in the scenario of a team of quadrotors with heterogeneous GNSS capabilities. These simulations answer an open question, showing that solving for trajectories by minimizing Kalman covariance computed in a noiseless environment is reasonable and that the optimized trajectories offer visible reductions in positioning uncertainty in the presence of noise.

We propose a limb trajectory planning method that considers both body and limb dynamics in robots, particularly suitable for those with non-trivial limb mass. To simplify the complexity and computation cost of using the full-body dynamics of the limbs, a reduced-order model that can simulate the dynamic characteristics of the original limb is proposed. The performance of the model is experimentally validated using an exemplary single leg-wheel of the leg-wheel transformable robot. The limb trajectory optimization is developed using a genetic algorithm that considers many aspects, including body and limb dynamics, limb workspace, limb motion continuity, body tip-over stability, and power consumption. The performance of the proposed limb trajectory planning strategy is experimentally validated using the same leg-wheel transformable robot, and the results confirm the effectiveness of the strategy.

Robotic systems with hundreds or even thousands of robots are widely implemented in logistic and industrial applications. In such systems, cooperative path planning is of great importance, as local congestion and motion conflict may greatly degrade system performance, especially in the presence of uncertainties. Our idea is to consider traffic flow equilibrium in path planning to relieve any potential congestion and increase efficiency. In this paper, we propose a hierarchical framework, which includes a traffic flow prediction layer, a sector-level planning layer, and a road-level coordination layer. In traffic flow prediction, we propose a spatio-temporal graph neural network that integrates local information to predict the evolution of future robot density distribution. In sector-level planning, we generate sector-level paths that consider travel distance and traffic flow equilibrium simultaneously. In road-level coordination, we implement the conflict-based search algorithm within each sector to ensure conflict-free local paths. In addition, we also explicitly consider motion/communication uncertainties that are unavoided in practical systems. We validate our effectiveness in simulations with over 1000 robots, what’s more, real experiments are provided.

Significant progress has been made in training multimodal trajectory forecasting models for autonomous driving. However, effectively integrating these models with downstream planners and model-based control approaches is still an open problem. Although these models have conventionally been evaluated for open-loop prediction, we show that they can be used to parameterize autoregressive closed-loop models without retraining. We consider recent trajectory prediction approaches which leverage learned anchor embeddings to predict multiple trajectories, finding that these anchor embeddings can parameterize discrete and distinct modes representing high-level driving behaviors. We propose to perform fully reactive closed-loop planning over these discrete latent modes, allowing us to tractably model the causal interactions between agents at each step. We validate our approach on a suite of more dynamic merging scenarios, finding that our approach avoids the frozen robot problem which is pervasive in conventional planners. Our approach also outperforms the previous state-of-the-art in CARLA on challenging dense traffic scenarios when evaluated at realistic speeds.

Robot navigation within complex environments requires precise state estimation and localization to ensure robust and safe operations. For ambulating mobile robots like robot snakes, traditional methods for sensing require multiple embedded sensors or markers, leading to increased complexity, cost, and increased points of failure. Alternatively, deploying an external camera in the environment is very easy to do, and marker-less state estimation of the robot from this camera’s images is an ideal solution: both simple and cost-effective. However, the challenge in this process is in tracking the robot under larger environments where the cameras may be moved around without extrinsic calibration, or maybe when in motion (e.g., a drone following the robot). The scenario itself presents a complex challenge: single-image reconstruction of robot poses under noisy observations. In this paper, we address the problem of tracking ambulatory mobile robots from a single camera. The method combines differentiable rendering with the Kalman filter. This synergy allows for simultaneous estimation of the robot’s joint angle and pose while also providing state uncertainty which could be used later on for robust control. We demonstrate the efficacy of our approach on a snake-like robot in both stationary and non-stationary (moving) cameras, validating its performance in both structured and unstructured scenarios. The results achieved show an average error of 0.05 m in localizing the robot’s base position and 6 degrees in joint state estimation. We believe this novel technique opens up possibilities for enhanced robot mobility and navigation in future exploratory and search-and-rescue missions.

Machine Learning (ML) has replaced handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world’s first real-world autonomous driving dataset and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to make safe and efficient decisions. We introduce a new large-scale dataset that consists of 1282 hours of diverse driving scenarios from 4 cities (Las Vegas, Boston, Pittsburgh, and Singapore) and includes high-quality auto-labeled object tracks and traffic light data. We mine and taxonomize common & rare driving scenarios which are used during evaluation to get fine-grained insights into the performance and characteristics of a planner. Beyond the dataset, we provide a simulation and evaluation framework that enables a planner’s actions to be simulated in closed-loop to account for interactions with other traffic participants. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods. Find the nuPlan dataset and code at nuplan.org.

Modern algorithms allow robots to reach a greater level of autonomy and fulfill more challenging tasks. However, on-board limitations regarding computational and battery resources are hindering factors regarding the deployment of such algorithms particularly on mobile robots. Although offloading a majority of the algorithmic components to the edge or even cloud offers an attractive option to leverage massive computing power in robotics applications, safety and reliability remain critical issues. This paper presents a minimalistic safety fallback mechanism when offloading mobile robot navigation to the edge, that ensures safe and collision-free navigation even in the presence of failures in the connection between the on-board and edge-devices. We show the effectiveness of our approach through extensive testing in three different relevant scenarios in a simulated warehouse environment. Our experiments demonstrate the effects of different fallback strategies and show how our proposed approach is able to ensure safety while allowing the robot to continue its mission during an interrupted connection and thus avoiding unnecessary downtime.

Research in powered prosthesis control has explored the use of impedance-based control algorithms due to their biomimetic capabilities and intuitive structure. Modern impedance controllers feature parameters that smoothly vary over gait phase and task according to a data-driven model. However, these recent efforts only use continuous impedance control during stance and instead utilize discrete transition logic to switch to kinematic control during swing, necessitating two separate models for the different parts of the stride. In contrast, this paper presents a controller that uses smooth impedance parameter trajectories throughout the gait, unifying the stance and swing periods under a single, continuous model. Furthermore, this paper proposes a basis model to represent inter-task relationships in the impedance parameters—a strategy that has previously been shown to improve model accuracy over classic linear interpolation methods. In the proposed controller, a weighted sum of Fourier series is used to model the impedance parameters of each joint as continuous functions of gait cycle progression and task. Fourier series coefficients are determined via convex optimization such that the controller best reproduces the joint torques and kinematics in a reference able-bodied dataset. Experiments with a powered knee-ankle prosthesis show that this simpler, unified model produces competitive results when compared to a more complex hybrid impedance-kinematic model over varying walking speeds and inclines.

In single-incision laparoscopic surgery (SILS), magnetic anchoring and guidance system (MAGS) is a promising technique to prevent clutter in the surgical workspace and provide a larger vision field. Existing camera designs mainly rely on rigid structure design, resulting in risks of losing magnetic coupling and impacting tissue during the insertion and coupling procedure. In this paper, we proposed a wireless MAGS consisting of soft material and structure design. The camera can bend at the exit of the trocar and maintain strong coupling with the external actuator. The operation principle and modeling were established to investigate the parameter design. An easier insertion procedure was introduced and demonstrated in the experiment. The bendability was tested showing the camera could reach 20° in bending angle and 16.4mm in displacement. The insertion and deployment took less than 2 minutes on average.

Point clouds play a crucial role in robots and intelligent vehicles. Noise caused by fog droplets seriously degrades the quality of point clouds. Previous researches have shown that the extent of degradation is correlated with visibility. The fog attenuation coefficient is associated with visibility. In light of this background, this paper proposes a noise-distribution-based defogging method for point clouds. Our approach hinges on the estimation of the fog attenuation coefficient, facilitated by road-based prior knowledge. Subsequently, our method integrates the fog-induced noise distribution inferred from the LiDAR imaging model with the spatially non-uniform distribution of point clouds caused by LiDAR structure. The fused results are input to a statistical filter based on the relative sparsity of noise to achieve defogging. This paper is one of the early works focusing on point cloud defogging. Its core insight lies in the estimation of the attenuation coefficient and the employment of fog-induced noise distribution for defogging. Experiments demonstrate that our method can accurately mitigate the impact of fog and meanwhile enhance the performance of 3D object detection network.

Generating human-like robot motions is pivotal for achieving smooth human-robot interactions. Such motions contribute to better predictions of robot motions by humans, thus leading to more intuitive interaction and increased acceptability. Human likeness in robot motions has been conventionally measured and realized via the optimization of human-likeness metrics. However, the abundance of such metrics and the absence of standardized criteria impede their usage in novel contexts. In this work, we introduce a unified human-likeness metric built from a hierarchically weighted sum of individual metrics. The proposed metric is derived from a thorough analysis of eleven existing human-likeness criteria and is applicable across various tasks and robot models. We evaluate its performance in the context of motion retargeting of bimanual tasks with three different humanoid robots.

Interactive visual grounding in Human-Robot Interaction (HRI) is challenging yet practical due to the inevitable ambiguity in natural languages. It requires robots to disambiguate the user’s input by active information gathering. Previous approaches often rely on predefined templates to ask disambiguation questions, resulting in performance reduction in realistic interactive scenarios. In this paper, we propose TiO, an end-to-end system for interactive visual grounding in human-robot interaction. Benefiting from a unified formulation of visual dialog and grounding, our method can be trained on a joint of extensive public data, and show superior generality to diversified and challenging open-world scenarios. In the experiments, we validate TiO on GuessWhat?! and InViG benchmarks, setting new state-of-the-art performance by a clear margin. Moreover, we conduct HRI experiments on the carefully selected 150 challenging scenes as well as real-robot platforms. Results show that our method demonstrates superior generality to diversified visual and language inputs with a high success rate. Codes and demos are available on https://jxu124.github.io/TiO/.