RSS 2025 Papers

Environment prediction frameworks are critical for the safe navigation of autonomous vehicles (AVs) in dynamic settings. LiDAR-generated occupancy grid maps (L-OGMs) offer a robust bird’s-eye view scene representation, enabling self-supervised joint scene predictions while exhibiting resilience to partial observability and perception detection failures. Prior approaches have focused on deterministic L-OGM prediction architectures within the grid cell space. While these methods have seen some success, they frequently produce unrealistic predictions and fail to capture the stochastic nature of the environment. Additionally, they do not effectively integrate additional sensor modalities present in AVs. Our proposed framework, Latent Occupancy Prediction (LOPR), performs stochastic L-OGM prediction within the latent space of a generative architecture and allows for conditioning on RGB cameras, maps, and planned trajectories. We decode predictions using either a single-step decoder, which provides high-quality predictions in real-time, or a diffusion-based batch decoder, which can further refine the decoded frames to address temporal consistency issues and reduce compression losses. Our experiments on the nuScenes and Waymo Open datasets show that all variants of our approach qualitatively and quantitatively outperform prior approaches.

Recent advances in safe autonomy open new opportunities in assisting humans in safety-critical and time-sensitive tasks such as motorsports. However, existing safe control algorithms predominantly focus on fully automated settings and often undermine key requirements in human–AI shared control domains—maintaining human agency and respecting their decisions. Preserving human agency is crucial in human–AI collaboration to reduce the risks of “automation surprise”, where human operators may feel confused, uncomfortable, or even irritated by the AI’s assistance. In this work, we propose Human-Centered Safety Filter (HCSF), a principled shared autonomy framework that improves robustness in human decision-making while minimally affecting human agency. HCSF is based on a novel model-free safety formulation that allows the AI “assistant” to step in gradually, rather than at the last possible moment, to ensure that AI interventions feel smooth and are only as large as necessary. In contrast to conventional safety filters that prioritize safety without taking into account human decisions, HCSF minimally modifies human actions and actively promotes the transparency of AI assistance. We evaluate HCSF with Assetto Corsa, a high-fidelity car racing simulator, where human drivers are assisted by an AI assistant to race against an in-game opponent. Extensive human trials demonstrate that HCSF enhances safety without compromising the driver’s agency, leading to improved user confidence, comfort, and overall satisfaction.

For robots to become efficient helpers in the home, they must learn to perform new mobile manipulation tasks simply by watching humans perform them. Learning from a single video demonstration from a human is challenging as the robot needs to first extract from the demo what needs to be done and how, translate the strategy from a third to a first-person perspective, and then adapt it to be successful with its own morphology. Furthermore, to mitigate the dependency on costly human monitoring, this learning process should be performed in a safe and autonomous manner. We present SafeMimic, a framework to learn new mobile manipulation skills safely and autonomously from a single third-person human video. Given an initial human video demonstration of a multi-step mobile manipulation task, SafeMimic first parses the video into segments, inferring both the semantic changes caused and the motions the human executed to achieve them and translating them to an egocentric reference. Then, it adapts the behavior to the robot’s own morphology by sampling candidate actions around the human ones, and verifying them for safety before execution in a receding horizon fashion using an ensemble of safety value functions trained in simulation. When safe forward progression is not possible, SafeMimic backtracks to previous states and attempts a different sequence of actions, adapting both the trajectory and the grasping modes when required for its morphology. As a result, SafeMimic yields a strategy that succeeds in the demonstrated behavior and learns task-specific grasps that reduce exploration in future attempts. Our experiments show that our method allows robots to safely and efficiently learn multi-step mobile manipulation behaviors from a single human demonstration, from different users, and in different environments, with improvements over state-of-the-art baselines across seven tasks.

A common problem when using model predictive control (MPC) in practice is the satisfaction of safety beyond the prediction horizon. While theoretical works have shown that safety can be guaranteed by enforcing a suitable terminal set constraint or a sufficiently long prediction horizon, these techniques are difficult to apply and thus rarely used by practitioners, especially in the case of general nonlinear dynamics. To solve this, we make a tradeoff between exact recursive feasibility, computational tractability, and applicability to black-box dynamics by learning an approximate discrete-time control barrier function and incorporating it into variational inference MPC (VIMPC), a sampling-based MPC paradigm. To handle the resulting state constraints, we further propose a new sampling strategy that greatly reduces the variance of the estimated optimal control, improving the sample efficiency and enabling real-time planning on CPU. The resulting Neural Shield-VIMPC (NS-VIMPC) controller yields substantial safety improvements compared to existing sampling-based MPC controllers, even under badly designed cost functions. We validate our approach in both simulation and real-world hardware experiments.

Learning visuomotor policy from human demonstrations serves as an effective method for robots to acquire complex tasks. However, data collection on mobile platforms such as drones is extremely challenging, resulting in most research being conducted with robots in stationary conditions for data collection and policy training. Nonetheless, these policies struggle to maintain operational stability when the robot is in motion, primarily due to the low frequency of decision-making, inconsistencies in the input data distribution and differences in the motion characteristics of different robots. To address these issues, we propose a system named STDArm, which effectively transfers stationary-trained action strategies to dynamic states, including the robot’s ego-motion or wobble caused by structural factors. Specifically, we first design an action manager to efficiently store and manage the sequence of actions asynchronously output by the policy network, and to stably output actions at a higher control frequency as needed. Next, we predict the future state of the robotic arm’s base by observing its historical states, thereby developing an end-effector stabilizer to counteract the robot’s movements. Additionally, we incorporate a warm-up step during the robot’s initialization phase to estimate the parameters used in the system online, ensuring tight integration between the stabilizer and policy modules. We conduct comprehensive evaluations of the proposed system on two types of robotic arms, two types of mobile platforms, and three tasks. Experimental results indicate that the STDArm system significantly improves task execution success rates during the robot’s movement.

Real-world tasks such as garment manipulation and table rearrangement demand robots to perform generalizable, highly precise, and long-horizon actions. Although imitation learning has proven to be an effective approach for teaching robots new skills, large amounts of expert demonstration data are still indispensible for these complex tasks, resulting in high sample complexity and costly data collection. To address this, we propose Semantic Keypoint Imitation Learning (SKIL), a framework which automatically obtain semantic keypoints with help of vision foundation models, and forms the descriptor of semantic keypoints that enables effecient imitation learning of complex robotic tasks with significantly lower sample complexity. In real world experiments, SKIL doubles the performance of baseline methods in tasks such as picking a cup or mouse, while demonstrating exceptional robustness to variations in objects, environmental changes, and distractors. For long-horizon tasks like hanging a towel on a rack where previous methods fail completely, SKIL achieves a mean success rate of 70% with as few as 30 demonstrations. Furthermore, SKIL naturally supports cross-embodiment learning due to its semantic keypoints abstraction, our experiments demonstrate that even human videos bring considerable improvement to the learning performance. All these results demonstrate the great success of SKIL in achieving data-efficint generalizable robotic learning.

Despite recent advances in learning-based controllers for legged robots, deployments in human-centric environments remain limited by safety concerns. Most of these approaches use position-based control, where policies output target joint angles that must be processed by a low-level controller (e.g., PD or impedance controllers) to compute joint torques. Although impressive results have been achieved in controlled real-world scenarios, these methods often struggle with compliance and adaptability when encountering environments or disturbances unseen during training, potentially resulting in extreme or unsafe behaviors. Inspired by how animals achieve smooth and adaptive movements by controlling muscle extension and contraction, torque-based policies offer a promising alternative by enabling precise and direct control of the actuators in torque space. This approach facilitates more effective interactions with the environment, resulting in safer and more adaptable behaviors. However, challenges such as a highly nonlinear state space and inefficient exploration during training have hindered their broader adoption. To address these limitations, we propose SATA, a bio-inspired framework that mimics key biomechanical principles and adaptive learning mechanisms observed in animal locomotion. Our approach effectively addresses the inherent challenges of learning torque-based policies by significantly improving early-stage exploration, leading to high-performance final policies. Remarkably, our method achieves zero-shot sim-to-real transfer, eliminating the need for additional fine-tuning on hardware. Our experimental results indicate that SATA demonstrates remarkable compliance and safety, even in challenging environments such as soft/slippery terrain or narrow passages, and under significant external disturbances (e.g., pushing/pulling/pressing on the robot, or manually moving individual legs). These results highlight its potential for practical deployments in human-centric and safety-critical scenarios.

Robust and adaptive robotic peg-in-hole assembly under tight tolerance is critical to various industrial applications. Still, it remains an open challenge due to perception and physical uncertainties from contact-rich interactions that easily exceed the allowed clearance. In this paper, we study how to leverage the contact between the peg and its matching hole to eliminate uncertainties in the assembly process under unstructured settings. By exploring the role of compliance under contact constraints, we present a manipulation system that plans collision-inclusive interactions for the peg to 1) iteratively identify its task environment to localize the target hole and 2) exploit environmental contact constraints to refine insertion motions into the target hole without precise perception, facilitating a robust solution to peg-in-hole assembly. By conceptualizing the above process as the composition of funneling in different state spaces, we present a formal approach to construct manipulation funnels as an uncertainty-absorbing paradigm for peg-in-hole assembly. The proposed system effectively generalizes diverse peg-in-hole scenarios across varying scales, shapes, and materials in a learning-free manner. Extensive real-world experiments on a NIST Assembly Task Board validate its robustness in real-world applications.

Scaling robot learning requires data collection pipelines that scale favorably with human effort. In this work, we propose

The performance of imitation learning policies often hinges on the datasets with which they are trained. Consequently, investment in data collection for robotics has grown across both industrial and academic labs. However, despite the marked increase in the quantity of demonstrations collected, little work has sought to assess the quality of said data despite mounting evidence of its importance in other areas such as vision and language. In this work, we take a critical step towards addressing the data quality in robotics. Given a dataset of demonstrations, we aim to estimate the relative quality of individual demonstrations in terms of both state diversity and action predictability. To do so, we estimate the average contribution of a trajectory towards the mutual information between states and actions in the entire dataset, which precisely captures both the entropy of the state distribution and the state-conditioned entropy of actions. Though commonly used mutual information estimators require vast amounts of data often beyond the scale available in robotics, we introduce a novel technique based on k-nearest neighbor estimates of mutual information on top of simple VAE embeddings of states and actions. Empirically, we demonstrate that our approach is able to partition demonstration datasets by quality according to human expert scores across a diverse set of benchmarks spanning simulation and real world environments.

Data scaling and standardized evaluation benchmarks have driven remarkable advances in natural language processing and computer vision. However, in robotics, scaling up data and establishing evaluation protocols pose significant challenges. Directly collecting real-world data is inefficient and resource-intensive, while benchmarking in real-world scenarios also remains highly challenging. Synthetic data and simulation environments present a promising alternative, yet existing efforts often fail to fully leverage the potential of simulation, resulting in limited data quality, diversity, and fragmented benchmarks. To address these challenges, we introduce RoboVerse, a simulation platform built on a unified data format. RoboVerse supports multiple simulators and robots, enabling seamless switching between different simulators and embodiments. In addition, by leveraging our unified data format, it enables multiple workflows to efficiently collect tasks and trajectories from various sources with high fidelity and diversity. Using RoboVerse, we generate the largest high-quality synthetic dataset to date in a unified format, along with a standardized benchmark system that reliably evaluates policies and supports assessment across different levels of generalization. We employ the RoboVerse workflows and conduct extensive experiments, demonstrating that our platform and dataset significantly enhance the performance of imitation learning, reinforcement learning, and world model learning, facilitating the transfer from simulation to real-world applications. These results demonstrate the reliability of our dataset and benchmark, highlighting RoboVerse as an effective solution for advancing simulation-assisted robot learning.

Developing robust and general-purpose manipulation policies is a key goal in robotics. To achieve effective generalization, it is essential to construct comprehensive datasets that encompass a large number of demonstration trajectories and diverse tasks. Unlike vision or language data, which can be sourced from the internet, robotic datasets require detailed observations and manipulation actions, necessitating significant investments in both hardware-software infrastructure and human labor. While existing works have focused on assembling various individual robot datasets, there is still a lack of a unified data collection standard and insufficient high-quality data across diverse tasks, scenarios, and robot types. In this paper, we introduce RoboMIND (Multi-embodiment Intelligence Normative Data for Robot Manipulation), a dataset containing 107k demonstration trajectories across 479 diverse tasks involving 96 object classes. RoboMIND is collected through human teleoperation and encompasses comprehensive robotic-related information, including multi-view observations, proprioceptive robot state information, and linguistic task descriptions. To ensure dataset consistency and reliability for imitation learning, RoboMIND is built on a unified data collection platform and standardized protocol, covering four distinct robotic embodiments: the Franka Emika Panda, the UR-5e, the AgileX dual-arm robot, and a humanoid robot with dual dexterous hands. Our dataset also includes 5k real-world failure demonstrations, each accompanied by detailed causes, enabling failure reflection and correction during policy learning. Additionally, we create a digital twin environment in the Isaac Sim simulator, replicating the real-world tasks and assets, which facilitates the low-cost collection of additional training data and enables efficient evaluation. To demonstrate the quality and diversity of our dataset, we conducted extensive experiments using various Imitation Learning methods for single task setting and state-of-the-art Vision-Language-Action (VLA) models for multi-task scenarios. By leveraging RoboMIND, the VLA models achieved high manipulation success rates and demonstrated strong generalization capabilities.

Designing dynamically feasible trajectories for rigid bodies is a fundamental problem in robotics. Although direct trajectory optimization is widely applied to solve this problem, state-of-the-art methods overlook the manifold structures of rigid bodies, resulting in slow convergence. This paper introduces a Riemannian optimization framework for direct trajectory optimization of rigid bodies on matrix Lie groups. The proposed approach first leverages the Lie Group Variational Integrator to formulate the discrete rigid body dynamics. The paper then derives the exact first- and second-order Riemannian derivatives of the dynamics using the matrix Lie group structure. Finally, this work applies a line-search Riemannian Interior Point Method (RIPM) to perform trajectory optimization. The paper demonstrates that both the derivative evaluations and Newton steps required to solve the RIPM exhibit linear complexity with respect to the planning horizon and system degrees of freedom. Simulation results illustrate that the proposed method is an order of magnitude faster than conventional methods.

Preventing collisions in multi-robot navigation is crucial for deployment. This requirement hinders the use of learning-based approaches, such as multi-agent reinforcement learning (MARL), on their own due to their lack of safety guarantees. Traditional control methods, such as reachability and control barrier functions, can provide rigorous safety guarantees when interactions are limited only to a small number of robots. However, conflicts between the constraints faced by different agents pose a challenge to safe multi-agent coordination.

Humans can accomplish complex contact-rich tasks using vision and touch, with highly reactive capabilities such as quick adjustments to environmental changes and adaptive control of contact forces; however, this remains challenging for robots. Existing visual imitation learning (IL) approaches rely on action chunking to model complex behaviors, which lacks the ability to respond instantly to real-time tactile feedback during the chunk execution. Furthermore, most teleoperation systems struggle to provide fine-grained tactile/force feedback, which limits the range of tasks that can be performed. To address these challenges, we introduce TactAR, a low-cost teleoperation system that provides real-time tactile feedback through Augmented Reality (AR), along with Reactive Diffusion Policy (RDP), a novel slow-fast visual-tactile imitation learning algorithm for learning contact-rich manipulation skills. RDP employs a two-level hierarchy: (1) a slow latent diffusion policy for predicting high-level action chunks in latent space at low frequency, (2) a fast asymmetric tokenizer for closed-loop tactile feedback control at high frequency. This design enables both complex trajectory modeling and quick reactive behavior within a unified framework. Through extensive evaluation across three challenging contact-rich tasks, our approach demonstrates superior performance compared to state-of-the-art visual IL baselines while maintaining fast reactivity to tactile feedback. Furthermore, experiments show that RDP is applicable across different tactile / force sensors. More videos and results can be found in the supplementary files.

Dexterous manipulation is a fundamental capability for robotic systems to interact with the physical world, yet progress in this area has been limited by hardware. An ideal robotic hand must balance precision, compactness, strength, and affordability—requirements that remain challenging to achieve simultaneously. Existing hand designs impose trade-offs based on available control methods and target applications. However, learning-based approaches present an opportunity to rethink some of these trade-offs, particularly to address challenges associated with tendon-driven actuation and low-cost materials. In this work, we present RUKA, a tendon-driven humanoid hand that is simple, affordable, and capable. Made from 3D-printed parts and off-the-shelf components, RUKA has 5 fingers (including an opposable thumb) and underactuated control of 18 degrees of freedom, enabling diverse human-like grasps. Its tendon-driven actuation allows powerful grasping in a compact, human-sized form. To tackle tendon-driven control challenges, we learn joint-to-actuator and fingertip-to-actuator models using motion-capture data, leveraging the hand’s morphological accuracy. We extensively evaluate RUKA against commonly used robotic hands and demonstrate its superior reachability, durability, and strength. We further apply RUKA in bimanual teleoperation tasks and showcase that RUKA can be used to perform various dexterous tasks. By addressing important trade-offs in robotic hand design, we believe RUKA opens new possibilities for advancing manipulation research and expanding its accessibility. All code, data, design models, and assembly instructions are open-source and available at our website.

Global localization is a fundamental capability required for long-term and drift-free robot navigation. However, current methods fail to relocalize when faced with significantly different viewpoints. We present ROMAN (Robust Object Map Alignment Anywhere), a robust global localization method capable of localizing in challenging and diverse environments based on creating and aligning maps of open-set and view-invariant objects. To address localization difficulties caused by feature-sparse or perceptually aliased environments, ROMAN formulates and solves a registration problem between object submaps using a unified graph-theoretic global data association approach that simultaneously accounts for object shape and semantic similarities and a prior on gravity direction. Through a set of challenging large-scale multi-robot or multi-session SLAM experiments in indoor, urban, and unstructured/forested environments, we demonstrate that ROMAN achieves higher accuracy in terms of object map alignment than other registration methods and lower trajectory estimation errors than SLAM systems that use visual features for loop closures. Our code will be made open-source.

Recent advances in robotic foundation models have enabled the development of generalist policies that can adapt to diverse tasks. While these models show impressive flexibility, their performance heavily depends on the quality of their training data. In this work, we propose Reinforcement Learning Distilled Generalists (RLDG), a method that leverages reinforcement learning to generate high-quality training data for fine-tuning generalist policies. Through extensive real-world experiments on precise manipulation tasks like connector insertion and assembly, we demonstrate that generalist policies trained with RL-generated data consistently outperform those trained with human demonstrations, achieving up to 40% higher success rates while generalizing better to new tasks. We also provide a detailed analysis that reveals this performance gain stems from both optimized action distributions and improved state coverage. Our results suggest that combining task-specific RL with generalist policy distillation offers a promising approach for developing more capable and efficient robotic manipulation systems that maintain the flexibility of foundation models while achieving the performance of specialized controllers.