ICRA 2024 Papers

Artificial cognitive architectures traditionally rely on complex memory models to encode, store, and retrieve information. However, the conventional practice of transferring all data from working memory (WM) to long-term memory (LTM) leads to high data volumes and challenges in efficient information processing and access. Deciding what information to retain or discard within a robot’s LTM is particularly challenging since knowledge about future data utilization is absent. Drawing inspiration from human forgetting this paper implements and evaluates novel forgetting techniques that allow consolidation in the robot’s LTM only when new information is encountered. The proposed approach combines fast filtering during data transfer to the robot’s LTM with slower yet more precise forgetting mechanisms that are periodically evaluated for offline data deletion inside the LTM. We compare different mechanisms, utilizing metrics such as data similarity, data age, and consolidation frequency. The efficacy of forgetting techniques is evaluated by comparing their performance in a task where two ARMAR robots search through their LTM for past object locations in episodic ego-centric images and robot state data. Experimental results show that our forgetting techniques significantly reduce the space requirements of a robot’s LTM while maintaining its capacity to successfully perform tasks relying on LTM information. Notably, similarity-based forgetting methods outperform frequency- and time-based approaches. The combination of online frequency-based, online similarity-based, offline similarity-based, and time-based decay methods shows superior performance compared to using individual forgetting strategies.

We present ForceSight, a system for text-guided mobile manipulation that predicts visual-force goals using a text-conditioned vision transformer. Given a single RGBD image and a text prompt, ForceSight determines a target end-effector pose in the camera frame (kinematic goal) and the associated forces (force goal). Together, these two components form a visual-force goal. Prior work has demonstrated that deep models outputting human-interpretable kinematic goals can enable dexterous manipulation by real robots. Forces are critical to manipulation, yet have typically been relegated to low-level execution in these systems. When deployed on a mobile manipulator equipped with an eye-in-hand RGBD camera, ForceSight performed tasks such as precision grasps, drawer opening, and object handovers with an 81% success rate in unseen environments with object instances that differed significantly from the training data. In a separate experiment, relying exclusively on visual servoing and ignoring force goals dropped the success rate from 90% to 45%, demonstrating that force goals can significantly enhance performance. The appendix, videos, code, and trained models are available at https://force-sight.github.io/.

This work demonstrates the utility of dual-arm robots with dual-wrist force-torque sensors in manipulating a Deformable Linear Object (DLO) within an unknown environment that imposes constraints on the DLO’s movement through contacts and fixtures. We propose a strategy to estimate the pose of unknown environmental contacts encountered during the manipulation of a DLO, classifying the induced constraints as unilateral, bilateral and fully constrained, exploiting the redundancy of force sensors. A semantic approach to define environmental constraints is introduced and incorporated into a graph-based model of the DLO. This model remains accurate as long as the DLO is under tension and is dynamically updated throughout the manipulation process, built by sequencing a set of primitives. The estimation strategy is validated through simulations and real-world experiments, demonstrating its potential in handling DLOs under various, possibly uncertain, constraints.

Nonlinear model-predictive control has recently shown its practicability in robotics. However it remains limited in contact interaction tasks due to its inability to leverage sensed efforts. In this work, we propose a novel model-predictive control approach that incorporates direct feedback from force sensors while circumventing explicit modeling of the contact force evolution. Our approach is based on the online estimation of the discrepancy between the force predicted by the dynamics model and force measurements, combined with high-frequency nonlinear model-predictive control. We report an experimental validation on a torque-controlled manipulator in challenging tasks for which accurate force tracking is necessary. We show that a simple reformulation of the optimal control problem combined with standard estimation tools enables to achieve state-of-the-art performance in force control while preserving the benefits of model-predictive control, thereby outperforming traditional force control techniques. This work paves the way toward a more systematic integration of force sensors in model predictive control.

Soft continuum robots enable new application areas in contrast to standard rigid robots, such as interaction with a varying environment. Due to their compliant continuous structure, they are inherently safe and adaptive to environmental conditions. In this paper, the interaction with the environment is performed at the tool-center-point of a soft continuum manipulator and is realized by a hybrid force-position control. For this, a force estimation model is derived to substitute the force sensor at the tool-center-point. The force estimation is probabilistic and relies on normal distributions considering model parameters and deviations from model identification of the soft continuum robot. It also provides a qualitative measure for the contact estimation. This paper first presents the probabilistic force estimation model and then shows the hybrid force-position control using the presented model. From the results, it is concluded that force sensing is replaceable for the environment interaction.

Legged robots have demonstrated high efficiency and effectiveness in unstructured and dynamic environments. However, it is still challenging for legged robots to achieve rapid and efficient locomotion on deformable, yielding substrates, such as granular terrains. We present an enhanced resistive force model for bipedal walkers on soft granular terrains by introducing effective intrusion depth correction. The enhanced force model captures fundamental kinetic results considering the robot foot shape, walking gait speed variation, and energy expense. The model is validated by extensive foot intrusion experiments with a bipedal robot. The results confirm the model accuracy on the given type of granular terrains. The model can be further integrated with the motion control of bipedal robotic walkers.

In this study, we address the off-road traversability estimation problem, that predicts areas where a robot can navigate in off-road environments. An off-road environment is an unstructured environment comprising a combination of traversable and non-traversable spaces, which presents a challenge for estimating traversability. This study highlights three primary factors that affect a robot’s traversability in an off-road environment: surface slope, semantic information, and robot platform. We present two strategies for estimating traversability, using a guide filter network (GFN) and footprint supervision module (FSM). The first strategy involves building a novel GFN using a newly designed guide filter layer. The GFN interprets the surface and semantic information from the input data and integrates them to extract features optimized for traversability estimation. The second strategy involves developing an FSM, which is a self-supervision module that utilizes the path traversed by the robot in pre-driving, also known as a footprint. This enables the prediction of traversability that reflects the characteristics of the robot platform. Based on these two strategies, the proposed method overcomes the limitations of existing methods, which require laborious human supervision and lack scalability. Extensive experiments in diverse conditions, including automobiles and unmanned ground vehicles, herbfields, woodlands, and farmlands, demonstrate that the proposed method is compatible for various robot platforms and adaptable to a range of terrains. Code is available at https://github.com/yurimjeon1892/FtFoot.

In Cloud Robotics, long system latency due to varying network conditions can cause instability and collisions. However, this can be minimized in the almost univeral case where there are multiple sources available for cloud servers. By extending anycast routing, we introduce FogROS2-Latency-Sensitive, a Fog Robotics framework that offers secure, location-independent connections between robots and latency-sensitive cloud-based servers. FogROS2-LS offloads conventional on-board state estimators and feedback controllers to Cloud and Edge compute hardware without modifying existing applications in ROS2. In the presence of multiple identical services, FogROS2-LS dynamically identifies and transitions to the optimal service deployment that meets latency requirements, thereby empowering robots with limited on-board computing capacity to safely and efficiently navigate dynamic, human-dense environments. We evaluate FogROS2-LS with two latency sensitive case studies: (1) Collision Avoidance: a robot arm guided by visual feedback from consistent distance estimation and collision checking on Cloud and Edge. FogROS2-LS reduces collision failures by up to 8.5x by selecting the best available server, and (2) Target Tracking: FogROS2-LS enables robust and continuous target following and can recover from network failures. Videos and code are available on the website https://sites.google.com/view/fogros2-ls.

Cloud service providers provide over 50,000 distinct and dynamically changing set of cloud server options. To help roboticists make cost-effective decisions, we present FogROS2-Config, an open toolkit that takes ROS2 nodes as input and automatically runs relevant benchmarks to quickly return a menu of cloud compute services that tradeoff latency and cost. Because it is infeasible to try every hardware configuration, FogROS2-Config quickly samples tests a small set of edge-case servers. We evaluate FogROS2-Config on three robotics application tasks: visual SLAM, grasp planning. and motion planning. FogROS2-Config can reduce the cost by up to 20x. By comparing with a Pareto frontier for cost and latency by running the application task on feasible server configurations, we evaluate cost and latency models and confirm that FogROS2-Config selects efficient hardware configurations to balance cost and latency. Videos and code are available on the website https://sites.google.com/view/fogros2-config

Multi-object tracking (MOT) presents a crucial challenge in robotics. Due to limited resources embedded in robots, one time step per processing time for algorithms can be considerably large. This scenario necessitates the operation of MOT at a low frame rate. However, algorithms within the MOT research field have been constructed around datasets functioning at 10–30 frames per second (fps) which can be difficult to operate in the limited resources. In response to it, we introduce a new algorithm, called FocoTrack, which maintains tracking ability in four situations, one of which is when objects are overlapped by each other. Our algorithm exhibits remarkable performance without using any deep appearance descriptor, surpassing existing MOT methods which even use the deep appearance descriptor on a 2.5 fps dataset. We also demonstrate strong results with our algorithm on DanceTrack dataset at 20 fps and provide comprehensive insights through detailed analysis of our tracking model.

Understanding airflow around a drone is critical for performing advanced maneuvers while maintaining flight stability. Recent research has worked to understand this flow by employing 2D and 3D flow sensors to measure flow from a single source like wind or the drone’s relative motion. Our current work advances flow detection by introducing a strategy to distinguish between two flow sources applied simultaneously from different directions. By densely packing an array of flow sensors (or whiskers), we alter the path of airflow as it moves through the array. We have named this technique “flow shadowing” because we take advantage of the fact that a downstream whisker shadowed (or occluded) by an upstream whisker receives less incident flow. We show that this relationship is predictable for two whiskers based on the percent of occlusion. We then show that a 2x2 spatial array of whiskers responds asymmetrically when multiple flow sources from different headings are applied to the array. This asymmetry is direction-dependent, allowing us to predict the headings of flow from two different sources, like wind and a drone’s relative motion.

We present distributed distance-based control (DDC), a novel approach for controlling a multi-agent system, such that it achieves a desired formation, in a resource-constrained setting. Our controller is fully distributed and only requires local state-estimation and scalar measurements of inter-agent distances. It does not require an external localization system or inter-agent exchange of state information. Our approach uses spatial-predictive control (SPC), to optimize a cost function given strictly in terms of inter-agent distances and the distance to the target location. In DDC, each agent continuously learns and updates a very abstract model of the actual system, in the form of a dictionary of three independent key-value pairs (\Delta \vec s,\Delta d)(\Delta \vec s,\Delta d), where ∆d is the partial derivative of the distance measurements along a spatial direction \Delta \vec s\Delta \vec s. This is sufficient for an agent to choose the best next action. We validate our approach by using DDC to control a collection of Crazyflie drones to achieve formation flight and reach a target while maintaining flock formation.

To achieve high-dexterity motion planning on floating-base systems, the base dynamics induced by arm motions must be treated carefully. In general, it is a significant challenge to establish a fixed-base frame during tasking due to forces and torques on the base that arise directly from arm motions (e.g. arm drag in low Reynolds environments and arm momentum in high Reynolds environments). While thrusters can in theory be used to regulate the vehicle pose, it is often insufficient to establish a stable pose for precise tasking, whether the cause be due to underactuation, modeling inaccuracy, suboptimal control parameters, or insufficient power. We propose a solution that asks the thrusters to do less high bandwidth perturbation correction by planning arm motions that induce zero perturbation on the base. We are able to cast our motion planner as a nonholonomic rapidly-exploring random tree (RRT) by representing the floating-base dynamics as pfaffian constraints on joint velocity. These constraints guide the manipulators to move on zero-perturbation manifolds (which inhabit a subspace of the tangent space of the internal configuration space). To invoke this representation (termed a perturbation map) we assume the body velocity (perturbation) of the base to be a joint-defined linear mapping of joint velocity and describe situations where this assumption is realistic (including underwater, aerial, and orbital environments). The core insight of this work is that when perturbation of the floating-base has affine structure with respect to joint velocity, it provides the system a class of kinematic reduction that permits the use of sample-based motion planners (specifically a nonholonomic RRT). We show that this allows rapid, exploration-geared motion planning for high degree of freedom systems in obstacle rich environments, even on floating-base systems with nontrivial dynamics.

This work validates through flight tests a previously developed wide-envelope singularity-free aerodynamic framework, called ϕ-theory, for modeling dual-engine tail-sitting flying-wing vehicles for optimization-based control. The ϕ-theory methodology imposes a specific geometry on aerodynamic coefficients that leads to polynomial differential equations of motion amenable to semidefinite programming optimization. Through ϕ-theory, we illustrate a typical predicted longitudinal and lateral flight envelope of a tail-sitting vehicle, which, while commonplace for fixed-wing aircraft in performance textbooks, is a novel figure that generalizes fixed-wing doghouse plots to tail-sitting vehicles. This flight envelope figure suggests a novel, natural and intuitive remote piloting interface that we validate in flight tests. Furthermore, we further validate ϕ-theory through the computation of flight features in simulation and their subsequent observation in flight tests.

A support structure for flexible displays such as OLED or flexible LEDs was developed using the flexible omnidirectional driving gear mechanism. It is a gear mechanism having two degrees of freedom on one surface. This flexible display mechanism is expected to be placed inside a car dashboard as a human interface and for workspace optimization. In this study, we propose a novel flexible omnidirectional driving gear for supporting flexible displays discussing its design, motion range, repeatability, positional accuracy, and adaptability to any guide surface through magnetic coupling. The experiments showed satisfactory results for positional accuracy and repeatability with adaptability over a wide range of curvatures.

Calibration of fixtures in robotic work cells is essential but also time consuming and error-prone, and poor calibration can easily lead to wasted debugging time in down-stream tasks. Contact-based calibration methods let the user measure points on the fixture’s surface with a tool tip attached to the robot’s end effector. Most such methods require the user to manually annotate correspondences on the CAD model, however, this is error-prone and a cumbersome user experience. We propose a correspondence-free alternative: The user simply measures a few points from the fixture’s surface, and our method provides a tight superset of the poses which could explain the measured points. This naturally detects ambiguities related to symmetry and uninformative points and conveys this uncertainty to the user. Perhaps more importantly, it provides guaranteed bounds on the pose. The computation of such bounds is made tractable by the use of a hierarchical grid on SE(3). Our method is evaluated both in simulation and on a real collaborative robot, showing great potential for easier and less error-prone fixture calibration. sites.google.com/view/ttpose

Due to their flexibility and ease of use, Dynamical Movement Primitives (DMPs) are widely used in robotics applications and research. DMPs combine linear dynamical systems to achieve robustness to perturbations and adaptation to moving targets with non-linear function approximators to fit a wide range of demonstrated trajectories.We propose a novel DMP formulation with a generalized logistic function as a delayed goal system. This formulation inherently has low initial jerk, and generates the bell-shaped velocity profiles that are typical of human movement. As the novel formulation is more expressive, it is able to fit a wide range of human demonstrations well, also without a non-linear forcing term. We exploit this increased expressiveness by automating the fitting of the dynamical system parameters through opti-mization. Our experimental evaluation demonstrates that this optimization regularizes the forcing term, and improves the interpolation accuracy of parametric DMPs.

Accurate 3D object pose estimation is key to enabling many robotic applications that involve challenging object interactions. In this work, we show that the density field created by a state-of-the-art efficient radiance field reconstruction method is suitable for highly accurate and robust pose estimation for objects with known 3D models, even when they are very small and with challenging reflective surfaces. We present a fully automatic object pose estimation system based on a robot arm with a single wrist-mounted camera, which can scan a scene from scratch, detect and estimate the 6-Degrees of Freedom (DoF) poses of multiple objects within a couple of minutes of operation. Small objects such as bolts and nuts are estimated with accuracy on order of 1mm.

A miniature haptic stimulation device utilizes focused ultrasound to deliver a tactile haptic sensation to the finger. The 1-3 piezocomposite device has a 1 cm2 footprint, which is an order of magnitude smaller than other ultrasonic haptic devices and is a good candidate for wearable tactile rendering systems. The device focuses energy to a 1 mm3 voxel. The current prototype was validated with a small, preliminary human subject study and requires an average input voltage of 68.8 V to elicit tactile sensation. The sensory drive voltage threshold will decrease with future refinement of mechanical impedance matching and focusing.