IROS 2024 Papers

In complex indoor structural scenes such as shopping centers and malls, camera pose estimation using pure point features is easy to fail due to the difficulty in extracting sufficient and stable point features from weak textures or dynamic environments. Recent works have attempted to address these challenges by introducing line features. However, the addition of line features increases the number of parameters and landmarks for BA (Bundle Adjustment), leading to efficiency reduction. This is a common issue in multi-modal SLAM (Simultaneous Localization And Mapping). To address this issue, this paper proposes a novel visual-depth-inertial odometry (ASML-VDIO) framework by combining RGB-D and IMU sensors. To improve the efficiency of BA, the proposed landmark classification method classifies 3D landmarks into accurate landmarks and other landmarks based on spatial consistency verification and depth range limitation. Then, accurate landmarks are fixed, and only other landmarks are optimized in the optimization of BA. Furthermore, to remove line features extracted from dynamic objects (pedestrian, shopping-car, etc), we propose a dynamic line removal method that combines geometric constraints and motion constraints of line features. Finally, the method is evaluated on public and author-collected datasets, showing competitive accuracy and robustness in complex indoor structural scenes while 71% speedup on optimization thread with same constraints.

LiDAR-Inertial Odometry (LIO) demonstrates outstanding accuracy and stability in general low-speed and smooth motion scenarios. However, in high-speed and intense motion scenarios, such as sharp turns, two primary challenges arise: firstly, due to the limitations of IMU frequency, the error in estimating significantly non-linear motion states escalates; secondly, drastic changes in the Field of View (FOV) may diminish the spatial overlap between LiDAR frame and pointcloud map (or between frames), leading to insufficient data association and constraint degradation.To address these issues, we propose a novel Adaptive Sliding window LIO framework (AS-LIO) guided by the Spatial Overlap Degree (SOD). Initially, we assess the SOD between the LiDAR frames and the registered map, directly evaluating the adverse impact of current FOV variation on pointcloud alignment. Subsequently, we design an adaptive sliding window to manage the continuous LiDAR stream and control state updates, dynamically adjusting the update step according to the SOD. This strategy enables our odometry to adaptively adopt higher update frequency to precisely characterize trajectory during aggressive FOV variation, thus effectively reducing the non-linear error in positioning. Meanwhile, the historical constraints within the sliding window reinforce the frame-to-map data association, ensuring the robustness of state estimation. Experiments show that our AS-LIO framework can quickly perceive and respond to challenging FOV change, outperforming other state-of-the-art LIO frameworks in terms of accuracy and robustness.

Sequential decision-making can be formulated as a text-conditioned video generation problem, where a video planner, guided by a text-defined goal, generates future frames visualizing planned actions, from which control actions are subsequently derived. In this work, we introduce Active Region Video Diffusion for Universal Policies (ARDuP), a novel framework for video-based policy learning that emphasizes the generation of active regions, i.e. potential interaction areas, enhancing the conditional policy’s focus on interactive areas critical for task execution. This innovative framework integrates active region conditioning with latent diffusion models for video planning and employs latent representations for direct action decoding during inverse dynamic modeling. By utilizing motion cues in videos for automatic active region discovery, our method eliminates the need for manual annotations of active regions. We validate ARDuP’s efficacy via extensive experiments on simulator CLIPort and the real-world dataset BridgeData v2, achieving notable improvements in success rates and generating convincingly realistic video plans.

Robot Imitation Learning (IL) is a crucial technique in robot learning, where agents learn by mimicking human demonstrations. However, IL encounters scalability challenges stemming from both non-user-friendly demonstration collection methods and the extensive time required to amass a sufficient number of demonstrations for effective training. In response, we introduce the Augmented Reality for Collection and generAtion of DEmonstrations (ARCADE) framework, designed to scale up demonstration collection for robot manipulation tasks. Our framework combines two key capabilities: 1) it leverages AR to make demonstration collection as simple as users performing daily tasks using their hands, and 2) it enables the automatic generation of additional synthetic demonstrations from a single human-derived demonstration, significantly reducing user effort and time. We assess ARCADE’s performance on a real Fetch robot across three robotics tasks: 3-Waypoints-Reach, Push, and Pick-And-Place. Using our framework, we were able to rapidly train a policy using vanilla Behavioral Cloning (BC), a classic IL algorithm, which excelled across these three tasks. We also deploy ARCADE on a real household task, Pouring-Water, achieving an 80% success rate.

Dexterous manipulation, particularly adept coordinating and grasping, constitutes a fundamental and indispensable capability for robots, facilitating the emulation of human-like behaviors. Integrating this capability into robots empowers them to supplement and even supplant humans in undertaking increasingly intricate tasks in both daily life and industrial settings. Unfortunately, contemporary methodologies encounter serious challenges in devising manipulation trajectories owing to the intricacies of tasks, the expansive robotic manipulation space, and dynamic obstacles. We propose a novel approach, APEX, to address all these difficulties by introducing a collision-free latent diffusion model for both robotic motion planning and manipulation. Firstly, we simplify the complexity of real-life ambidextrous dual-arm robotic manipulation tasks by abstracting them as aligning two vectors. Secondly, we devise latent diffusion models to produce a variety of robotic manipulation trajectories. Furthermore, we integrate obstacle information utilizing a classifier-guidance technique, thereby guaranteeing both the feasibility and safety of the generated manipulation trajectories. Lastly, we validate our proposed algorithm through extensive experiments conducted on the hardware platform of ambidextrous dual-arm robots. Our algorithm consistently generates successful and seamless trajectories across diverse tasks, surpassing conventional robotic motion planning algorithms. These results carry significant implications for the future design of diffusion robots, enhancing their capability to tackle more intricate robotic manipulation tasks with increased efficiency and safety. Complete video demonstrations of our experiments can be found in https://sites.google.com/view/apex-dual-arm/home.

We present AMCO, a novel navigation method for quadruped robots that adaptively combines vision-based and proprioception-based perception capabilities. Our approach uses three cost maps: general knowledge map; traversability history map; and current proprioception map; which are derived from a robot’s vision and proprioception data, and couples them to obtain a coupled traversability cost map for navigation. The general knowledge map encodes terrains semantically segmented from visual sensing, and represents a terrain’s typically expected traversability. The traversability history map encodes the robot’s recent proprioceptive measurements on a terrain and its semantic segmentation as a cost map. Further, the robot’s present proprioceptive measurement is encoded as a cost map in the current proprioception map. As the general knowledge map and traversability history map rely on semantic segmentation, we evaluate the reliability of the visual sensory data by estimating the brightness and motion blur of input RGB images and accordingly combine the three cost maps to obtain the coupled traversability cost map used for navigation. Leveraging this adaptive coupling, the robot can depend on the most reliable input modality available. Finally, we present a novel planner that selects appropriate gaits and velocities for traversing challenging outdoor environments using the coupled traversability cost map. We demonstrate AMCO’s navigation performance in different real-world outdoor environments and observe 10.8%-34.9% reduction w.r.t. two stability metrics, and up to 50% improvement in terms of success rate compared to current navigation methods.

We present AGL-NET, a novel learning-based method for global localization using LiDAR point clouds and satellite maps. AGL-Net tackles two critical challenges: bridging the representation gap between image and points modalities for robust feature matching, and handling inherent scale discrepancies between global view and local view. To address these challenges, AGL-Net leverages a unified network architecture with a novel two-stage matching design. The first stage extracts informative neural features directly from raw sensor data and performs initial feature matching. The second stage refines this matching process by extracting informative skeleton features and incorporating a novel scale alignment step to rectify scale variations between LiDAR and map data. Furthermore, a novel scale and skeleton loss function guides the network toward learning scale-invariant feature representations, eliminating the need for pre-processing satellite maps. This significantly improves real-world applicability in scenarios with unknown map scales. To facilitate rigorous performance evaluation, we introduce a meticulously designed dataset within the CARLA simulator specifically tailored for metric localization training and assessment.

Despite important progress in recent years, social robots are still far away from showing advanced behavior for interaction and adaptation in human environments. Thus, we are interested in studying social cognition in human-robot interaction (HRI), notably in improving communication skills relying on joint attention (JA) and knowledge sharing. Since JA involves low-level cognitive processes in humans, we take into account the implications of Moravec’s Paradox and focus on the aspect of knowledge representation. Inspired by 4E cognition principles, we study egocentric localization through the concept of sensory ego-sphere. We propose a neural network architecture named AEGO to model attention for each agent in interaction and show how to fuse information in a common representation space. From the perspective of dynamic fields theory, AEGO takes into account the dynamics of bottom-up and top-down modulation processes and the effects of neural excitatory and inhibitory synaptic interaction. In this work we evaluate the model in simulation and experiments with the robot Pepper in JA tasks based on proprioception, vision, rudimentary natural language and Hebbian plasticity. Results show that AEGO is convenient for HRI, allowing the human and the robot to share attention and knowledge about objects in scenarios close to everyday situations. AEGO constitutes a novel brain-inspired architecture to model attention that is suitable for multi-agent applications relying on social cognition skills, having the potential to generalize to several robotics platforms and HRI scenarios.

Soft robotics has emerged as the standard solution for grasping deformable objects, and has proven invaluable for mobile robotic exploration in extreme environments. However, despite this growth, there are no widely adopted computational design tools that produce quality, manufacturable designs. To advance beyond the diminishing returns of heuristic bio-inspiration, the field needs efficient tools to explore the complex, non-linear design spaces present in soft robotics, and find novel high-performing designs. In this work, we investigate a hierarchical design optimization methodology which combines the strengths of topology optimization and quality diversity optimization to generate diverse and high-performance soft robots by evolving the design domain. The method embeds variably sized void regions within the design domain and evolves their size and position, to facilitating a richer exploration of the design space and find a diverse set of high-performing soft robots. We demonstrate its efficacy on both benchmark topology optimization problems and soft robotic design problems, and show the method enhances grasp performance when applied to soft grippers. Our method provides a new framework to design parts in complex design domains, both soft and rigid.

This paper describes a parallel-actuated robotic mechanism designed to provide two degrees-of-freedom (DOF) to the end-effector relative to a fixed base. In a potential application as a head-neck traction brace, these two independent DOFs are the vertical translation of the head with respect to shoulders and a specified orientation of the head in lateral bending. Motivated by recommended clinical methods to apply traction forces on the head, it is designed to provide vertical traction force on the head while tilted in a specific orientation. The design has four chains starting from a base stationed at the shoulders, each chain having 5 DOFs. Each chain imposes a single constraint on the motion of the end-effector. Together, four chains would apply four constraints, allowing only two DOFs of motion to the end-effector. Two out of four component chains are actively driven by linear actuators. Our kinematic studies show that the achievable workspace of this mechanism with a specific stroke length of actuators of ± 50 mm results in 175-222 mm of vertical translation and up to ± 9 ◦ of lateral bending. The lateral bending is coupled to the flexion/extension angle of the end-effector. A physical prototype was constructed to investigate the functional realization of the design in hardware. Overall, the physical prototype validated the motion of the theoretical model despite potential errors in the fabrication, making the design a candidate for potential head-neck traction application.

Skin cancer represents a significant global health concern, with melanoma alone accounting for thousands of deaths annually. Early diagnosis is crucial for improving survival rates and reducing healthcare costs. While traditional diagnostic approaches involve visual inspection followed by biopsy, emerging technologies offer less invasive options with improved precision. In this study, a novel non-invasive device was designed, developed, and validated to employ near-infrared reflectance spectroscopy for skin lesion analysis. Furthermore, this work presents a machine learning approach aimed at classifying different types of skin lesions, as well as a new sequential approach to distinguish benign from malignant lesions based on spectral data and exploring the impact of anamnestic features. The device was used in two independent hospitals in Italy to collect data from 69 patients in total, including various types of skin lesions, all of whom followed the standard protocol for screening and diagnosis intervention. The implemented model achieved a recall of 93.8% and an accuracy of 75% for melanoma and benign classification, and a recall of 100% and an accuracy of 98.6% in distinguishing non-melanoma cancer from benign lesions, demonstrating promising results for skin cancer diagnosis utilizing spectral and anamnestic data. In summary, this study contributes to the development of allied non-invasive diagnostic tools and underscores the potential of machine learning in dermatology using spectroscopic data.

Direct Visual Servoing (DVS) is a technique that controls the robot motion by using the pixel intensities captured by a camera. DVS demonstrates high accuracy at convergence, prompting the development of various methods aimed at expanding its convergence domain.In this paper, we propose a mathematical characterization of the DVS convergence domain with closed-form expressions for the controlled degrees of freedom. From these expressions, we concluded that the extent of the convergence domain is related to the presence of isotropic or defocus blur, a phenomenon that had only been observed previously as a trend in empirical experiments.

Multi-robot supervision becomes increasingly cognitively demanding as the ratio of robots to human supervisors rises, potentially leading to situational awareness (SA) losses and robot system failures. Nonverbal cues have been employed to direct supervisor attention and prevent awareness loss in diverse human-computer interaction (HCI) settings. This paper compares the effects of uni-modal and multi-modal audiovisual nonverbal cues on supervisor SA in a multi-robot supervision task. In a simulation-based navigation scenario, 50 participants monitored a multi-robot mission and responded to supervision requests from the robots. We evaluated supervisor SA using response speed and the situational awareness global assessment technique. Results demonstrate that supervisor awareness hinges on the communication method employed by the robots, with greater significance observed at higher awareness levels and when the robot-to-human ratio is higher. Findings also indicate the utility of sonification mapping in human-multirobot interactions and the benefits of multi-modal cues for sustaining awareness during multi-robot supervision.

With the aging of society, there has been an increase in the number of elderly individuals with limb movement disorders. Active rehabilitation training using limb rehabilitation electromechanical devices that incorporate multimodal sensing and monitoring functions can significantly contribute to the recovery of limb motor functions. This report introduces a wearable mechanical pressure-electrophysiological monitoring bimodal sensing system specifically designed for human limb rehabilitation devices. By utilizing just four electrodes (SE, CE/DE, GND, REF), this system enables simultaneous and co-located measurement of surface electromyographic (sEMG), pressure, and mechanomyography (MMG) signals. These signals can be utilized to analyze muscle tension, stiffness, and tremor information. At last, this sensing system was used to assess muscle contraction force and localized muscle fatigue. The time and frequency domain characteristics of physiological signals during exercise were thoroughly investigated. The wearable mechanical pressure-electrophysiological bimodal sensing system can provide valuable data references for rehabilitation robots or human limb rehabilitation device, which is of great significance in the diagnosis of muscular diseases and rehabilitation treatment.

Conventional bidirectional conveyance platforms use a flat translating belt or a series of spinning wheels or rollers to apply a shear force to payloads to move them. Wheel/roller-based conveyors in particular cannot double as a worktop when idle, do not support collision-free multi-object manipulation by default, and are not optimized to move objects that are either slippery or pliable—let alone both. This paper introduces a Voxel-Enabled Robotic Assistant (VERA), a network of intelligent table "partitions" whose topologically dynamic worktops enable omnidirectional conveyance; each partition is composed of a 2D array of "quadrants," axisymmetric modules that can be hot-swapped for maintenance or repairs; each quadrant contains a 2D array of "cells," unitary robotic submodules; each cell houses an independently controllable "voxel," the motorized rotary element that conveys an overhead object. The efficacy of a VERA prototype was determined by evaluating waypoint error as a range of payloads were maneuvered between trajectory waypoints. By conveying both pliable and rigid payloads having slippery textures, the faceted voxels outperformed those augmented to mimic the circular-profiled wheels/rollers of competitor systems. VERA also successfully performed collision-free multi-object planar manipulations planned by its pathfinding algorithm. In light of these results, VERA emerges as a promising material handling platform for use in "Future of Work" settings as the need for multi-purpose collaborative industrial robots continues to grow.

The ability to handle contact makes robots qualified for many complicated tasks, such as welding, hammering, and wiping. Robot cameras facilitate position planning and control without the geometric knowledge of contact surfaces since they can project contact surfaces onto a 2-dimensional image plane. However, existing hybrid vision-force control (HVFC) methods still rely on this knowledge to project the force on the constraint subspace and do not adequately leverage the redundant degrees of freedom (DoFs) for redundant robots with contact tasks. This paper proposes an enhanced HVFC solution for redundant robots equipped with an eye-to-hand camera to unify HVFC in the Cartesian space and impedance control in the joint nullspace into one closed-loop dynamics with rigorous stability guarantees. Any geometric knowledge of contact surfaces is not required by projecting the force into the redundant space of the visual task rather than the surface’s normal space. Experiments on a seven- DoF collaborative robot have verified that the proposed method is qualified for simultaneous contact tasks in the Cartesian space and compliant interaction in the joint nullspace.

Safe and efficient mobility is vital for mobile robots navigating long-range indoor crowd environments, such as supermarkets, restaurants, and railway stations. Traditional path planning methods are challenged because of the high dynamics of pedestrians and constrained feasible regions. Existing long-range deep reinforcement learning (DRL) path planning methods often exhibit low success rates and driving speeds in long-range navigation tasks under crowded conditions. To overcome these issues, we propose a new two-stage DRL method, known as TSDRL, where the long-range navigation task is divided into subgoal generation (SG) and planning refinement (PR) stages. In the SG stage, the agent is trained to learn a decision-making policy to generate subgoals at each decision time to avoid dense crowds. In the PR stage, the agent learns a safer and more efficient planning policy based on each subgoal generated in the SG stage to improve the robot’s movement safety and speed. Simulated experiments show that our method outperforms traditional and long-range DRL path planning methods in terms of safety, efficiency, generalization, and robustness. Furthermore, we evaluate our approach using the Turtlebot2 platform in a real-world setting, demonstrating that the robot can navigate safely and efficiently while avoiding dense crowds.

This work describes the development, demonstration, and performance evaluation study of a wearable human machine interface for robotic teleoperation. We present a novel tetherless human machine interface in the form of a backpack, wearable 3D arm motion capture sensors, finger flexion sensors, and pneumatic haptic feedback muscles. The system is integrated in a complete teleoperation framework, enabling users to be immersed in a remote environment through virtual reality headgear, facilitating intuitive manipulation of an industrial articulated arm. The human machine interface samples the kinematic configuration of the user’s arm, hand, and fingers using multiple inertial measurement units and capacitive sensors respectively, and streams it to the teleoperation software stack. The gripping forces experienced at the robot’s end-effector are acquired using a custom three-dimensional Hall-effect magnetic sensor. The system simultaneously renders the kinesthetic and tactile feedback on the user’s fingers through custom designed pneumatically actuated soft robotic haptic muscles. The efficacy of the human machine interface and the teleoperation system was tested and evaluated by conducting user studies, which showed 31.4% faster teleoperation compared to a keypad controller, and 60% less gripping force utilized with haptics enabled. The findings of the study guided the design and prototype development of a printed electronics based stretchable sleeve and glove motion capture unit to improve the portability, ergonomics, and user experience of the human machine interface.

In this work, an upgraded exoskeleton design is presented with enhanced trajectory tracking and mechanical transparency. Compared to the first version, the design features a 3-DoF actuated shoulder joint and a mechanism to regulate the pretension of Bowden cables. Force/torque sensors are installed to directly measure the interaction forces between the human arm and the exoskeleton at the connecting points. Three control strategies were evaluated to follow a desired trajectory; A PD controller, a PD controller with friction observer, and an adaptive controller based on Radial Basis Function (RBF). These strategies also form the basis for an admittance control, aimed at improving the exoskeleton’s mechanical transparency during interaction with the human arm. Simulations and experimental results demonstrate that the PD control, supported by friction estimation via a momentum observer, achieves superior tracking performance. Moreover, the system’s mechanical transparency is enhanced using the admittance RBF-based controller, showing marginally superior results.