IROS 2024 Papers

Flexible sensors for joint angle measurement play a crucial role in various human-robot interaction applications. In previous studies, sensors with various sensing mechanisms have been developed. Among them, optical waveguide sensors exhibit high resistance to environmental factors (such as temperature and humidity) and low sensitivity to electromagnetic interference. Researchers have enhanced the sensitivity of optical waveguide sensors to tensile strain by doping other substances (such as graphite) into the optical core material of the optical waveguide. However, the sensitivity of measuring joint angles based on tensile strain principles remains relatively low. In nature, arthropods utilize crack-like structures near their leg joints to perceive minute mechanical stress changes. Here, we propose a curvature sensor based on a Flexible Optical Crack Waveguide Structure (FOCWS) inspired by the arthropod sensory systems. By cutting the optical core, we increase its light power loss during bending strain, thereby enhancing the sensor’s sensitivity to angle measurement. The characteristics of light propagation and geometric parameters were studied through simulation, and experiments were designed to validate the simulation results. The average sensitivity is 0.068 dB/°, which is nearly 300 times higher compared to uncut optical waveguide.

Recent advancements in neural implicit fields for Simultaneous Localization and Mapping (SLAM) have provided breakthroughs. However, the benefits of reconstruction results to the perception ability of robot are minimal. Therefore, we propose FI-SLAM, a dense semantic instance SLAM system based on neural implicit representation, which significantly aids robots in better understanding the scene. FI-SLAM employs a coordinate and plane joint encoding method, which reduces the difficulty of feature storage by flattening the feature space. Furthermore, to improve representation efficiency, we use the method of adjacent feature level linear interpolation to describe features. We propose a feature fusion (FF) method to merge the object features with the scene features. The fused feature vector enhances the reconstruction accuracy of the local scene while ensuring the global reconstruction effect. It has improved the global reconstruction effect of the scene and the accuracy of camera tracking. Numerous experiments on synthetic and real-world datasets demonstrate that our method can assure accurate tracking precision, high-fidelity reconstruction results, and complete semantic instance maps. In summary, the algorithm we proposed heavily augments the scene perception capabilities of robot.

Robotic surgery is a rapidly developing field that can greatly benefit from the automation of surgical tasks. However, training techniques such as Reinforcement Learning (RL) require a high number of task repetitions, which are generally unsafe and impractical to perform on real surgical systems. This stresses the need for simulated surgical environments, which are not only realistic, but also computationally efficient and scalable. We introduce FF-SRL (Fast and Flexible Surgical Reinforcement Learning), a high-performance learning environment for robotic surgery. In FF-SRL both physics simulation and RL policy training reside entirely on a single GPU. This avoids typical bottlenecks associated with data transfer between the CPU and GPU, leading to accelerated learning rates. Our results show that FF-SRL reduces the training time of a complex tissue manipulation task by an order of magnitude, down to a couple of minutes, compared to a common CPU/GPU simulator. Such speed-up may facilitate the experimentation with RL techniques and contribute to the development of new generation of surgical systems. To this end, we make our code publicly available to the community.

The ability of resource-constrained biological systems such as fruitflies to perform complex and high-speed maneuvers in cluttered environments has been one of the prime sources of inspiration for developing vision-based autonomous systems. To emulate this capability, the perception pipeline of such systems must integrate information cues from tasks including optical flow and depth estimation, object detection and tracking, and segmentation, among others. However, the conventional approach of employing slow, synchronous inputs from standard frame-based cameras constrains these perception capabilities, particularly during high-speed maneuvers. Recently, event-based sensors have emerged as low latency and low energy alternatives to standard frame-based cameras for capturing high-speed motion, effectively speeding up perception and hence navigation. For coherence, all the perception tasks must be trained on the same input data. However, present-day datasets are curated mainly for a single or a handful of tasks and are limited in the rate of the provided ground truths. To address these limitations, we present Flying Event Dataset fOr Reactive behAviour (FEDORA) - a fully synthetic dataset for perception tasks, with raw data from frame-based cameras, event-based cameras, and Inertial Measurement Units (IMU), along with ground truths for depth, pose, and optical flow at a rate much higher than existing datasets.

Trajectory prediction plays a significant role in autonomous driving, with current challenges primarily focused on capturing complex interactions in traffic scenes. Previous methods usually directly encode non-interactive and interactive information together, and then decode them for trajectory prediction. However, given the complexity inherent property in the trajectory generation process (e.g., the generation of trajectory points are influenced by the interactions among multiple moving agents, as well as the interactions between agents and the static environment), previous approaches fail to precisely capture separate variations of the trajectory generation process. In this paper, we propose a general and plug-and-play feature decoupling framework for trajectory prediction called FDNet, which can learn the interactive and non-interactive factors in the latent space to capture separate variations of the trajectory generation process. At its core, FDNet is comprised of a Non-interactive Feature Extraction Module to extract non-interactive features, and an Interactive Feature Decoupling Module to decouple interactive features. Extensive experiments conducted on Argoverse and nuScenes demonstrate that FDNet significantly improves the performance of existing methods.

In this work, we introduce the EyeSight Hand, a 7 degrees of freedom (DoF) humanoid hand featuring integrated vision-based tactile sensors tailored for enhanced whole-hand manipulation. Additionally, we introduce an actuation scheme centered around quasi-direct drive actuation to achieve human-like strength and speed while ensuring robustness for large-scale data collection. We evaluate the EyeSight Hand on three challenging tasks: bottle opening, plasticine cutting, and plate pick and place, which require a blend of complex manipulation, tool use, and precise force application. Imitation learning models trained on these tasks, with a vision dropout strategy, showcase the benefits of tactile feedback in enhancing task success rates. Our results reveal that the integration of tactile sensing dramatically improves task performance, underscoring the critical role of tactile information in dexterous manipulation.

The utilization of mobile robots equipped with multiple light detection and ranging (LiDAR) sensors, capable of perceiving their surroundings, is on the rise due to the miniaturization and cost reduction of LiDAR technology. This paper introduces a target-less extrinsic calibration method for multiple LiDARs with non-overlapping fields of view (FoV). The proposed method leverages accumulated point clouds of the floor plane and objects obtained during robot motion. It enables accurate calibration, even in challenging configurations where LiDARs are directed towards the floor plane, which can introduce biased feature values. Additionally, the method incorporates a noise removal module that takes into account the scanning pattern to address bleeding points, which are significant sources of error in point cloud alignment when using high-density LiDARs. Evaluations conducted through simulation demonstrate that the proposed method achieves higher accuracy in extrinsic calibration with two and four LiDARs compared to conventional methods, regardless of the type of objects. Furthermore, experiments conducted using a real mobile robot validate the effectiveness of our proposed noise removal module in precisely eliminating noise compared to conventional methods. The estimated extrinsic parameters successfully contribute to the creation of consistent 3D maps.

Traditional interaction perception of parallel robots relies on a six-dimensional force sensor for contact sensing at their distal end. However, the sensor body occupies the space of moving platform and also increases the load on the robot actuations. To enable both minimization and embodied intelligence, this paper proposes an external interaction estimation method with embodied mechanical intelligence by embedding two single-axis force sensors in each leg of 6-PSS parallel robot. The method uses a backward propagation neural network optimized by sparrow search algorithm, and it can simultaneously estimate the external force and its position using information from multiple single-axis force sensors and the encoder of driving motor. The experimental platform is established to collect the data and train the network. The result shows that the force estimation mean error is 2.4% and the position estimation error is 2.9%. A demonstration with a virtual display interface showing the reconstructed parallel robot pose, and the interaction force and its pose using the proposed estimation method, indicates the effectiveness of the proposed interaction method with embodied mechanical intelligence for 6-PSS parallel robot

We propose a hierarchical task and motion planning (TAMP) for autonomous construction that manipulates deformable objects, such as terrain excavation. The TAMP is required to generate an efficient task plan to meet high-level construction goals at sites with environmental diversity while ensuring motion feasibility. The difficulty, however, is to manipulate deformable objects containing nonlinear dynamics with a target given by a continuous value as the task specification. Optimization-based TAMP with signal temporal logic specifications in robotics is promising because of its continuous task specification and formulation as a nonlinear programming problem. The key to its application to extensive, long-term planning at real construction sites is a computationally efficient and stable formulation. We introduce a new expression for deformable objects with a simple differentiable function and a system model that can represent mode transitions based on machine action on the objects. This allows TAMP to be formulated as simultaneously selecting an action for objects and planning the motion to execute it. Furthermore, a hierarchical method that gives appropriate initial values is combined to improve optimality for large-scale nonlinear problems. From the verification by numerical experiments, the proposed method can generate a plan that minimizes the time to meet the task goal, even when the area is expanded.

The hybrid discrete/continuous nature of task and motion planning (TAMP) results often in a combinatorial explosion. This challenge is even more pronounced in multi-robot TAMP problems due to the increase in dimensionality of the action space. Previous works use action feasibility prediction as a heuristic to accelerate TAMP. However, these methods are limited to box-shaped objects and specific single or dual robot settings. In this paper, we propose a feasibility-enabled multi-robot TAMP algorithm capable of tackling complex multi-robot manipulation problems. Also, we expand on our previous work on action and grasp feasibility prediction [1] by extending its use to mesh-shaped objects. We demonstrate the performance of our method compared to a non feasibility-informed baseline, and show its ability to handle TAMP problems requiring the collaboration of multiple robots.

Integrated Task and Motion Planning (TAMP) offers opportunities for achieving generalized autonomy in robots but also poses challenges. It involves searching in both symbolic task space and high-dimensional motion space, while also addressing geometrically infeasible actions within its hierarchical process. We introduce a novel TAMP decision-making framework, utilizing an extended decision tree for both symbolic task planning and high-dimensional motion variable binding. Employing top-k planning, we generate a skeleton space with diverse candidate plans, seamlessly integrating it with motion variable spaces into an extended decision space. Subsequently, Monte-Carlo Tree Search (MCTS) is utilized to maintain a balance between exploration and exploitation at decision nodes, ultimately yielding optimal solutions. Our approach combines symbolic top-k planning with concrete motion variable binding, leveraging MCTS for proven optimality, resulting in a powerful algorithm for handling combinatorial complexity in long-horizon manipulation tasks. Empirical evaluations demonstrate the algorithm’s effectiveness in diverse, challenging robot tasks, in comparison with the baseline methods.

Visual Odometry (VO) is one of the fundamental tasks in computer vision for robotics. However, its performance is deeply affected by High Dynamic Range (HDR) scenes, omnipresent outdoor. While new Automatic-Exposure (AE) approaches to mitigate this have appeared, their comparison in a reproducible manner is problematic. This stems from the fact that the behavior of AE depends on the environment, and it affects the image acquisition process. Consequently, AE has traditionally only been benchmarked in an online manner, making the experiments non-reproducible. To solve this, we propose a new methodology based on an emulator that can generate images at any exposure time. It leverages BorealHDR, a unique multi-exposure stereo dataset collected over 10 km, on 55 trajectories with challenging illumination conditions. Moreover, it includes lidar-inertial-based global maps with pose estimation for each image frame as well as Global Navigation Satellite System (GNSS) data, for comparison. We show that using these images acquired at different exposure times, we can emulate realistic images, keeping a Root-Mean-Square Error (RMSE) below 1.78 % compared to ground truth images. To demonstrate the practicality of our approach for offline benchmarking, we compared three state-of-the-art AE algorithms on key elements of Visual Simultaneous Localization And Mapping (VSLAM) pipeline, against four baselines. Consequently, reproducible evaluation of AE is now possible, speeding up the development of future approaches. Our code and dataset are available on-line at this link: https://github.com/norlab-ulaval/BorealHDR

We present a novel pneumatic augmentation to traditional electric motor-actuated legged robot to increase intermittent power density to perform infrequent explosive hopping behaviors. The pneumatic system is composed of a pneumatic pump, a tank, and a pneumatic actuator. The tank is charged up by the pump during regular hopping motion that is created by the electric motors. At any time after reaching a desired air pressure in the tank, a solenoid valve is utilized to rapidly release the air pressure to the pneumatic actuator (piston) which is used in conjunction with the electric motors to perform explosive hopping, increasing maximum hopping height for one or subsequent cycles. We show that, on a custom-designed one-legged hopping robot, without any additional power source and with this novel pneumatic augmentation system, their associated system identification and optimal control, the robot is able to realize highly explosive hopping with power amplification per cycle by a factor of approximately 5.4 times the power of electric motor actuation alone.

Mechanical metamaterials are microscale patterned structures that are designed to have specific mechanical properties at a macro-scale that are atypical of natural materials. Robotic manipulators composed of these materials can exhibit deformation and motion capabilities that can be customized and easily fabricated. However, as of now, the motion capability of such manipulators are encoded in their physical composition and cannot be changed. This paper presents multimodal metamaterial-based robot prototypes which can switch between the behaviors found in two different metamaterials. Two such robots are explored, a bending/shearing robot and a bending/twisting robot. The robot design is described in detail, including how the robots toggle between behavior modes via mechanical actuation of a sliding rod insert. Multi-modal robots are compared to their single-mode equivalents to characterize their capabilities. The single-mode behaviors are largely preserved in the multi-modal innovations. The multi-modal prototypes also demonstrate variable rigidity. We discuss the feasibility of using robots of this design as part of a robotic surgical system.

Autonomous driving presents a complex challenge, which is usually addressed with artificial intelligence models that are end-to-end or modular in nature. Within the landscape of modular approaches, a bio-inspired neural circuit policy model has emerged as an innovative control module, offering a compact and inherently interpretable system to infer a steering wheel command from abstract visual features. Here, we take a leap forward by integrating a variational autoencoder with the neural circuit policy controller, forming a solution that directly generates steering commands from input camera images. By substituting the traditional convolutional neural network approach to feature extraction with a variational autoencoder, we enhance the system’s interpretability, enabling a more transparent and understandable decision-making process.In addition to the architectural shift toward a variational autoencoder, this study introduces the automatic latent perturbation tool, a novel contribution designed to probe and elucidate the latent features within the variational autoencoder. The automatic latent perturbation tool automates the interpretability process, offering granular insights into how specific latent variables influence the overall model’s behavior. Through a series of numerical experiments, we demonstrate the interpretative power of the variational autoencoder-neural circuit policy model and the utility of the automatic latent perturbation tool in making the inner workings of autonomous driving systems more transparent.

This work explores non-prehensile manipulation (NPM) and whole-body interaction as strategies for enabling robotic manipulators to conduct manipulation tasks despite experiencing locked multi-joint (LMJ) failures. LMJs are critical system faults where two or more joints become inoperable; they impose constraints on the robot’s configuration and control spaces, consequently limiting the capability and reach of a prehensile-only approach. This approach involves three components: i) modeling the failure-constrained workspace of the robot, ii) generating a kinodynamic map of NPM actions within this workspace, and iii) a manipulation action planner that uses a sim-in-the-loop approach to select the best actions to take from the kinodynamic map. The experimental evaluation shows that our approach can increase the failure-constrained reachable area in LMJ cases by 79%. Further, it demonstrates the ability to complete real-world manipulation with up to 88.9% success when the end-effector is unusable and up to 100% success when it is usable.

Self-supervised multi-frame depth estimation methods only require unlabeled monocular videos for training. However, most existing methods face challenges, including accuracy degradation caused by moving objects in dynamic scenes and scale ambiguity due to the absence of real-world references. In this field, the emergence of low-cost LiDAR sensors highlights the potential to improve the robustness of multi-frame depth estimation by exploiting accurate sparse measurements at the correct scale. Moreover, the LiDAR ranging points often intersect moving objects, providing more precise depth cues for them. This paper explores the impact of few-beam LiDAR data on self-supervised multi-frame depth estimation, proposing a method that fuses multi-frame matching and sparse depth features. It significantly enhances depth estimation robustness, particularly in scenarios involving moving objects and textureless backgrounds. We demonstrate the effectiveness of our approach through comprehensive experiments, showcasing its potential to address the limitations of existing methods and paving the way for more robust and reliable depth estimation based on this paradigm.

Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We evaluated five constrained policy optimization algorithms for quadrupedal locomotion using three different robot models. Our aim is to evaluate their applicability in real-world scenarios. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.