IROS 2024 Papers

Most existing soft robotic actuators require multiple chambers to achieve multi-directional bending. In this paper, we present the design, fabrication, and analysis of a novel circular actuator. The actuator is made from a single soft material and is capable of bi-directional bending using only positive pressure in a single chamber. To accurately predict the required pressure for all bending angles, we developed an analytical model for the full range of motion - both uncurling and curling. We tested and analyzed actuators fabricated out of four types of silicone and found that the softest actuators have a bending range from the initial bending angle, 210°, to 0°, and to 225° in the other direction in 112 kPa. When constraining the actuators, we found they can create up to 5.1 N of blocked force in 210 kPa. Additionally, we demonstrate that a single circular actuator can grasp objects of various weights up to 800 g from both the inside and outside.

Cross-Modal Retrieval (CMR), which retrieves relevant items from one modality (e.g., audio) given a query in another modality (e.g., visual), has undergone significant advancements in recent years. This capability is crucial for robots to integrate and interpret information across diverse sensory inputs. However, the retrieval space in existing robotic CMR approaches often consists of only one modality, which limits the performance of the robot. In this paper, we propose a novel CMR model that incorporates three different modalities, i.e., visual, audio, and tactile, for enhanced multi-modal object retrieval, referred to as VAT-CMR. In this model, multi-modal representations are first fused to provide a holistic view of object features. Then, to mitigate the semantic gaps between representations of different modalities, a dominant modality is selected during the classification training phase to improve the distinctiveness of the representations and enhance the retrieval performance. To evaluate our proposed approach, we conducted a case study and the results demonstrate that our VAT-CMR model surpasses competing approaches. Further, our proposed dominant modality selection significantly enhances cross-retrieval accuracy.

Intelligent control methods have led to a significant simplification of the robotic arm modeling and control tuning process, and thus they have been widely used. To further improve the precision of robotic arm motion control, this paper proposes a robotic arm motion control strategy based on a cascaded feature-enhanced elastic-net broad learning system (CFE-EN-BLS). This will fully extract data features to improve motion control accuracy. Moreover, ElasticNet regression is introduced to reduce feature redundancy. Finally, Lyapunov stability theory is introduced to constrain the learning parameters of the proposed learning method to enhance the convergence of the control strategy. The simulation and experiment show that the proposed control strategy can realize high-precision trajectory tracking control of the robotic arm.

Net-winged midge larvae (genus Liponeura) can achieve robust attachment and crawl on the slippery surface in the fast stream with their powerful abdominal suckers. The rigid spine-like structures distributed in the sucker cavity called microtrichia, have been proven to be crucial in the adhesion process. In this work, we carry out a design of the biomimetic sucker with the spine-like structures and then implement various tests using biomimetic suckers to verify the adhesion capacity enhancement brought by spine-like structures. Finally, we assemble the suckers in a quadruped crawling robot capable of locomotion in both aerial and aquatic environments with a speed of 56.4 mm/s (0.225 BL/s) and can crawl upstream against a turbulent flow with a speed of 38.2 mm/s (0.152 BL/s). This study will inspire biomimetic design in future robotics, and pave the way for future robots to realize long-term observation and monitoring in complex environments.

During vitreoretinal surgery, surgeons are required to precisely manipulate surgical tools within a confined workspace of an eye, which is roughly 2.5 cm spherical in shape. Because the surgical view can only be obtained by a microscope placed above the eyeball through the pupil, the eyeball needs to be moved or rotated during the operation to see a larger portion of the retina. At this point, general Remote Center of Motion (RCM) mechanisms require additional actuators or manual modification. On the other hand, a programmable RCM mechanism can reduce surgery time without a physical alignment procedure. This study introduces a novel six-degree-of-freedom (DoF) programmable RCM mechanism capable of generating the RCM at random positions in 3D space. Our approach combines two planar 5-bar linkage mechanisms placed in parallel, creating a double-layered configuration to establish the programmable RCM mechanism. We optimized the workspace of each planar mechanism to a customized workspace for a general eyeball model using genetic algorithms, focusing on maximizing the manipulability of the target workspace. The Phantom Omni device was utilized as a remote controller to remotely control the proposed mechanism in a transparent eyeball model with a diameter of 4 cm. Evaluation of the functionality of the programmable RCM mechanism at various RCM points showed that the overall error was less than 1 millimeter. The repeatability of the mechanism was tested and showed an accuracy of about 127 micrometers.

Telemanipulation enables humans to perform tasks in dangerous environments without exposing them to any risk. The COVID-19 pandemic sadly showed, that these environments can also include the treatment and interaction with infected patients. Since human-robot interactions demand for low interaction forces yet high precision, telemanipulation often results in a high mental workload for the operator. To overcome this, we present a virtual guidance approach to perform telemanipulated insertion tasks. A nasopharyngeal swap sampling procedure is taken as use case. We extend our previously presented approach by adding an additional position fixture, introducing distance-dependent variable stiffness values and guaranteeing stability using energy tanks. Based on RGB-D data, the operator is guided towards a desirable insertion line while approaching the nostril. The distance-dependent stiffness values increase the smoothness of the fixture. Since variable stiffness values can result in unstable behavior, energy tanks for the fixtures are introduced. Experiments show the improvements compared to our previous approach. Further, a comparison between guided and unguided samplings performed by an expert user gives a first impression of the improvements resulting from the fixture.

6-DoF grasp detection of small-scale grasps is crucial for robots to perform specific tasks. This paper focuses on enhancing the recognition capability of small-scale grasping, aiming to improve the overall accuracy of grasping prediction results and the generalization ability of the network. We propose an enhanced receptive field method that includes a multi-radii cylinder grouping module and a passive attention module. This method enhances the receptive field area within the graspable space and strengthens the learning of graspable features. Additionally, we design a graspable balance sampling module based on a 3D segmentation network, which enables the network to focus on features of small objects, thereby improving the recognition capability of small-scale grasping. Our network achieves state-of-the-art performance on the GraspNet-1Billion dataset, with an overall improvement of approximately 10% in average precision@k (AP). Furthermore, we deployed our grasp detection model on pybullet grasping platform and in real-world scenarios, which validates the effectiveness of our method.

LiDAR-based 3D perception is a focal point in autonomous vehicle research due to its efficacy in real-world environments and falling costs. However, recent research reveals challenges with LiDAR sensing under corruptions that occur due to adverse weather conditions and sensor-level errors, known as common corruptions. In particular, the majority of these corruptions lead to sparsity or noise in LiDAR point clouds, degrading the performance of downstream perception tasks. To address this, we propose a blind inpainting method named 3DR-DIFF, utilizing diffusion networks to reconstruct and segment corrupted point clouds. 3DR-DIFF comprises two key components: a corrupted region prediction network, acting as a binary mask predictor, and a conditional diffusion network. The evaluation results demonstrate that the 3DR-DIFF is able to reconstruct the LiDAR samples with a depth error of less than 0.56 mean absolute error (MAE) and an intensity error of 0.02 MAE, along with an average segmentation performance of 0.43 mean intersection over union. Furthermore, benchmarking results highlight that 3DR-DIFF outperforms state-of-the-art methods in reconstructing LiDAR beam-missing scenarios, exhibiting an approximately 9.2% lower error for a degradation of 1 MAE.

Reliable multimodal sensor fusion algorithms require accurate spatiotemporal calibration. Recently, targetless calibration techniques based on implicit neural representations have proven to provide precise and robust results. Nevertheless, such methods are inherently slow to train given the high computational overhead caused by the large number of sampled points required for volume rendering. With the recent introduction of 3D Gaussian Splatting as a faster alternative to implicit representation methods, we propose to leverage this new rendering approach to achieve faster multi-sensor calibration. We introduce 3DGS-Calib, a new calibration method that relies on the speed and rendering accuracy of 3D Gaussian Splatting to achieve multimodal spatiotemporal calibration that is accurate, robust, and with a substantial speed-up compared to methods relying on implicit neural representations. We demonstrate the superiority of our proposal with experimental results on sequences from KITTI-360, a widely used driving dataset.

We present the design, implementation, and evaluation of 3D-BLUE, an ultra-low-power underwater 3D localization system that can be deployed on compact robots to accurately localize them in shallow underwater environments. 3D-BLUE’s design introduces two core components. First, it adapts a recent ultra-low-power underwater acoustic communication technology (called piezo-electric backscatter) to the underwater robotics localization problem; specifically, it integrates backscatter nodes into the underwater robot and uses them for localizing it. Second, it leverages the physical properties of the backscatter technology to efficiently extract spatio-temporal-spectral features from the backscatter signal; using these features, it devises a particle-filter-based algorithm to localize the corresponding robot accurately in challenging shallow-water environments. We implemented an end-to-end prototype of 3D-BLUE on a BlueROV2 robot and custombuilt backscatter localization system, and evaluated it in dozens of experimental trials in a pool. Our results demonstrate that 3D-BLUE can localize the robot with an accuracy of around 0.25m at close range and an accuracy of around 1.4m at a range of 10m. This high localization accuracy opens important commercial, naval, and environmental applications in challenging shallow-water environments such as shores, rivers, pools, and narrow waterways.

Ultrasound (US) imaging of the carotid artery (CA) is a non-invasive diagnostic tool widely used in the medical field to assess the condition of the carotid artery, thereby predicting the risk of cardiovascular and cerebrovascular diseases. However, implementing this method in primary healthcare can be challenging due to the requirement for professionally trained sonographers. With the adoption of US robotic devices, the probe pose can be acquired while scanning, offering the possibility for 3D reconstruction and providing analyses that are not dependent on operator experience. This article introduces a method to semi-automatically acquire serialized US images of the common carotid artery (CCA). The method involves a specially designed robotic device built with a 6-RSU parallel mechanism, which is controlled according to robot pose, force sensor data and synchronous US images. To validate the images acquired, a method is proposed to segment the intima-media of CCA and calculate the intima-media thickness (IMT), which is a key indicator for cerebrovascular events prediction. After that, we propose an algorithm to reconstruct CCA into 3D voxel data with patient movement and cardiac cycle compensated, and a longitudinal view US image of CCA can be resliced from the voxel. The methods are tested on human subjects and the results indicate that the system and workflow can provide both quantitative and qualitative information of CCA for further diagnosis.

With the rapid advancement of hardware and software technologies, research in autonomous driving has seen significant growth. The prevailing framework for multi-sensor autonomous driving encompasses sensor installation, perception, path planning, decision-making, and motion control. At the perception phase, a common approach involves utilizing neural networks to infer 3D bounding box (Bbox) attributes from raw sensor data, including classification, size, and orientation.In this paper, we present a novel attribute and its corresponding algorithm: 3D object visibility. By incorporating multi-task learning, the introduction of this attribute, visibility, negligibly affects the model's effectiveness and efficiency. Our proposal of this attribute and its computational strategy aims to expand the capabilities for downstream tasks, thereby enhancing the safety and reliability of real-time autonomous driving in real-world scenarios.

Camera-based 3D object detectors, prized for their broader applicability and cost-effectiveness compared to LiDAR sensors, still grapple with the inherently ill-posed nature of depth extraction from images. In this work, we present a novel approach that employs a transformer-based backbone and a fused geometry volume to bolster feature richness and elevate detection accuracy. Firstly, we propose the Stereo Pyramid Transformer backbone to extract features from stereo images, which can capture global information and establish cross-image semantic connections. Then, to tackle the challenge posed by small baseline binocular cameras, we propose to fuse stereo geometry volumes constructed by Stereo Plane Sweeping Volume (SPSV), Monocular Semantic Volume (MSV), and Lifted Volume (LV) to create finely detailed feature volumes. Through extensive experiments on both the KITTI and our datasets, our approach not only surpasses all existing transformer-based stereo 3D detection methods but also marks a significant milestone by achieving comparable performance with the leading CNN-based 3D detectors for the first time.

While visual sensing is often the predominant modality for a robot to localize objects in the environment, tactile and force sensing become crucial when objects are occluded, poorly visible, or buried. However, existing works on locating buried objects rely solely on force measurements at a single contact point on the robot end-effector, making 3D localization very challenging. This paper presents an alternative approach using a tactile sensor that measures both normal and shear forces (i.e. 3-axis) on distributed points; three Long Short-Term Memory (LSTM) models are trained with real-world data to perform real-time 3D localization (i.e. distance, direction and depth) of an object buried within a granular material. Our experimental results suggest that measuring both normal and shear forces (instead of just normal) on distributed contact points (instead of only one point) is essential for the accurate 3D localization of buried objects.

The use of mobile robots has become increasingly common in multiple areas of daily life. To increase their autonomy for performing various tasks, efficient navigation skills are essential. The most crucial component of such navigation is the ability to calculate a global path between two points. The global path planning problem for mobile robots is typically limited to two-dimensional environments, in which the environment is projected onto a planar surface. While this approach works well in structured environments like industrial settings, it may not be suitable for all applications of mobile robots. With modern walking robots, capable of navigating complex terrain, more advanced path planning approaches are necessary. This work proposes a path-planning approach that utilizes the entire three-dimensional space, allowing for navigation in even the most challenging terrain. The central idea is to extend a traditional A* path planner to work directly on a fast volumetric map structure to generate optimal paths through the environment. Multiple optimizations and adjustments are introduced to improve the algorithm’s performance. By applying morphology operators to sparse maps, sensor inaccuracies during the map construction are mitigated. Additionally, adjustments are made to handle the added complexity introduced by the extra search space dimension and to comply with the limitations of autonomous walking robots. This is paired with an efficient caching strategy to enhance the overall path-planning speed. The capability of the path planning approach is evaluated using both artificial and real-world maps. The results demonstrate that this approach shows great potential for enabling mobile ground robots to autonomously navigate even the most demanding terrains utilizing the entire three-dimensional space.

This paper presents a novel approach for affordance-informed robotic manipulation by introducing 3D keypoints to enhance the understanding of object parts’ functionality. The proposed approach provides direct information about what the potential use of objects is, as well as guidance on where and how a manipulator should engage, whereas conventional methods treat affordance detection as a semantic segmentation task, focusing solely on answering the what question. To address this gap, we propose a Fusion-based Affordance Keypoint Network (FAKP-Net) by introducing 3D keypoint quadruplet that harnesses the synergistic potential of RGB and Depth image to provide information on execution position, direction, and extent. Benchmark testing demonstrates that FAKP-Net outperforms existing models by significant margins in affordance segmentation task and keypoint detection task. Real-world experiments also showcase the reliability of our method in accomplishing manipulation tasks with previously unseen objects. Our source code and video demo will be public.

Open Spina Bifida (OSB) is a congenital neural tube defect where a major component of the procedure to repair the defect involves the closure of a lesion wound through suturing. For a minimally invasive approach, tools entering the uterus to access the fetus should be as thin as possible to minimize maternal risk. This work presents the design of a 3 degrees-of-freedom, 2mm diameter tool wrist with a bending range of motion from 0° to 90°. This wrist is capable of generating up to 2N of force measured from the end of the wrist and achieving a bending curvature of 107m-1 (9.35mm bending radius). A pseudo-rigid body kinematic model has been implemented for the control of this tool with a protocol for singularity mitigation and avoidance. Timed teleoperation studies explicitly demonstrate that the tool is able to reliably execute suturing with a fastest achieved time of under 3 minutes for a simple interrupted suturing technique.

Robotic branch pruning, a rapidly growing field addressing labor shortages in agriculture, requires detailed perception of branch geometry and topology. However, point clouds obtained in agricultural settings often lack completeness, limiting pruning accuracy. This work addressed point cloud quality via a closed-loop approach, (Real2Sim)−1. Leveraging a Real-to-Simulation (Real2Sim) data generation pipeline, we generated simulated 3D apple trees based on realistically characterized apple tree information without manual parameterization. These 3D trees were used to train a simulation-based deep model that jointly performs point cloud completion and skeletonization on real-world partial branches, without extra real-world training. The Sim2Real qualitative results showed the model’s remarkable capability for geometry reconstruction and topology prediction. Additionally, we quantitatively evaluated the Sim2Real performance by comparing branch-level trait characterization errors using raw incomplete data and the best complete data. The Mean Absolute Error (MAE) reduced by 75% and 8% for branch diameter and branch angle estimation, respectively, which indicates the effectiveness of the Real2Sim data in a zero-shot generalization setting. The characterization improvements contributed to the precision and efficacy of robotic branch pruning.