IROS 2023 Papers

Locomotion in the wild requires the quadruped robot to have strong capabilities in adaptation and robustness. The deep reinforcement learning (DRL) exhibits the huge potential in environmental adaptability, while its stability issues remain open. On the other hand, the quadruped robot dynamic model contains a lot of useful information that is beneficial to the robust control. The combination of DRL with model-based control may take both strengths and hold promises in better robustness. In this paper, the DRL and the proposed model-based controller are firmly integrated in a novel manner such that the proposed model-based controller is able to rectify the gait commands generated by DRL based on the system dynamic model so as to enhance the robustness of the quadruped robot against the external disturbances. Besides, a potential energy function is introduced to achieve the compliant contact. The stability of the proposed method is ensured in terms of passivity analysis. Several physical experiments are carried out to verify the performance of the proposed method.

Recently several fusion and switching based approaches have been presented to solve the problem of Visual Place Recognition. In spite of these systems demonstrating significant boost in VPR performance they each have their own set of limitations. The multi-process fusion systems usually involve employing brute force and running all available VPR techniques simultaneously while the switching method attempts to negate this practise by only selecting the best suited VPR technique for given query image. But switching does fail at times when no available suitable technique can be identified. An innovative solution would be an amalgamation of the two otherwise discrete approaches to combine their competitive advantages while negating their shortcomings. The proposed, Switch-Fuse system, is an interesting way to combine both the robustness of switching VPR techniques based on complementarity and the force of fusing the carefully selected techniques to significantly improve performance. Our system holds a structure superior to the basic fusion methods as instead of simply fusing all or any random techniques, it is structured to first select the best possible VPR techniques for fusion, according to the query image. The system combines two significant processes, switching and fusing VPR techniques, which together as a hybrid model substantially improve performance on all major VPR data sets illustrated using PR curves.

Perception systems for ornithopters face severe challenges. The harsh vibrations and abrupt movements caused during flapping are prone to produce motion blur and strong lighting condition changes. Their strict restrictions in weight, size, and energy consumption also limit the type and number of sensors to mount onboard. Lightweight traditional cameras have become a standard off-the-shelf solution in many flapping-wing designs. However, bioinspired event cameras are a promising solution for ornithopter perception due to their microsecond temporal resolution, high dynamic range, and low power consumption. This paper presents an experimental comparison between frame-based and an event-based camera. Both technologies are analyzed considering the particular flapping-wing robot specifications and also experimentally analyzing the performance of well-known vision algorithms with data recorded onboard a flapping-wing robot. Our results suggest event cameras as the most suitable sensors for ornithopters. Nevertheless, they also evidence the open challenges for event-based vision on board flapping-wing robots.

Multi-Robot Systems (MRS) are transforming diverse domains like logistics, cargo management, and agriculture. However, ensuring that the behavior is correct under various network conditions within such complex environments is challenging, and meeting the desired automation goals is difficult. We propose the CORNET 2.0 co-simulation framework to jointly and accurately simulate multi-agent robotic systems within physical environments and the communication network models within such environments. Our modular framework allows diverse robot and network models to seamlessly integrate to simulate the robot's autonomy, physical space, and network features, such as latency, throughput, and loss intrinsic to the network topology and communication technology. A key novelty of CORNET 2.0 is its accurate synchronizing of mobility and time, which ensures that the physical location of a robot at a point in time, and the network properties and packets that flow from that location, are aligned. This is vital to model and validate MRS coordination algorithms that rely on network interactions. We provide a detailed evaluation of CORNET 2.0 in modeling real-world MRS use cases, such as leader-follower and warehouse environments, that help highlights the benefits.

For many manipulation tasks in environments such as laboratory or a kitchen, the presence of two robot arms is important to enable collaborative tasks requiring two arms (e.g. lid removal or tool use) or to improve the efficiency of scheduling of tasks. Currently, the development of multi-arm manipulation solutions has largely focused on 6 degrees of freedom articulated robot arms. However, cartesian robots have many advantages, including their precision, reliability, efficiency, and simple path planning. By developing a cartesian platform such that the end effectors of two mirrored systems can interact freely without collisions in 5 degrees of freedom, we can leverage the advantages of cartesian robots (high precision, simple planning, and low-cost hardware) and show robot cooperation. We equip each robot with end-effectors with different skills to increase the range of tasks the robots can cooperatively complete. To exploit this robotic hardware, we have developed a task-allocation and path-planning algorithm that enables these two mirror robots to work together to solve tasks collaboratively, exploiting the different skills and workspace of the two robots. We show how this robot can be used for cooperative tasks in lab automation, including pick and place, unscrewing vial caps, liquid pouring, and weighing. These demonstrate the feasibility and capabilities of the proposed robotic system for cooperative automation using cartesian robots.

Hands are one of the most intricate elements of a humanoid due to their role as end-effectors interacting with their non-linear surrounding environment. This paper aims to present the design of a bioinspired underactuated robotic hand with an improved dexterity that is capable of adaptive grasping and manipulation of a wide-range of objects using a dual-tendon mechanism. The proposed design is focused on the key elements of scalability, modularity, ease of fabrication and cost efficiency to meet several imperative constraints of space applications. These features are achieved by introducing a novel actuation mechanism, manufacturing methods, and component design. In particular, monolithic finger modules are fabricated by fusing and integrating both hard and soft materials analogous to bones wrapped in muscles using economical and readily-available materials and machines (intermediate 3D printer). Weight-to-power ratio, actuation optimisation, design trade-offs, and various potential applications of the proposed adaptive hand is discussed in this paper. Furthermore, the prototype is subjected to evaluation of its performance in different scenarios that ultimately confirms its improved dexterity and gripping power compared to the literature.

A robotic finger is successfully designed, fabricated, analyzed, and examined. The finger consists of bones, joint ligaments, and an extensor hood. Driven by two tendons, it is two degrees on the freedom finger. Although the behavior of this design is not uniform, it provides useful dexterity, sensitivity, and versatility. The artificial bone is lightweight and compact. The actuation is backdrivable in good visibility. Tendon muscles effectively actuate the finger in a grasp direction. This exceptional behavior has the potential to be suitable for a flexible task and may lead to a next-generation gripper and prosthetic hand.

This paper addresses the lack of research on periodic reinforcement learning for physical robot control by presenting a 3-phase periodic Bayesian reinforcement learning method for uncertain environments. Drawing on cognition theory, the proposed approach achieves effective convergence with fewer training episodes. The coach-based demonstration phase narrows the search space and establishes a foundation for a coarse-to-fine control strategy. The reconnaissance phase enhances adaptability by discovering a valuable global repre-sentation, and the operation phase produces accurate robotic control by applying the learned representation and periodically updating local information. Comparative analysis with state-of-the-art methods validates the efficacy of our approach on exemplar control tasks in simulation and a biomedical project involving a simulated cranial window task.

We present a novel technique to estimate the 6D pose of objects from single images where the 3D geometry of the object is only given approximately and not as a precise 3D model. To achieve this, we employ a dense 2D-to-3D correspondence predictor that regresses 3D model coordinates for every pixel. In addition to the 3D coordinates, our model also estimates the pixel-wise coordinate error to discard correspondences that are likely wrong. This allows us to generate multiple 6D pose hypotheses of the object, which we then refine iteratively using a highly efficient region-based approach. We also introduce a novel pixel-wise posterior formulation by which we can estimate the probability for each hypothesis and select the most likely one. As we show in experiments, our approach is capable of dealing with extreme visual conditions including overexposure, high contrast, or low signal-to-noise ratio. This makes it a powerful technique for the particularly challenging task of estimating the pose of tumbling satellites for in-orbit robotic applications. Our method achieves state-of-the-art performance on the SPEED+ dataset and has won the SPEC2021 post-mortem competition. Code, trained models, and the used satellite model will be made publicly available.

To automatically localize a target object in an image is crucial for many computer vision applications. To represent the 2D object, ellipse labels have recently been identified as a promising alternative to axis-aligned bounding boxes. This paper further considers 3D-aware ellipse labels, i.e., ellipses which are projections of a 3D ellipsoidal approximation of the object, for 2D target localization. Indeed, projected ellipses carry more geometric information about the object geometry and pose (3D awareness) than traditional 3D-agnostic bounding box labels. Moreover, such a generic 3D ellipsoidal model allows for approximating known to coarsely known targets. We then propose to have a new look at ellipse regression and replace the discontinuous geometric ellipse parameters with the parameters of an implicit Gaussian distribution encoding object occupancy in the image. The models are trained to regress the values of this bivariate Gaussian distribution over the image pixels using a statistical loss function. We introduce a novel non-trainable differentiable layer, E-DSNT, to extract the distribution parameters. Also, we describe how to readily generate consistent 3D-aware Gaussian occupancy parameters using only coarse dimensions of the target and relative pose labels. We extend three existing spacecraft pose estimation datasets with 3D-aware Gaussian occupancy labels to validate our hypothesis. Labels and source code are publicly accessible here: https://cvi2.uni.lu/3d-aware-obj-loc/.

Robotic pruning of dormant grapevines is an area of active research in order to promote vine balance and grape quality, but so far robotic efforts have largely focused on planar, simplified vines not representative of commercial vineyards. This paper aims to advance the robotic perception capabilities necessary for pruning in denser and more complex vine structures by extending plant skeletonization techniques. The proposed pipeline generates skeletal grapevine models that have lower reprojection error and higher connectivity than baseline algorithms. We also show how 3D and skeletal information enables prediction accuracy of pruning weight for dense vines surpassing prior work, where pruning weight is an important vine metric influencing pruning site selection.

In robotic laser surgery, shape prediction of an one-shot ablation crater is an important problem for minimizing errant overcutting of healthy tissue during the course of pathological tissue resection and precise tumor removal. Since it is difficult to physically model the laser-tissue interaction due to the variety of optical tissue properties, complicated process of heat transfer, and uncertainty about the chemical reaction, we propose a 3D crater prediction model based on an entirely data-driven method without any assumptions of laser settings and tissue properties. Based on the crater prediction model, we formulate a novel robotic laser planning problem to determine the optimal laser incident configuration, which aims to create a crater that aligns with the surface target (e.g. tumor, pathological tissue). To solve the one-shot ablation crater prediction problem, we model the 3D geometric relation between the tissue surface and the laser energy profile as a non-linear regression problem that can be represented by a single-layer perceptron (SLP) network. The SLP network is encoded in a novel kinematic model to predict the shape of the post-ablation crater with an arbitrary laser input. To estimate the SLP network parameters, we formulate a dataset of one-shot laser-phantom craters reconstructed by the optical coherence tomography (OCT) B-scan images. To verify the method. The learned crater prediction model is applied to solve a simplified robotic laser planning problem modelled as a surface alignment error minimization problem. The initial results report about (91.2\pm 3.0)\%(91.2\pm 3.0)\% 3D-crater-Intersection-over-Union (3D-crater-IoU) for the 3D crater prediction and an average of about 98.0% success rate for the simulated surface alignment experiments.

Based on the neural radiance fields (NeRF), we present a pipeline for generating novel views from a single 360° panoramic image. Prior research relied on the neighborhood interpolation capability of multi-layer perceptions to complete missing regions caused by occlusion. This resulted in artifacts in their predictions. We propose 360FusionNeRF, a semi-supervised learning framework that employs geometric supervision and semantic consistency to guide the progressive training process. Firstly, the input image is reprojected to 360° images, and depth maps are extracted at different camera positions. In addition to the NeRF color guidance, the depth supervision enhances the geometry of the synthesized views. Furthermore, we include a semantic consistency loss that encourages realistic renderings of novel views. We extract these semantic features using a pre-trained visual encoder CLIP, a Vision Transformer (ViT) trained on hundreds of millions of diverse 2D photographs mined from the web with natural language supervision. Experiments indicate that our proposed method is capable of producing realistic completions of unobserved regions while preserving the features of the scene. 360FusionNeRF consistently delivers state-of-the-art performance when transferring to synthetic Structured3D dataset (PSNR ~ 5%, SSIM ~3% LPIPS ~13%), real-world Matterport3D dataset (PSNR ~3%, SSIM ~3% LPIPS ~9%) and Replica360 dataset (PSNR ~8%, SSIM ~2% LPIPS ~18%). We provide the source code at https://github.com/MetaSLAM/360FusionNeRF.

When a conventional robot arm, driven by magnetic motors and reduction gears, operates at high speed, it can produce an excessive force due to unexpected contact. Toward the development of a safe robot arm, the present study investigated a 2-degree-of-freedom robot arm driven by direct-drive electrostatic film motors. The motors operate synchronously, except when they are subjected to an excessive force. This loss of synchronicity is referred to as a “step-out” process, and in the present study it was used to produce a variable torque limiter. By setting appropriate limits, contact forces caused by unexpected collisions could be suppressed. The torque limiter was applied to a robot arm with a length of 50 cm. Experiments showed that the arm was impact-resistant; when it was struck by a hammer, the torque limiter was activated and suppressed the collision force, allowing the robot arm to successfully resume its motion. When the robot arm unexpectedly hit an obstacle while moving at 1.2 m/s, the collision force was maintained at 21 N by the torque limiter, which is sufficiently small for a high-speed collision. The experiments verified the effectiveness of the electrostatic film motor as a variable torque limiter to realize safe contact.