IROS 2023 论文列表

Simulating camera sensors is a crucial task in autonomous driving. Although neural radiance fields are exceptional at synthesizing photorealistic views in driving simulations, they still fail to generate extrapolated views. This paper proposes to incorporate map priors into neural radiance fields to synthesize out-of-trajectory driving views with semantic road consistency. The key insight is that map information can be utilized as a prior to guiding the training of the radiance fields with uncertainty. Specifically, we utilize the coarse ground surface as uncertain information to supervise the density field and warp depth with uncertainty from unknown camera poses to ensure multi-view consistency. Experimental results demonstrate that our approach can produce semantic consistency in deviated views for vehicle camera simulation. The supplementary video can be viewed at https://youtu.be/jEQWr-Rfh3A.

Colloidal particles trapped by a focused laser at the air-liquid interface provide an interesting assembly dynamic. In this study, we demonstrated manipulating optical force-induced swarms via dynamic locomotion of assemblies built with holographic optical tweezers. This manipulation approach builds the foundation for autonomous control of building assemblies at the air-liquid interface, which is the first time optical micro-robots have performed this feat. Our proposed semi-autonomous control allows users to produce small dynamic secondary assemblies at the interface, which are transported to and merged with a main static assembly. This static-dynamic approach grows assemblies up to ~2.1 times larger than conventional methods. Manipulation and control of large-scale optical force-induced assemblies in real-time to create re-configurable swarms has the potential to lead the development of new technology and approaches for complex tasks, such as the development of new material, transportation of biological matter, studying biofilm formation created by bacteria colonies at the air-liquid interface, and more.

To implement successful bipedal locomotion in a robot, its center of pressure (CoP) is placed on a supporting area. To achieve locomotion, many studies have focused on the lower body. Given that the human upper body has a large mass and its behavior influences locomotion even in the case of the robot, this study investigates the effect of the upper body, which contains moving arms and a twisting trunk, on CoP. The dynamics is analyzed using a simple model that has a passive viscoelastic trunk joint around the vertical axis and arms that oscillates back and forth. From the derivations of CoP and trunk joint trajectory, three important findings are made: (i) the CoP oscillates along the lateral direction only when the arm swings in an anterior-posterior asymmetric manner, (ii) the trajectories of CoP and trunk joint are in anti-phase, (iii) the phase of CoP along the lateral direction is influenced by the swinging cycle and viscoelasticity of the trunk joint, the mechanical elements of the upper body. An experiment using a robot verified the first and second findings, and simulation verified the last finding. The last finding will contribute to making a feedback controller that converges to the desired phase, which is an important factor for successful bipedal locomotion.

Vision-based perception has become prevalent in robotic applications, especially in those where the control loop relies on visual data, such as visual servoing. For those applications, ensuring that the features or target object remain visible to the camera is critical, necessitating visibility-aware control. In this paper, we propose a method to guarantee the visibility of a dynamic object using a constrained kinematic controller and Vector Field Inequalities (VFIs) to include a linear visibility constraint. Unlike existing methods, we introduce constraints into the kinematic controller to ensure the target's visibility without needing a trajectory optimizer or local planner. Our method maintains the target object in the camera field of view (FoV) by representing the FoV with four infinite planes and maintaining the distance between the target object and each plane higher than a predefined distance. We evaluated the proposed approach using a mobile manipulator in two simulations involving cluttered environments: the first scenario involves a stationary target object, whereas the second scenario presents a more challenging workspace involving a moving target. Our results demonstrate that the proposed approach successfully maintains the target within the FoV while avoiding obstacles in the workspace, showing the potential of our method to improve the safety and reliability of visual-servoing-based robotic systems.

Magnetic microrobots exhibit enormous potential in targeted drug delivery owing to the remote wireless manipulation and minimum invasion for medical treatment. High degree of freedom offers the magnetic propelled robots extraordinary application prospect since they can be controlled precisely when different magnetic fields sources working cooperatively. However, the biocompatibility of microrobots have attracted sustained and general concern. Therefore, it is highly necessary to develop a promising carrier with high biocompatibility and investigate the mechanism of drug loading-release triggered by special microenvironment in the targeted region. In this paper, we proposed a magnetically controlled cell robots (MCRs) based on macrophages propelled by a rotating magnetic field. The innovative MCRs exhibit good biocompatibility and low toxicity by optimizing the concentration of polylysine-coated Fe nanoparticles (PLL@FeNPs) to 40 µg/mL. These MCRs loaded with murine interleukin-12 (IL-12), murine chemokine (C-C motif) ligand 5 (CCL-5), and murine C-X-C motif chemokine ligand 10 (CXCL-10) which can stimulate T cell differentiation and recruitment of monocytes, respectively. The macrophages showed an obvious M1-polarization tendency of macrophages to phagocytose intracellular pathogens and resist the growth of tumor cells. Under the control of a magnetic propelling system composed of 3 pairs of Helmholtz coil, the cell robot can be propelled wirelessly and moved along a predefined path with high accuracy. Moreover, the MCRs could approach to cancer cells and stop at places of interest in vitro. In conclusion, we have accomplished the preliminary construction of a targeted drug delivery system which displays great immune-enhancing potential for targeted drug delivery.

Indoor magnetic fields are a combination of Earth's magnetic field and disruptions induced by ferromag-netic objects, such as steel structural components in buildings. As a result of these disruptions, pervasive in indoor spaces, mag-netic field data is often omitted from navigation algorithms in indoor environments. This paper leverages the spatially-varying disruptions to Earth's magnetic field to extract positional information for use in indoor navigation algorithms. The algorithm uses a rate gyro and an array of four magnetometers to estimate the robot's pose. Additionally, the magnetometer array is used to compute attitude-invariant measurements associated with the magnetic field and its gradient. These measurements are used to detect loop closure points. Experimental results indicate that the proposed approach can estimate the pose of a ground robot in an indoor environment within meter accuracy.

A magnetically controlled steerable robot with the capability of flexible navigation and intraoperative ultrasonic imaging and position sensing in neurosurgery is introduced in this paper. The robot system uses a piezoelectric transducer as the ultrasonic imaging beacon and applies a permanent magnet as the actuation unit, which can steer the flexible robot following a planned complex path to access a target in the brain. In order to enable the robot to navigate flexibly within tissues, a method is proposed to enhance magnetic actuation force by utilizing an array of small magnetic blocks instead of using a large magnet. The improvement of the array for magnetic field distribution was verified by simulation. Simulation studies also involved modeling the ultrasonic transmitting and receiving functions of the piezoelectric transducer to verify the feasibility of using them for intraoperative imaging. In addition, a prototype of the robot is fabricated and tested. The feasibility of the proposed magnetically controlled steerable robot for navigation in soft tissue is verified.

This article proposes a novel indoor magnetic field-based place recognition algorithm that is accurate and fast to compute. For that, we modified the generalized “Hough Transform” to process magnetic data (MagHT). It takes as input a sequence of magnetic measures whose relative positions are recovered by an odometry system and recognizes the places in the magnetic map where they were acquired. It also returns the global transformation from the coordinate frame of the input magnetic data to the magnetic map reference frame. Experimental results on several real datasets in large indoor environments demonstrate that the obtained localization error, recall, and precision are similar to or are better than state-of-the-art methods while improving the runtime by several orders of magnitude. Moreover, unlike magnetic sequence matching-based solutions such as DTW, our approach is independent of the path taken during the magnetic map creation.

Machine learning-based approaches for soft robot proprioception have recently gained popularity, in part due to the difficulties in modeling the relationship between sensor signals and robot shape. However, to date, there exists no systematic analysis of the required design choices to set up a machine learning pipeline for soft robot proprioception. Here, we present the first study examining how design choices on different levels of the machine learning pipeline affect the performance of a neural network for predicting the state of a soft robot. We address the most frequent questions researchers face, such as how to choose the appropriate sensor and actuator signals, process input and output data, deal with time series, and pick the best neural network architecture. By testing our hypotheses on data collected from two vastly different systems–an electrically actuated robotic platform and a pneumatically actuated soft trunk–we seek conclusions that may generalize beyond one specific type of soft robot and hope to provide insights for researchers to use machine learning for soft robot proprioception.

Local feature detection and description are two essential steps in many visual applications. Most learned local feature methods require high-quality labeled data to achieve superior performance, but such labels are often expensive. To address this problem, we propose MUFeat, an unsupervised learning framework of jointly learning local feature detector and descriptor without requirement of ground-truth correspondences. MUFeat trains the network based on the putative matches from the pretrained model and two proposed unsupervised loss functions. Furthermore, the MUFeat framework includes a pyramidal feature hierarchy network to obtain keypoints and descriptors from feature maps. Experiments indicate that MUFeat outperforms most state-of-the-art supervised learning methods on image matching, medical image registration and visual localization tasks.

Social robots have gained widespread attention for their potential to assist people in diverse domains, such as living assistance and logistics transportation. Human-accompanying, i.e., walking side-by-side with a person, is an expected and essential capability for social robots. However, due to the complexity of motion coordination between the target person and the mobile robot, the accompanying action is still unstable. In this study, we propose a human-accompanying control strategy to improve the motion coordination for better practicability of the human-accompanying robot. Our approach allows the robot to adapt to the motion variations of the target person and avoid obstacles while accompanying them. First, a human-robot interaction model based on the separation-bearing-orientation scheme is developed to ascertain the relative position and orientation between the robot and the target person. Then, a human-accompanying controller based on behavioral dynamics and model predictive control (MPC) is designed to avoid obstacles and simultaneously track the direction and velocity of the target person. Experimental results indicate that the proposed method can effectively achieve side-by-side accompanying by simultaneously controlling the relative position, direction, and velocity between the target person and robot.

We present MOTLEE, a distributed mobile multi-object tracking algorithm that enables a team of robots to collaboratively track moving objects in the presence of localization error. Existing approaches to distributed tracking make limiting assumptions regarding the relative spatial relationship of sensors, including assuming a static sensor network or that perfect localization is available. Instead, we develop an algorithm based on the Kalman-Consensus filter for distributed tracking that properly leverages localization uncertainty in collaborative tracking. Further, our method allows the team to maintain an accurate understanding of dynamic objects in the environment by realigning robot frames and incorporating frame alignment uncertainty into our object tracking formulation. We evaluate our method in hardware on a team of three mobile ground robots tracking four people. Compared to previous works that do not account for localization error, we show that MOTLEE is resilient to localization uncertainties, enabling accurate tracking in distributed, dynamic settings with mobile tracking sensors.

In this paper, we present a novel method for mobile manipulators to perform multiple contact-rich manipulation tasks. While learning-based methods have the potential to generate actions in an end-to-end manner, they often suffer from insufficient action accuracy and robustness against noise. On the other hand, classical control-based methods can enhance system robustness, but at the cost of extensive parameter tuning. To address these challenges, we present MOMA-Force, a visual-force imitation method that seamlessly combines representation learning for perception, imitation learning for complex motion generation, and admittance whole-body control for system robustness and controllability. MOMA-Force enables a mobile manipulator to learn multiple complex contact-rich tasks with high success rates and small contact forces. In a real household setting, our method outperforms baseline methods in terms of task success rates. Moreover, our method achieves smaller contact forces and smaller force variances compared to baseline methods without force imitation. Overall, we offer a promising approach for efficient and robust mobile manipulation in the real world. Videos and more details can be found on https://visual-force-imitation.github.io.

With the recent advances in autonomous driving and the decreasing cost of LiDARs, the use of multimodal sensor systems is on the rise. However, in order to make use of the information provided by a variety of complimentary sensors, it is necessary to accurately calibrate them. We take advantage of recent advances in computer graphics and implicit volumetric scene representation to tackle the problem of multi-sensor spatial and temporal calibration. Thanks to a new formulation of the Neural Radiance Field (NeRF) optimization, we are able to jointly optimize calibration parameters along with scene representation based on radiometric and geometric measurements. Our method enables accurate and robust calibration from data captured in uncontrolled and unstructured urban environments, making our solution more scalable than existing calibration solutions. We demonstrate the accuracy and robustness of our method in urban scenes typically encountered in autonomous driving scenarios.

This paper presents MIMIR-UW, a multipurpose underwater synthetic dataset for SLAM, depth estimation, and object segmentation to bridge the gap between theory and application in underwater environments. MIMIR-UW integrates three camera sensors, inertial measurements, and ground truth for robot pose, image depth, and object segmentation. The underwater robot is deployed within a pipe exploration scenario, carrying artificial lights that create uneven lighting, in addition to natural artefacts such as reflections from natural light and backscattering effects. Four environments totalling eleven tracks are provided, with various difficulties regarding light conditions or dynamic elements. Two metrics for dataset evaluation are proposed, allowing MIMIR-UW to be compared with other datasets. State-of-art methods on SLAM, segmentation and depth estimation are deployed and benchmarked on MIMIR-UW. Moreover, the dataset's potential for sim-to-real transfer is demonstrated by leveraging the segmentation and depth estimation models trained on MIMIR-UW in a real pipeline inspection scenario. To the best of the authors' knowledge, this is the first underwater dataset targeted for such a variety of methods. The dataset is publicly available online. https://github.com/remaro-network/MIMIR-UW/

The spectrum of applications of robotic systems is constantly being expanded in the research, industrial and even the defense sectors, ranging from manipulation and assembly to critical infrastructure monitoring and post-disaster response. Nevertheless, contemporary robotic capabilities are significantly hindered when traversing through or interacting with complex, unstructured and dynamic environments. To alleviate for that, we introduce in the current work MIGHTY (Multi-functional Intelligent Gripping with High Tolerance), a novel, lightweight, sensor-enhanced vacuum suction cup providing not only enhanced attachment capabilities on a plethora of surfaces of varying roughness, but also robust and accurate contact and force estimation. Its unique design facilitates multiple functionalities, from acting as a gripper for object manipulation to operating as a foot for stable walking and steep surface climbing. The proposed suction cup was extensively assessed under varying experimental setups, in order to validate its capacity to sense the applied force and torque and the torque's axis, as well as its ability to attach on a variety of surfaces. In all cases remarkable results were demonstrated, attesting for its effectiveness and robustness.

Text detection is frequently used in vision-based mobile robots when they need to interpret texts in their surroundings to perform a given task. For instance, delivery robots in multilingual cities need to be capable of doing multilingual text detection so that the robots can read traffic signs and road markings. Moreover, the target languages change from region to region, implying the need of efficiently re-training the models to recognize the novel/new languages. However, collecting and labeling training data for novel languages are cumbersome, and the efforts to re-train an existing/trained text detector are considerable. Even worse, such a routine would repeat whenever a novel language appears. This motivates us to propose a new problem setting for tackling the aforementioned challenges in a more efficient way: “We ask for a generalizable multilingual text detection framework to detect and identify both seen and unseen language regions inside scene images without the requirement of collecting supervised training data for unseen languages as well as model re-training”. To this end, we propose “MENTOR”, the first work to realize a learning strategy between zero-shot learning and few-shot learning for multilingual scene text detection. During the training phase, we leverage the “zero-cost” synthesized printed texts and the available training/seen languages to learn the meta-mapping from printed texts to language-specific kernel weights. Meanwhile, dynamic convolution networks guided by the language-specific kernel are trained to realize a detection-by-feature-matching scheme. In the inference phase, “zero-cost” printed texts are synthesized given a new target language. By utilizing the learned meta-mapping and the matching network, our “MENTOR” can freely identify the text regions of the new language. Experiments show our model can achieve comparable results with supervised methods for seen languages and outperform other methods in detecting unseen languages. Show More

Elevation maps are commonly used to represent the environment of mobile robots and are instrumental for locomotion and navigation tasks. However, pure geometric information is insufficient for many field applications that require appearance or semantic information, which limits their applicability to other platforms or domains. In this work, we extend a 2.5D robot-centric elevation mapping framework by fusing multi-modal information from multiple sources into a popular map representation. The framework allows inputting data contained in point clouds or images in a unified manner. To manage the different nature of the data, we also present a set of fusion algorithms that can be selected based on the information type and user requirements. Our system is designed to run on the GPU, making it real-time capable for various robotic and learning tasks. We demonstrate the capabilities of our framework by deploying it on multiple robots with varying sensor configurations and showcasing a range of applications that utilize multi-modal layers, including line detection, human detection, and colorization.

Linear policies are the simplest class of policies that can achieve stable bipedal walking behaviors in both simulation and hardware. However, a significant challenge in deploying them widely is the difficulty in extending them to more dynamic behaviors like hopping and running. Therefore, in this work, we propose a new class of linear policies in which template models can be embedded. In particular, we show how to embed Spring Loaded Inverted Pendulum (SLIP) model in the policy class and realize perpetual hopping in arbitrary directions. The spring constant of the template model is learned in addition to the remaining parameters of the policy. Given this spring constant, the goal is to realize hopping trajectories using the SLIP model, which are then tracked by the bipedal robot using the linear policy. Continuous hopping with adjustable heading direction was achieved across different terrains in simulation with heading and lateral velocities of up to O.5m/ sec and 0.05m/ sec, respectively. The policy was then transferred to the hardware, and preliminary results (> 10 steps) of hopping were achieved.