ICRA 2023 Papers

This paper proposes a novel real-time semantic segmentation network via frequency domain learning, called FDLNet, which revisits the segmentation task from two critical perspectives: spatial structure description and multilevel feature fusion. We first devise an image-size convolution (IS-Conv) as a global frequency-domain learning operator to capture long-range dependency in a single shot. To model spatial structure information, we construct the global structure representation path (GSRP) based on IS-Conv, which learns a unified edge-region representation with affordable complexity. For efficient and lightweight multi-level feature fusion, we propose the factorized stereoscopic attention (FSA) module, which alleviates semantic confusion and reduces feature redundancy by introducing level-wise attention before channel and spatial attention. Combining the above modules, we propose a concise semantic segmentation framework named FDLNet. We experimentally demonstrate the effectiveness and superiority of the proposed method. FDLNet achieves state-of-the-art performance on the Cityscapes, which reports 76.32% mIoU at 150+ FPS and 79.0% mIoU at 41+ FPS. The code is available at https://github.com/qyan0131/FDLNet.

In the context of robotics, accurate ground-truth positioning is the cornerstone for the development of mapping and localization algorithms. In outdoor environments and over long distances, total stations provide accurate and precise measurements, that are unaffected by the usual factors that deteriorate the accuracy of Global Navigation Satellite System (GNSS). While a single robotic total station can track the position of a target in three Degrees Of Freedom (DOF), three robotic total stations and three targets are necessary to yield the full six DOF pose reference. Since it is crucial to express the position of targets in a common coordinate frame, we present a novel extrinsic calibration method of multiple robotic total stations with field deployment in mind. The proposed method does not require the manual collection of ground control points during the system setup, nor does it require tedious synchronous measurement on each robotic total station. Based on extensive experimental work, we compare our approach to the classical extrinsic calibration methods used in geomatics for surveying and demonstrate that our approach brings substantial time savings during the deployment. Tested on more than 30 km of trajectories, our new method increases the precision of the extrinsic calibration by 25 % compared to the best state-of-the-art method, which is the one taking manually static ground control points.

This paper proposed an adaptive robust sliding mode control (SMC) with a nonlinear sliding perturbation observer (SPO) for robot manipulators. SPO estimates the perturbation (nonlinearities, uncertainties, and disturbances) with minimal system information and enhances the controller performance. The estimation is mainly dependent on the selection of SMCSPO gain, and if not tuned well, it might result in increased error dynamics of the system. Therefore, minimizing the error dynamics by improving the estimation is the primary goal of this research. In this regard, the current study accomplishes adaptation of controller gain in real-time by using an optimization technique called extremum seeking (ES). The quality adaptation is controlled with the help of a cost function. Based on the Lyapunov-based stability analysis of SMCSPO, the cost function consisting of the estimation error of the observer and error dynamics is proposed. The unique cost function now guarantees the tracking performance within the defined error tolerance. The effectiveness of the proposed algorithm is illustrated and validated in simulation and experiments. It is shown that the adaptation based on ES with the proposed cost function converges to the optimal control gain enabling the reduced estimation error and error dynamics with enhanced tracking performance.

Visual imitation learning provides an effective framework to learn skills from demonstrations. However, the quality of the provided demonstrations usually significantly affects the ability of an agent to acquire desired skills. Therefore, the standard visual imitation learning assumes near-optimal demonstrations, which are expensive or sometimes prohibitive to collect. Previous works propose to learn from noisy demonstrations; however, the noise is usually assumed to follow a context-independent distribution such as a uniform or gaussian distribution. In this paper, we consider another crucial yet underexplored setting - imitation learning with task-irrelevant yet locally consistent segments in the demonstrations (e.g., wiping sweat while cutting potatoes in a cooking tutorial). We argue that such noise is common in real world data and term them as “extraneous” segments. To tackle this problem, we introduce Extraneousness-Aware Imitation Learning (EIL), a self-supervised approach that learns visuomotor policies from third-person demonstrations with extraneous subsequences. EIL learns action-conditioned observation embeddings in a self-supervised manner and retrieves task-relevant observations across visual demonstrations while excluding the extraneous ones. Experimental results show that EIL outperforms strong baselines and achieves comparable policies to those trained with perfect demonstration on both simulated and real-world robot control tasks. The project page can be found here: https://sites.google.com/view/eil-website.

Robotic task planning is computationally challenging. To reduce planning cost and support life-long operation, we must leverage prior planning experience. To this end, we address the problem of extracting reusable and generalizable abstract skills from successful plan executions. In previous work, we introduced a supporting framework, allowing us, theoretically, to extract an abstract skill from a single execution and later automatically adapt it and reuse it in new domains. We also proved that, given a library of such skills, we can significantly reduce the planning effort for new problems. Nevertheless, until now, abstract-skill extraction could only be performed manually. In this paper, we finally close the automation loop and explain how abstract skills can be practically and automatically extracted. We start by analyzing the desired qualities of an abstract skill and formulate skill extraction as an optimization problem. We then develop two extraction algorithms, based on the novel concept of abstraction-critical state detection. As we show experimentally, the approach is independent of any planning domain.

Recently, legged robots have been used for various purposes, such as exploring unknown terrain or interacting with the world. For control and planning legged systems during interactive operations, it is essential to estimate and respond to external forces. However, in legged system, it becomes difficult to estimate forces due to highly dynamic situations. There are several studies that use a force sensor on the foot and end effector, but these approaches have disadvantages in terms of cost and sustainability. Therefore, in this paper, we propose an improved method for estimating external forces without a force sensor. First, each leg force was obtained using the system dynamics of the robot with a disturbance observer. Then, by preintegration, it was tightly coupled with other sensors to estimate the pose and external force simultaneously. Despite the impact and slip, we estimate external forces accurately in standing and walking motions. Moreover, we compared pose estimation performance with VINS-Mono [1], and there is no significant accuracy degradation in spite of highly dynamic force residual.

We present an approach for estimating a mobile robot's pose w.r.t. the allocentric coordinates of a network of static cameras using multi-view RGB images. The images are processed online, locally on smart edge sensors by deep neural networks to detect the robot and estimate 2D keypoints defined at distinctive positions of the 3D robot model. Robot keypoint detections are synchronized and fused on a central backend, where the robot's pose is estimated via multi-view minimization of reprojection errors. Through the pose estimation from external cameras, the robot's localization can be initialized in an allocentric map from a completely unknown state (kidnapped robot problem) and robustly tracked over time. We conduct a series of experiments evaluating the accuracy and robustness of the camera-based pose estimation compared to the robot's internal navigation stack, showing that our camera-based method achieves pose errors below 3 cm and 1° and does not drift over time, as the robot is localized allocentrically. With the robot's pose precisely estimated, its observations can be fused into the allocentric scene model. We show a real-world application, where observations from mobile robot and static smart edge sensors are fused to collaboratively build a 3D semantic map of a ~240 m2 indoor environment.

Optical Coherence Elastography (OCE) is a method that discerns local tissue stiffness using optical information. This method has recently been explored for laryngeal cancer tumor margin detection but has not been widely deployed clinically. Part of the challenge hindering such clinical deployment is the need for controlled high-precision mechanical probing of the tissue. This paper explores the concept of robot-assisted optical coherence elastography(OCE) and presents a preliminary system integration used to demonstrate the approach for stiffness mapping and discerning tumor margins. The approach is demonstrated on a custom Cartesian stage robot, and a custom-built OCE system comprised of an 830 nm broad-band laser with a vector-analysis method for phase gradient estimation and strain imaging. The paper illustrates one of the advantages of robot-controlled probing in terms of increasing the accuracy of the OCE system in a large range of displacement and strain. By leveraging motion information from the robot, online re-calibration of the OCE strain map may be achieved, thereby reducing OCE errors. After calibration, it is shown that the error in estimating the local Young's modulus is 0.485% in the silicon phantom and 0.531% in the agar phantom. These results suggest that future integration of optical coherence tomography(OCT) in clinically deployable robots may offer advantages in enabling local stiffness map estimation using OCE.

The prediction of surrounding agents' motion is a key for safe autonomous driving. In this paper, we explore navigation maps as an alternative to the predominant High Definition (HD) maps for learning-based motion prediction. Navigation maps provide topological and geometrical information on road-level, HD maps additionally have centimeter-accurate lane-level information. As a result, HD maps are costly and time-consuming to obtain, while navigation maps with near-global coverage are freely available. We describe an approach to integrate navigation maps into learning-based motion prediction models. To exploit locally available HD maps during training, we additionally propose a model-agnostic method for knowledge distillation. In experiments on the publicly available Argoverse dataset with navigation maps obtained from OpenStreetMap, our approach shows a significant improvement over not using a map at all. Combined with our method for knowledge distillation, we achieve results that are close to the original HD map-reliant models. Our publicly available navigation map API for Argoverse enables researchers to develop and evaluate their own approaches using navigation maps4.

Lower-limb exoskeletons and smart walkers are robotic devices to assist patients in regaining their autonomy after a stroke. The integration of these devices enables gait rehabilitation and functional compensation, promoting natural over-ground walking. This article presents the Adaptable Robotic Platform for Gait Rehabilitation and Assistance (AGoRA V2 platform), which integrates the new AGoRA V2 Smart Walker and the AGoRA V2 unilateral lower-limb exoskeleton. It was evaluated with 14 healthy subjects using physiological and kinematic variables and a perception assessment. The study entailed four conditions: Without exoskeleton (WOE), With Exoskeleton (WE&T), With Walker (WW), and With Platform (WP). Results indicate a reduction in the muscle activity of the Rectus Femoris (18%) and Vastus Lateralis (15%), comparing WE&T and WP, as well as walking without any device (WOE) and using any robotic device (WE&T, WW, WP). Results suggest the importance of combining the exoskeleton with the robotic walker and the assistance of each device independently. Moreover, using the complete platform induces slower gait patterns than the walker, as the mean impulse force and linear velocity decrease by 42% and 44%, respectively. These results demonstrate that the platform contributes to safety, and improvements in gait parameters and muscular activity, indicating the system's potential to act as a modular device according to users' conditions and therapeutic goals.

Injections into specific retinal layers of the eye present a serious challenge to surgeons in terms of accuracy and perception. The emergence of new gene therapies further emphasizes the need for effective tools for localized drug delivery. Unlike the dominant approach of delivering drugs via a transvitreal intraocular pathway, this paper demonstrates the feasibility of delivering injections into the space between the choroid and the retina using an external approach. The design of a cooperative robotic system for enabling robot-assisted extraocular subretinal injections is presented. The system uses a distal micromanipulator that can serve as a hand-held tool for OCT-aided injection or attach to a six degree of freedom (DOF) serial robot arm for cooperative manipulation. The kinematics and control of the robot for constrained cooperative control motions to enable safe needle injection is presented and experimentally evaluated. These results suggest that the proposed external drug delivery approach is feasible, thereby enabling the advantages of preserving the integrity of the retina and omitting the necessity for vitrectomy.

We develop a resilient binary hypothesis testing frame-work for decision making in adversarial multi-robot crowdsensing tasks. This framework exploits stochastic trust observations between robots to arrive at tractable, resilient decision making at a centralized Fusion Center (FC) even when i) there exist malicious robots in the network and their number may be larger than the number of legitimate robots, and ii) the FC uses one-shot noisy measurements from all robots. We derive two algorithms to achieve this. The first is the Two Stage Approach (2SA) that estimates the legitimacy of robots based on received trust observations, and provably minimizes the probability of detection error in the worst-case malicious attack. Here, the proportion of malicious robots is known but arbitrary. For the case of an unknown proportion of malicious robots, we develop the Adversarial Generalized Likelihood Ratio Test (A-GLRT) that uses both the reported robot measurements and trust observations to estimate the trustworthiness of robots, their reporting strategy, and the correct hypothesis simultaneously. We exploit special problem structure to show that this approach remains computationally tractable despite several unknown problem parameters. We deploy both algorithms in a hardware experiment where a group of robots conducts crowdsensing of traffic conditions on a mock-up road network similar in spirit to Google Maps, subject to a Sybil attack. We extract the trust observations for each robot from actual communication signals which provide statistical information on the uniqueness of the sender. We show that even when the malicious robots are in the majority, the FC can reduce the probability of detection error to 30.5% and 29% for the 2SA and the A-GLRT respectively.

Supernumerary robotic limbs (SRLs) gained increasing interest in the last years for their applicability as healthcare and assistive technologies. These devices can either support or augment human sensorimotor capabilities, allowing users to complete tasks that are more complex than those feasible for their natural limbs. However, for a successful coordination between natural and artificial limbs, intuitiveness of interaction and perception of autonomy are key enabling features, especially for people suffering from motor disorders and impairments. The development of suitable human-robot interfaces is thus fundamental to foster the adoption of SRLs. With this work, we describe how to control an extra degree of freedom by taking advantage of what we defined the Intrinsic Kinematic Null Space, i.e. the redundancy of the human kinematic chain involved in the ongoing task. Obtained results demonstrated that the proposed control strategy is effective for performing complex tasks with a supernumerary robotic finger, and that practice improves users' control ability.

This work explores scene graphs as a distilled representation of high-level information for autonomous driving, applied to future driver-action prediction. Given the scarcity and strong imbalance of data samples, we propose a self-supervision pipeline to infer representative and well-separated embeddings. Key aspects are interpretability and explainability; as such, we embed in our architecture attention mechanisms that can create spatial and temporal heatmaps on the scene graphs. We evaluate our system on the ROAD dataset against a fully-supervised approach, showing the superiority of our training regime.

Action advising is a knowledge transfer technique for reinforcement learning based on the teacher-student paradigm. An expert teacher provides advice to a student during training in order to improve the student's sample efficiency and policy performance. Such advice is commonly given in the form of state-action pairs. However, it makes it difficult for the student to reason with and apply to novel states. We introduce Explainable Action Advising, in which the teacher provides action advice as well as associated explanations indicating why the action was chosen. This allows the student to self-reflect on what it has learned, enabling advice generalization and leading to improved sample efficiency and learning performance - even in environments where the teacher is sub-optimal. We empirically show that our framework is effective in both single-agent and multi-agent scenarios, yielding improved policy returns and convergence rates when compared to state-of-the-art methods.

Local-HDP (Local Hierarchical Dirichlet Process) is a hierarchical Bayesian method recently used for open-ended 3D object category recognition. It has been proven to be efficient in real-time robotic applications. However, the method is not robust to a high degree of occlusion. We address this limitation in two steps. First, we propose a novel semantic 3D object-parts segmentation method that has the flexibility of Local-HDP. This method is shown to be suitable for open-ended scenarios where the number of 3D objects or object parts are not fixed and can grow over time. We show that the proposed method has a higher percentage of mean intersection over union, using a smaller number of learning instances. Second, we integrate this technique with a recently introduced argumentation-based online incremental learning method, enabling the model to handle a high degree of occlusion. We show that the resulting model produces explicit explanations for the 3D object category recognition task.

Ultrasound imaging is a commonly used modality for several diagnostic and therapeutic procedures. However, the diagnosis by ultrasound relies heavily on the quality of images assessed manually by sonographers, which diminishes the objectivity of the diagnosis and makes it operator-dependent. The supervised learning-based methods for automated quality assessment require manually annotated datasets, which are highly labour-intensive to acquire. These ultrasound images are low in quality and suffer from noisy annotations caused by inter-observer perceptual variations, which hampers learning efficiency. We propose an UnSupervised UltraSound image Quality assessment Network, US2QNet, that eliminates the burden and uncertainty of manual annotations. US2QNet uses the variational autoencoder embedded with the three modules, pre-processing, clustering and post-processing, to jointly enhance, extract, cluster and visualize the quality feature representation of ultrasound images. The pre-processing module uses filtering of images to point the network's attention towards salient quality features, rather than getting distracted by noise. Post-processing is proposed for visualizing the clusters of feature representations in 2D space. We validated the proposed framework for quality assessment of the urinary bladder ultrasound images. The proposed framework achieved 78% accuracy and superior performance to state-of-the-art clustering methods. The project page with source codes is available at https://sites.google.com/view/US2QNet.

We present the results of experiments performed using a small autonomous underwater vehicle to determine the location of an isobath within a bounded area. The primary contribution of this work is to implement and integrate several recent developments real-time planning for environmental map-ping, and to demonstrate their utility in a challenging practical example. We model the bathymetry within the operational area using a Gaussian process and propose a reward function that represents the task of mapping a desired isobath. As is common in applications where plans must be continually updated based on real-time sensor measurements, we adopt a receding horizon framework where the vehicle continually computes near-optimal paths. The sequence of paths does not, in general, inherit the optimality properties of each individual path. Our real-time planning implementation incorporates recent results that lead to performance guarantees for receding-horizon planning.

The control system of industrial robots is often model-based, and the quality of the model of high importance. Therefore, a fast and easy-to-use process for finding the model parameters from a combination of prior knowledge and measurement data is required. It has been shown that the experiment design can be improved in terms of short experiment times and an accurate parameter estimate if the robot configurations for the identification experiments are selected carefully. Estimates of the information matrix can be generated based on simulations for a number of candidate configurations, and an optimization problem can be solved for finding the optimal configurations. This work shows that the proposed method for improved experiment design works with a real manipulator, i.e. it is demonstrated that the experiment time is reduced significantly and the accuracy of the parameter estimate can be maintained or reduced if experiments are conducted only in the optimal manipulator configurations. It is also shown that the model improvement is relevant for realizing accurate control. Finally, the experimental data reveals that, in order to further improve the model accuracy, a more advanced model structure is needed for taking into account the commonly present nonlinear transmission stiffness of the robotic joints.