ICRA 2023 Papers

This paper presents an Augmented Reality (AR)assisted robot learning framework for Minimally Invasive Surgery (MIS) tasks. The proposed framework exploits an external optical tracking system to collect human demonstration. Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR) are utilized to encode and generate a robust desired trajectory for transferring to the real robot for the MIS task. The HoloLens 2 Head-Mounted-Display (HMD) is integrated for intuitive visualization of the robot configuration under the constraint of a small incision on the patient's abdominal cavity during the demonstration phase. Experiments are conducted to verify the feasibility and performance of the proposed framework and compared it with the kinesthetic teaching-based modality in a tumor resection MIS task. The results illustrate that the proposed AR-assisted robot learning framework requires lower workload demand, achieves higher performance and efficiency, and ensures the feasibility of the learned results for reproduction on a real robot for MIS tasks.

Field-control-based nanorobotic manipulation of ions at the single atomic level is an enabling technique for such applications as in-situ prototyping and characterization for fundamental research and rapid product development of nanoscale and quantum devices such as sensors, batteries, neuromorphic devices, and neuro/brain interfaces. Taking the motion of quantum dots (QDs) manipulated by an electrostatic field steered by a probe tip on a target surface as an example, here we show a deep-learning-based approach for their global motion tracking via the individual atoms both on the surface and inside the body. Transmission electron graphs, element analysis, and crystal topology acquired from an aberration-corrected transmission electron microscope (Cs-TEM) are used to identify the positions, types, and structures of the atoms to understand their kinematics. The results show the feasibility of multi-target tracking of homogeneous atoms by their spatial structure projection, which is very encouraging for further extension to the tracking and regulation of crystalline grains, swarms of ions, ion filaments, and single ions.

We propose LiDAR-Inertial Odometry via Simultaneous EGo-motion estimation and Multiple Object Tracking (LIO-SEGMOT), an optimization-based odometry approach targeted for dynamic environments. LIO-SEGMOT is formulated as a state estimation approach with asynchronous state update of the odometry and the object tracking. That is, LIO-SEGMOT can provide continuous object tracking results while preserving the keyframe selection mechanism in the odometry system. Meanwhile, a hierarchical criterion is designed to properly couple odometry and object tracking, preventing system instability due to poor detections. We compare LIO-SEGMOT against the baseline model LIO-SAM, a state-of-the-art LIO approach, under dynamic environments of the KITTI raw dataset and the self-collected Hsinchu dataset. The former experiment shows that LIO-SEGMOT obtains an average improvement 1.61% and 5.41% of odometry accuracy in terms of absolute translational and rotational trajectory errors. The latter experiment also indicates that LIO-SEGMOT obtains an average improvement 6.97% and 4.21% of odometry accuracy.

Recent progress in end-to-end Imitation Learning approaches has shown promising results and generalization capabilities on mobile manipulation tasks. Such models are seeing increasing deployment in real-world settings, where scaling up requires robots to be able to operate with high autonomy, i.e. requiring as little human supervision as possible. In order to avoid the need for one-on-one human supervision, robots need to be able to detect and prevent policy failures ahead of time, and ask for help, allowing a remote operator to supervise multiple robots and help when needed. However, the black-box nature of end-to-end Imitation Learning models such as Behavioral Cloning, as well as the lack of an explicit state-value representation, make it difficult to predict failures. To this end, we introduce Behavioral Cloning Value Approximation (BCVA), an approach to learning a state value function based on and trained jointly with a Behavioral Cloning policy that can be used to predict failures. We demonstrate the effectiveness of BCVA by applying it to the challenging mobile manipulation task of latched-door opening, showing that we can identify failure scenarios with with 86% precision and 81 % recall, evaluated on over 2000 real world runs, improving upon the baseline of simple failure classification by 10 percentage-points.

In the practical application of point cloud completion tasks, real data quality is usually much worse than the CAD datasets used for training. A small amount of noisy data will usually significantly impact the overall system's accuracy. In this paper, we propose a quality evaluation network to score the point clouds and help judge the quality of the point cloud before applying the completion model. We believe our scoring method can help researchers select more appropriate point clouds for subsequent completion and reconstruction and avoid manual parameter adjustment. Moreover, our evaluation model is fast and straightforward and can be directly inserted into any model's training or use process to facilitate the automatic selection and post-processing of point clouds. We propose a complete dataset construction and model evaluation method based on ShapeNet. We verify our network using detection and flow estimation tasks on KITTI, a real-world dataset for autonomous driving. The experimental results show that our model can effectively distinguish the quality of point clouds and help in practical tasks.

We present Aquarium, a differentiable fluid-structure interaction solver for robotics that offers stable simulation, accurately coupled fluid-robot physics in two dimensions, and full differentiability with respect to fluid and robot states and parameters. Aquarium achieves stable simulation with accurate flow physics by directly integrating over the incompressible Navier-Stokes equations using a fully implicit Crank-Nicolson scheme with a second-order finite-volume spa-tial discretization. The fluid and robot physics are coupled using the immersed-boundary method by formulating the no-slip condition as an equality constraint applied directly to the Navier-Stokes system. This choice of coupling allows the fluid-structure interaction to be posed and solved as a nonlinear optimization problem. This optimization-based formulation is then exploited using the implicit-function theorem to compute derivatives. Derivatives can then be passed to downstream gradient-based optimization or learning algorithms. We demon-strate Aquarium's ability to accurately simulate coupled fluid-robot physics with numerous 2D examples, including a cylinder in free stream and a soft robotic fish tail with hardware validation. We also demonstrate Aquarium's ability to provide analytical gradients by performing gradient-based shape-and-gait optimization of an oscillating diamond foil to maximize its generated thrust.

We study the sample placement and shortest tour problem for robots tasked with mapping environmental phenomena modeled as stationary random fields. The objective is to minimize the resources used (samples or tour length) while guaranteeing estimation accuracy. We give approximation algorithms for both problems in convex environments. These improve previously known results, both in terms of theoretical guarantees and in simulations. In addition, we disprove an existing claim in the literature on a lower bound for a solution to the sample placement problem.

Finding Nash equilibrial policies for two-player differential games requires solving Hamilton-Jacobi-Isaacs (HJI) PDEs. Self-supervised learning has been used to approximate solutions of such PDEs while circumventing the curse of dimensionality. However, this method fails to learn discontinuous PDE solutions due to its sampling nature, leading to poor safety performance of the resulting controllers in robotics applications when player rewards are discontinuous. This paper investigates two potential solutions to this problem: a hybrid method that leverages both supervised Nash equilibria and the HJI PDE, and a value-hardening method where a sequence of HJIs are solved with a gradually hardening reward. We compare these solutions using the resulting generalization and safety performance in two vehicle interaction simulation studies with 5D and 9D state spaces, respectively. Results show that with informative supervision (e.g., collision and near-collision demonstrations) and the low cost of self-supervised learning, the hybrid method achieves better safety performance than the supervised, self-supervised, and value hardening approaches on equal computational budget. Value hardening fails to generalize in the higher-dimensional case without informative supervision. Lastly, we show that the neural activation function needs to be continuously differentiable for learning PDEs and its choice can be case dependent.

We consider a service robot in a household environment given a sequence of high-level tasks one at a time. Most existing task planners, lacking knowledge of what they may be asked to do next, solve each task in isolation and so may unwittingly introduce side effects that make subsequent tasks more costly. In order to reduce the overall cost of completing all tasks, we consider that the robot must anticipate the impact its actions could have on future tasks. Thus, we propose anticipatory planning: an approach in which estimates of the expected future cost, from a graph neural network, augment model-based task planning. Our approach guides the robot towards behaviors that encourage preparation and organization, reducing overall costs in long-lived planning scenarios. We evaluate our method on blockworld environments and show that our approach reduces the overall planning costs by 5% as compared to planning without anticipatory planning. Additionally, if given an opportunity to prepare the environment in advance (a special case of anticipatory planning), our planner improves overall cost by 11%.

Estimating human joint moments using wearable sensors has utility for personalized health monitoring and generalized exoskeleton control. Data-driven models have potential to map wearable sensor data to human joint moments, even with a reduced sensor suite and without subject-specific calibration. In this study, we quantified the RMSE and R2 of a temporal convolutional network (TCN), trained to estimate human hip moments in the sagittal plane using exoskeleton sensor data (i.e., a hip encoder and thigh- and pelvis-mounted inertial measurement units). We conducted three analyses in which we iteratively retrained the network while: 1) varying the input sequence length of the model, 2) incorporating noncausal data into the input sequence, thus delaying the network estimates, and 3) time shifting the labels to train the model to anticipate (i.e., predict) human hip moments. We found that 930 ms of causal input data maintained model performance while minimizing input sequence length (validation RMSE and R2 of 0.141±0.014 Nm/kg and 0.883±0.025, respectively). Further, delaying the model estimate by up to 200 ms significantly improved model performance compared to the best causal estimators (p<0.05), improving estimator fidelity in use cases where delayed estimates are acceptable (e.g., in personalized health monitoring or diagnoses). Finally, we found that anticipating hip moments further in time linearly increased model RMSE and decreased R2 (p<0.05); however, performance remained strong (R2>0.85) when predicting up to 200 ms ahead.

In our daily life, when a small amount of sweat or water forms on a person's hand, we can empirically feel that the friction force of the hand increases, and the objects are gripped well. However, if sweat or water forms heavily, we can feel the friction decrease when holding an object. In this study, we analyzed the degree to which fingerprints and sweat present on a person's hand can affect the friction force between the hand and the gripping object. We fabricated an anthropomorphic robot hand with a fingerprint structure to set up an environment similar to that of the human hand, and performed object-holding and friction-change experiments by changing the amount of sweat to verify that this phenomenon can be applied to a robot hand. Furthermore, we for the first time proposed and developed a variable friction system using fluids and microstructures to solve the difficulty of anthropomorphic robot hand force control. By applying the manufactured variable friction system and performing an active friction control performance test and an object grip test of the robot hand, we validated that the fingerprint and sweat of a human hand can affect the grip of an actual object.

Human drivers are remarkably robust against various unexpected occurring variations and corruptions by understanding temporal changes and traffic scenes. In contrast, the neural network based autonomous navigation system can be easily affected by sensor data anomaly, like occlusion, sensor noise, challenging weather and illumination conditions. Such external disturbances are inevitable in practical driving applications. In this paper, we develop a semi-supervised anomaly detection module to detect the corrupted data while extracting the traffic scenario features. We further introduce an end-to-end robust autonomous navigation framework based on the idea that the consecutive frames of clean data depict a similar traffic scenario and the differences among the sequential data imply the dynamic state changes. By taking into consideration both spatial traffic scenario and temporal environmental variation, the model is able to achieve robust navigation against sensor data corruptions. We conduct experiments in CARLA platform and the evaluation results show the effectiveness of the proposed method.

As autonomous driving systems prevail, it is becoming increasingly critical that the systems learn from databases containing fine-grained driving scenarios. Most databases currently available are human-annotated; they are expensive, time-consuming, and subject to behavioral biases. In this paper, we provide initial evidence supporting a novel technique utilizing drivers' electroencephalography (EEG) signals to implicitly label hazardous driving scenarios while passively viewing recordings of real-road driving, thus sparing the need for manual annotation and avoiding human annotators' behavioral biases during explicit report. We conducted an EEG experiment using real-life and animated recordings of driving scenarios and asked participants to report danger explicitly whenever necessary. Behavioral results showed the participants tended to report danger only when overt hazards (e.g., a vehicle or a pedestrian appearing unexpectedly from behind an occlusion) were in view. By contrast, their EEG signals were enhanced at the sight of both an overt hazard and a covert hazard (e.g., an occlusion signalling possible appearance of a vehicle or a pedestrian from behind). Thus, EEG signals were more sensitive to driving hazards than explicit reports. Further, the Time-Series AI (TSAI, [1]) successfully classified EEG signals corresponding to overt and covert hazards. We discuss future steps necessary to materialize the technique in real life.

This paper introduces a new controller that stabilizes the motion of a spatial quadruped around sagittal-plane templates. It enables highly dynamic gaits and transitional maneuvers formed from parallel and sequential compositions of such planar templates in settings that require significant out-of-plane reactivity. The controller admits formal guarantees of stability with some modest assumptions. Experimental results validate the reliable execution of those planar template-based maneuvers, even in the face of large lateral, yaw, and roll incurring disturbances. This spatial anchor, fixed in parallel composition with a variety of different parallel and sequential compositions of sagittal plane templates, illustrates the robust portability of provably interoperable modular control components across a variety of hardware platforms and behaviors.

Recently, Vehicle-to-Everything (V2X) cooperative perception has attracted increasing attention. Infrastructure sensors play a critical role in this research field; however, how to find the optimal placement of infrastructure sensors is rarely studied. In this paper, we investigate the problem of infrastructure sensor placement and propose a pipeline that can efficiently and effectively find optimal installation positions for infrastructure sensors in a realistic simulated environment. To better simulate and evaluate LiDAR place-ment, we establish a Realistic LiDAR Simulation library that can simulate the unique characteristics of different popular LiDARs and produce high-fidelity LiDAR point clouds in the CARLA simulator. Through simulating point cloud data in different LiDAR placements, we can evaluate the perception accuracy of these placements using multiple detection models. Then, we analyze the correlation between the point cloud distribution and perception accuracy by calculating the density and uniformity of regions of interest. Experiments show that when using the same number and type of LiDAR, the placement scheme optimized by our proposed method improves the average precision by 15%, compared with the conventional placement scheme in the standard lane scene. We also analyze the correlation between perception performance in the region of interest and LiDAR point cloud distribution and validate that density and uniformity can be indicators of performance. Both the RLS Library and related code will be released at https://github.com/PJLab-ADG/LiDARSimLib-and-Placement-Evaluation.

This paper proposes a novel mathematical solution to solve the inverse kinematics (IK) of single section mobile continuum manipulators (SSMCMs). Thus, to achieve a given pose of the end-effector (EE), the proposed mathematical solution consists in determining the position and orientation parameters of the mobile platform and of a single section of the continuum manipulator. As advantages, the proposed mathematical solution eliminates the EE pose errors when the dynamic parameters are neglected and the continuum manipulator is cylindrical in shape. A simulation and an experiment validate the proposed approach.

Safety is paramount for robots used in environments they share with humans. In such scenarios, security is also growing in importance. However, conventional approaches to analysing safety requirements are aimed at identifying hazards only. Security-related aspects such as cyber threats, cyber attacks and vulnerabilities have hardly been integrated into analysis and design methods to date. The methods available so far for the joint analysis of safety and security are based on established methods of safety engineering, where the amount of information is very large and usually stored in text- and table-based documents. This makes it challenging for engineers to systematically assess and maintain safety and security information. Thus, adequate tool support for robot engineers is required to cope with the increased complexity and to manage the safety and security risks. In this paper, we demonstrate that robot's safety and security information can be expressed, stored, analysed and queried in a knowledge graph representation paving the way to automated analysis. More specifically, we apply an integrated, systems-oriented safety and security co-analysis approach, namely STPA-Safesec, to a robot performing disinfection tasks in domestic environments. By querying the resulting graph of safety and security artefacts, we automatically retrieve hazardous scenarios, identify gaps in the analysis and increase our understanding of the overall risks of the robot.

Obtaining the shape and size of a robot's workspace is essential for both its design and control. However, determining the accurate workspace of a multi-segment continuum robot by graphic or analytical methods is a challenging task due to its inherent flexibility and complex structure. Existing numerical methods have limitations when applied to a continuum robot. This paper presents an Equivalent Two Section (ETS) method for calculating the workspace of multi-segment continuum robots. This method is based on the forward kinematics and a piecewise constant curvature (PCC) model to determine the boundaries of the workspace. In order to verify the proposed method, simulation experiments are conducted using six different maximum bending angles and seven different number of segments. Results of the ETS method are compared to the true workspaces of these configurations estimated by an exhaustive approach. The results show that the proposed ETS method is both efficient and accurate, and has small estimation errors. Discussions on the advantages and limitations of the proposed ETS method are also presented.

Compared with rigid robots, soft robots have the advantages of inherent compliance, high adaptability, and impact tolerance. Many researchers are very interested in the motion design of soft robot underwater. In this paper, inspired by the method of octopus propulsion, a jet propulsion unit with 80% soft materials driven by pressure is designed. It can change the volume of its cavity to absorb and eject the fluid medium to make the robot move. According to the working characteristics of the jet unit, corresponding experiments are designed to analyze its force output, deformation, ejection flow, and pressure response characteristics. In order to expand the motion space of the robot, a buoyancy unit is designed to control the depth of the robot in the water. Three jet units and a buoyancy element are combined into a tetrahedron robot - jet soft robot (JSR). The feasibility of its motion is verified by experiments. Compared with other similar jet robots, the biggest feature of this robot is that the drive unit can bend or twist roughly along the centerline, which can prevent accidental collision and damage.