IROS 2021 Papers

Path planning of a tool in Minimally Invasive Surgery (MIS) can provide assistance to the surgeons by giving solutions for faster and safe tool movements during the surgery. However, the main challenge in this problem is to address non-uniform tool shape for planning that can change due to the tool’s dexterity. A typical robotic path planning approach by describing the robot’s feasible movements using C-space is applied in this work. Unlike the robotic path planning problem, the C-space description capturing the movement does not give any closed-form solution due to high degree of freedom associated with the tool moved by human hands. Hence, an interval-based approach is used for describing the C-space. The proposed interval-based approach is capable of dividing the space into feasible and non-feasible intervals of different sizes which helps to reduce the search area and cover the obstacles in a refined manner. This paper presents collision-free and fast path computation using interval arithmetic between any two points in a 2D- surgical environment cluttered with obstacles for a surgical tool robot.

Deep Reinforcement Learning (DRL) is a viable solution for automating repetitive surgical subtasks due to its ability to learn complex behaviours in a dynamic environment. This task automation could lead to reduced surgeon’s cognitive workload, increased precision in critical aspects of the surgery, and fewer patient-related complications. However, current DRL methods do not guarantee any safety criteria as they maximise cumulative rewards without considering the risks associated with the actions performed. Due to this limitation, the application of DRL in the safety-critical paradigm of robot-assisted Minimally Invasive Surgery (MIS) has been constrained. In this work, we introduce a Safe-DRL framework that incorporates safety constraints for the automation of surgical subtasks via DRL training. We validate our approach in a virtual scene that replicates a tissue retraction task commonly occurring in multiple phases of an MIS. Furthermore, to evaluate the safe behaviour of the robotic arms, we formulate a formal verification tool for DRL methods that provides the probability of unsafe configurations. Our results indicate that a formal analysis guarantees safety with high confidence such that the robotic instruments operate within the safe workspace and avoid hazardous interaction with other anatomical structures.

Sampling-based methods such as Rapidly-exploring Random Trees (RRTs) have been widely used for generating motion paths for autonomous mobile systems. In this work, we extend time-based RRTs with Control Barrier Functions (CBFs) to generate, safe motion plans in dynamic environments with many pedestrians. Our framework is based upon a human motion prediction model which is well suited for indoor narrow environments. We demonstrate our approach on a high-fidelity model of the Toyota Human Support Robot navigating in narrow corridors. We show in simulation results that our proposed online method can navigate safely in the presence of moving agents with unknown dynamics.

This paper proposes an online control framework for mobile robots to satisfy a complex mission given in the form of linear temporal logic (LTL) without colliding with moving obstacles in the environment. The proposed framework consists of three modules named the static planner, the local collision avoidance, and the patcher. The static planner is synthesized by solving a parity game for a finite abstraction of the robot model based on a world map with static obstacles to fulfill the LTL task. The local collision avoidance module computes a set of safe controls that guarantees a safe distance between the moving objects. Both of the modules can be rigorously computed offline only once via formal methods. The patcher is activated whenever a moving obstacle is detected and modifies the static plan online for a short horizon by using only provably safe controls. The resulting modified strategy can guarantee collision-free motion without losing the ability to satisfy the LTL task. As opposed to using assume-guarantee type of LTL tasks, the proposed framework can handle the situations where obstacle movement is unpredictable.

The applicability of reinforcement learning (RL) algorithms in real-world domains often requires adherence to safety constraints, a need difficult to address given the asymptotic nature of the classic RL optimization objective. In contrast to the traditional RL objective, safe exploration considers the maximization of expected returns under safety constraints expressed in expected cost returns. We introduce a model-based safe exploration algorithm for constrained high-dimensional control to address the often prohibitively high sample complexity of model-free safe exploration algorithms. Further, we provide theoretical and empirical analyses regarding the implications of model-usage on constrained policy optimization problems and introduce a practical algorithm that accelerates policy search with model-generated data. The need for accurate estimates of a policy’s constraint satisfaction is in conflict with accumulating model-errors. We address this issue by quantifying model-uncertainty as the expected Kullback-Leibler divergence between predictions of an ensemble of probabilistic dynamics models and constrain this error-measure, resulting in an adaptive resampling scheme and dynamically limited rollout horizons. We evaluate this approach on several simulated constrained robot locomotion tasks with high-dimensional action- and state-spaces. Our empirical studies find that our algorithm reaches model-free performances with a 10-20 fold reduction of training samples while maintaining approximate constraint satisfaction levels of model-free methods.

Autonomous vehicles must be able to understand the surrounding traffic flows and predict the future traffic conditions for planning a safe maneuver. During prediction, the action of autonomous vehicles should be considered, as it influences the interaction between vehicles sharing the same traffic scene and thus influences the future traffic flow. From this perspective, not only should the prediction be considered for planning, but also the action of autonomous vehicles generated by planning should be considered for traffic scene prediction. Therefore, prediction and planning must work interactively at every time step, considering results of each other. In this paper, we present a novel learning-based framework that simultaneously forecasts a nearby traffic scene and plans a maneuver of autonomous vehicle at every time step. Through experiments, we demonstrated that the proposed method exhibits better planning performance than baselines in complex traffic conditions involving various surrounding vehicles.

The perception of the environment plays a decisive role in the safe and secure operation of autonomous vehicles. The perception of the surrounding is way similar to human vision. The human’s brain perceives the environment by utilizing different sensory channels and develop a view-invariant representation model. In this context, different exteroceptive sensors like cameras, Lidar, are deployed on the autonomous vehicle to perceive the environment. These sensors have illustrated their benefit in the visible spectrum domain yet in the adverse weather conditions; for instance, they have limited operational capability at night, leading to fatal accidents. This work explores thermal object detection to model a view-invariant model representation by employing the self-supervised contrastive learning approach. We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning. Later, these learned feature representations are employed for thermal object detection using a multi-scale encoder-decoder transformer network. The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset. The experimental results illustrate the efficacy of the proposed method.

Place recognition gives a SLAM system the ability to correct cumulative errors. Unlike images that contain rich texture features, point clouds are almost pure geometric information which makes place recognition based on point clouds challenging. Existing works usually encode low-level features such as coordinate, normal, reflection intensity, etc., as local or global descriptors to represent scenes. Besides, they often ignore the translation between point clouds when matching descriptors. Different from most existing methods, we explore the use of high-level features, namely semantics, to improve the descriptor’s representation ability. Also, when matching descriptors, we try to correct the translation between point clouds to improve accuracy. Concretely, we propose a novel global descriptor, Semantic Scan Context, which explores semantic information to represent scenes more effectively. We also present a two-step global semantic ICP to obtain the 3D pose (x, y, yaw) used to align the point cloud to improve matching performance. Our experiments on the KITTI dataset show that our approach outperforms the state-of-the- art methods with a large margin. Our code is available at: https://github.com/lilin-hitcrt/SSC.

Many soft robots are capable of significantly changing their shape, an ability that can offer advantages in many applications. For instance, such a robot can flatten its body to fit under small gaps and expand to move over large obstacles. Further, because these shape changes are usually driven by a pressurized fluid, if they act over a large area, they have the potential to apply large forces to the world. However, when these same shape changes are used for the locomotion of an untethered robot, they tend to result in slow forward movement. Here we present a hybrid soft-rigid elongated-sphere robot that decouples shape change from locomotion. Pairing a compliant, inflatable outer skin, which changes volume by 15x to both fit under and roll over obstacles and can lift objects up to 30 kg, with a wheeled internal carriage, we obtain relatively fast locomotion. A new two-sided controllable adhesive between the internal carriage and the skin enables the carriage to climb vertically inside the skin, allowing the robot to climb external obstacles. We present the design of the robot, simple modeling of its behavior, and experimental testing. Our work advances the area of hybrid soft-rigid robotics by demonstrating how leveraging the strengths of both soft and rigid systems can have quantifiable performance benefits.

Freespace detection is a fundamental component of autonomous driving perception. Recently, deep convolutional neural networks (DCNNs) have achieved impressive performance for this task. In particular, SNE-RoadSeg, our previously proposed method based on a surface normal estimator (SNE) and a data-fusion DCNN (RoadSeg), has achieved impressive performance in freespace detection. However, SNE-RoadSeg is computationally intensive, and it is difficult to execute in real time. To address this problem, we introduce SNE-RoadSeg+, an upgraded version of SNE-RoadSeg. SNE-RoadSeg+ consists of 1) SNE+, a module for more accurate surface normal estimation, and 2) RoadSeg+, a data-fusion DCNN that can greatly minimize the trade-off between accuracy and efficiency with the use of deep supervision. Extensive experimental results have demonstrated the effectiveness of our SNE+ for surface normal estimation and the superior performance of our SNE-RoadSeg+ over all other freespace detection approaches. Specifically, our SNE-RoadSeg+ runs in real time, and meanwhile, achieves the state-of-the-art performance on the KITTI road benchmark. Our project page is at https://www.sne-roadseg.site/sne-roadseg-plus.

Soft robotics is a thriving branch of robotics which takes inspiration from nature and uses affordable flexible materials to design adaptable non-rigid robots. However, their flexible behavior makes these robots hard to model, which is essential for a precise actuation and for optimal control. For system modelling, learning-based approaches have demonstrated good results, yet they fail to consider the physical structure underlying the system as an inductive prior. In this work, we take inspiration from sensorimotor learning, and apply a Graph Neural Network to the problem of modelling a non-rigid kinematic chain (i.e. a robotic soft hand) taking advantage of two key properties: 1) the system is compositional, that is, it is composed of simple interacting parts connected by edges, 2) it is order invariant, i.e. only the structure of the system is relevant for predicting future trajectories. We denote our model as the "Sensorimotor Graph" since it learns the system connectivity from observation and uses it for dynamics prediction. We validate our model in different scenarios and show that it outperforms the non-structured baselines in dynamics prediction while being more robust to configurational variations, tracking errors or node failures.

In this paper, a learning-based whole-body loco-motion controller is proposed, which enables quadruped robots to perform running in the style of real animals. We use a low-level controller based on multi-rigid body dynamics to calculate desired torques for each joint, while the high-level neural network policy planning the expected gait and foothold. The policy is trained with reinforcement learning, so that the robot can track a variety of trajectories according to the gait patterns recorded from real-world dogs. We transfer the walking and running gait style to quadrupeds in simulation, involving pace, trot, high-speed gallop and natural transitions. The performance is evaluated by the synchronization rate of contact state between the policy result and the recorded sequence. In the experiments, the robot runs steadily at a speed of 2 m/s and showcases a notable synchronization rate of about 80%. Without prior knowledge, the policy demonstrates a realistic foothold distribution that covers the central area of the torso, which is prevalent in running animals.

Navigation planning for legged robots has distinct challenges compared to wheeled and tracked systems due to the ability to lift legs off the ground and step over obstacles. While most navigation planners assume a fixed traversability value for a single terrain patch, we overcome this limitation by proposing a reachability-based navigation planner for legged robots. We approximate the robot morphology by a set of reachability and body volumes, assuming that the reachability volumes need to always be in contact with the environment, while the body should be contact-free. We train a convolutional neural network to predict foothold scores which are used to restrict geometries which are considered suitable to step on. Using this representation, we propose a navigation planner based on probabilistic roadmaps. Through validation of only low-cost graph edges during graph expansion and an adaptive sampling scheme based on roadmap node density, we achieve real-time performance with fast update rates even in cluttered and narrow environments. We thoroughly validate the proposed navigation planner in simulation and demonstrate its performance in real-world experiments on the quadruped ANYmal.

Modern camera systems in robotics tend to pro-duce overwhelming amounts of visual information due to their high resolutions and high frame rates. This raises a fundamental question of how robots should focus attention on a region of the visual scene, and how they should process information in the periphery. This is particularly an issue for mobile robots, where the computational resources of low-power embedded computing boards tend to be much less than for workstations. In this paper, we look to biological design in the primate brain for inspiration on how to solve this problem. We develop a novel computational fusion of bottom-up and top-down visual saliency information. The bottom-up saliency is produced using standard colour, intensity, and motion image processing methods. The top-down saliency is produced using a deep convolutional neural network for object detection and recognition, with foveated images for computational efficiency. Regions of attention are obtained using a computational model of the basal ganglia, thought to be involved in optimal decision making. The model of the basal ganglia is based on the multi-hypothesis sequential probability ratio test (MSPRT). The visual saliency scheme is evaluated on omnidirectional video feed highlighting a proximity to human behaviour.

Motion planners for mobile robots in unknown environments face the challenge of simultaneously maintaining both robustness against unmodeled uncertainties and persistent feasibility of the trajectory-finding problem. That is, while dealing with uncertainties, a motion planner must update its trajectory, adapting to the newly revealed environment in real-time; failing to do so may involve unsafe circumstances. Many existing planning algorithms guarantee these by maintaining the clearance needed to perform an emergency brake, which is itself a robust and persistently feasible maneuver. However, such maneuvers are not applicable for systems in which braking is impossible or risky, such as fixed-wing aircraft. To that end, we propose a real-time robust planner that recursively guarantees persistent feasibility without any need of braking. The planner ensures robustness against bounded uncertainties and persistent feasibility by constructing a loop of sequentially composed funnels, starting from the receding horizon local trajectory’s forward reachable set. We implement the proposed algorithm for a robotic car tracking a speed-fixed reference trajectory. The experiment results show that the proposed algorithm can be run at faster than 16 Hz, while successfully keeping the system away from entering any dead end, to maintain safety and feasibility.

This paper presents a robust monocular visual teach-and-repeat (VT&R) navigation system for long-term operation in outdoor environments. The approach leverages deep-learned descriptors to deal with the high illumination variance of the real world. In particular, a tailored self-supervised descriptor, DarkPoint, is proposed for autonomous navigation in outdoor environments. We seamlessly integrate the localisation with control, in which proportional–integral control is used to eliminate the visual error with the pitfall of the unknown depth. Consequently, our approach achieves day-to-night navigation using a single-experience map and is able to repeat complex and fast manoeuvres. To verify our approach, we performed a vast array of navigation experiments in various outdoor environments, where both navigation accuracy and robustness of the proposed system are investigated. The experimental results show that our approach is superior to the baseline method with regards to accuracy and robustness.

Point set registration (PSR) is an essential problem in surgical navigation and image-guided surgery (IGS). It can help align the pre-operative volumetric images with the intra-operative surgical space. The performances of PSR are susceptible to noise and outliers, which are the cases in real-world surgical scenarios. In this paper, we provide a novel point set registration method that utilizes the features extracted from the PSs and can guarantee the convergence of the algorithm simultaneously. More specifically, we formulate the PSR with normal vectors by generalizing the bayesian coherent point drift (BCPD) into the six-dimension scenario. Our contributions can be summarized as follows. (1) The PSR problem with normal vectors is formulated by generalizing the Bayesian coherent point drift (BCPD) approach; (2) The updated parameters during the algorithm's iterations are given in closed-forms; (3) Extensive experiments have been done to verify the proposed approach and its significant improvements over the BCPD has been validated. We have validated our proposed registration approach on both the human femur model. Results demonstrate that our proposed method outperforms the state-of-the-art registration methods and the convergence is guaranteed at the same time.

We propose a novel approach for sampling-based and control-based motion planning. We combine a representation of the environment obtained via a modified version of optimal Rapidly-exploring Random Trees (RRT*), with landmark-based output-feedback controllers obtained via Control Lyapunov Functions, Control Barrier Functions, and robust Linear Programming. Our solution inherits many benefits of RRT*-like algorithms, such as the ability to implicitly handle arbitrarily complex obstacles. Additionally, it extends planning beyond the discrete nominal paths, as feedback controllers can correct deviations from such paths, and are robust to discrepancies between the planning and real environment maps. We test our algorithms first in simulations and then in experiments, evaluating the robustness of the approach to practical conditions, such as deformations of the environment, mismatches in the dynamical model of the robot, and measurements acquired with a camera with a limited field of view.

Progress in the last decade has brought about significant improvements in the accuracy and speed of SLAM systems, broadening their mapping capabilities. Despite these advancements, long-term operation remains a major challenge, primarily due to the wide spectrum of perturbations robotic systems may encounter.Increasing the robustness of SLAM algorithms is an ongoing effort, however it usually addresses a specific perturbation. Generalisation of robustness across a large variety of challenging scenarios is not well-studied nor understood. This paper presents a systematic evaluation of the robustness of open-source state-of-the-art SLAM algorithms with respect to challenging conditions such as fast motion, non-uniform illumination, and dynamic scenes. The experiments are performed with perturbations present both independently of each other, as well as in combination in long-term deployment settings in unconstrained environments (lifelong operation).The detailed results (approx. 20,000 experiments) along with comprehensive documentation of the benchmarking tool for integrating new datasets and evaluating SLAM algorithms not studied in this work are available at https://robustslam.github.io/evaluation.