IROS 2022 Papers

We propose a hierarchical planning algorithm that efficiently computes an optimal plan for finding a target object in large environments where a robot must simultaneously consider both navigation and manipulation. One key challenge that arises from large domains is the substantial increase in search space complexity that stems from considering mobile manipulation actions and the increase in number of objects. We offer a hierarchical planning solution that effectively handles such large problems by decomposing the problem into a set of low-level intra-container planning problems and a high-level key place planning problem that utilizes the low-level plans. To plan optimally, we propose a novel admissible heuristic function that, unlike previous methods, accounts for both navigation and manipulation costs. We propose two algorithms: one based on standard A* that returns the optimal solution, and the other based on Anytime Repairing A* (ARA*) which can trade-off computation time and solution quality, and prove they are optimal even when we use hierarchy. We show our method outperforms existing algorithms in simulated domains involving up to 6 times more number of objects than previously handled.

Research on Deep Neural Networks (DNNs) has focused on improving performance and accuracy for real-world deployments, leading to new models, such as Spiking Neural Networks (SNNs), and optimization techniques, e.g., quantization and pruning for compressed networks. However, the deployment of these innovative models and optimization techniques introduces possible reliability issues, which is a pillar for DNNs to be widely used in safety-critical applications, e.g., autonomous driving. Moreover, scaling technology nodes have the associated risk of multiple faults happening at the same time, a possibility not addressed in state-of-the-art resiliency analyses. Towards better reliability analysis for DNNs, we present enpheeph, a Fault Injection Framework for Spiking and Compressed DNNs. The enpheeph framework enables optimized execution on specialized hardware devices, e.g., GPUs, while providing complete customizability to investigate different fault models, emulating various reliability constraints and use-cases. Hence, the faults can be executed on SNNs as well as compressed networks with minimal-to-none modifications to the underlying code, a feat that is not achievable by other state-of-the-art tools. To evaluate our enpheeph framework, we analyze the resiliency of different DNN and SNN models, with different compression techniques. By injecting a random and increasing number of faults, we show that DNNs can show a reduction in accuracy with a fault rate as low as 7\times 10^{-7}7\times 10^{-7} faults per parameter, with an accuracy drop higher than 40%. Run-time overhead when executing enpheeph is less than 20% of the baseline execution time when executing 100 000 faults concurrently, at least 10× lower than state-of-the-art frameworks, making enpheeph future-proof for complex fault injection scenarios. We release the source code of our enpheeph framework under an open-source license at https://github.com/Alexei95/enpheeph.

Contrary to other standard cameras, event cam-eras interpret the world in an entirely different manner; as a collection of asynchronous events. Despite event camera's unique data output, many event feature detection and tracking algorithms have shown significant progress by making detours to frame-based data representations. This paper questions the need to do so and proposes a novel event data-friendly method that achieve simultaneous feature detection and tracking, called event Clustering-based Detection and Tracking (eCDT). Our method employs a novel clustering method, named as k-NN Classifier-based Spatial Clustering and Applications with Noise (KCSCAN), to cluster adjacent polarity events to retrieve event trajectories. With the aid of a Head and Tail Descriptor Matching process, event clusters that reappear in a different polarity are continually tracked, elongating the feature tracks. Thanks to our clustering approach in spatio-temporal space, our method automatically solves feature detection and feature tracking simultaneously. Also, eCDT can extract feature tracks at any frequency with an adjustable time window, which does not corrupt the high temporal resolution of the original event data. Our method achieves 30 % better feature tracking ages compared with the state-of-the-art approach while also having a low error approximately equal to it.

We consider time-optimal motion planning for dynamical systems that are translation-invariant, a property that holds for many mobile robots, such as differential-drives, cars, airplanes, and multirotors. Our key insight is that we can extend graph-search algorithms to the continuous case when used symbiotically with optimization. For the graph search, we introduce discontinuity-bounded A* (db-A*), a generalization of the A* algorithm that uses concepts and data structures from sampling-based planners. Db-A* reuses short trajectories, so-called motion primitives, as edges and allows a maximum user-specified discontinuity at the vertices. These trajectories are locally repaired with trajectory optimization, which also provides new improved motion primitives. Our novel kinodynamic motion planner, kMP-db-A*, has almost surely asymptotic optimal behavior and computes near-optimal solutions quickly. For our empirical validation, we provide the first benchmark that compares search-, sampling-, and optimization- based time-optimal motion planning on multiple dynamical systems in different settings. Compared to the baselines, kMP- db-A* consistently solves more problem instances, finds lower- cost initial solutions, and converges more quickly.

This paper presents a novel probabilistic approach to deep robot learning from demonstrations (LfD). Deep move-ment primitives (DMPs) are deterministic LfD model that maps visual information directly into a robot trajectory. This paper extends DMPs and presents a deep probabilistic model that maps the visual information into a distribution of effective robot trajectories. The architecture that leads to the highest level of trajectory accuracy is presented and compared with the existing methods. Moreover, this paper introduces a novel training method for learning domain-specific latent features. We show the superiority of the proposed probabilistic approach and novel latent space learning in the real-robot task of strawberry harvesting in the lab. The experimental results demonstrate that latent space learning can significantly improve model prediction performances. The proposed approach allows to sample trajectories from distribution and optimises the robot trajectory to meet a secondary objective, e.g. collision avoidance.

Reinforcement Learning (RL) has witnessed great strides for quadruped locomotion, with continued progress in the reliable sim-to-real transfer of policies. However, it remains a challenge to reuse a policy on another robot, which could save time for retraining. In this work, we present a framework for zero-shot policy retargeting wherein diverse motor skills can be transferred between robots of different shapes and sizes. The new framework centers on a planning-and-control pipeline that systematically integrates RL and Model Predictive Control (MPC). The planning stage employs RL to generate a dynamically plausible trajectory as well as the contact schedule, avoiding the combinatorial complexity of contact sequence optimization. This information is then used to seed the MPC to stabilize and robustify the policy roll-out via a new Hybrid Kinodynamic (HKD) model that implicitly optimizes the foothold locations. Hardware results show an ability to transfer policies from both the A1 and Laikago robots to the MIT Mini Cheetah robot without requiring any policy re-tuning.

People and robots may need to cross each other in narrow spaces when they are sharing environments. It is expected that autonomous robots will behave in these contexts safely but also show social behaviors. Thereby, developing an acceptable behavior for autonomous robots in the area mentioned above is a foreseeable problem for the Human-Robot Interaction (HRI) field. Our current work focuses on integrating legible non-verbal behaviors into the robot's social navigation to make nearby humans aware of its intended trajectory. Results from a within-subjects study involving 33 participants show that deictic gestures as navigational cues for humanoid robots result in fewer navigation conflicts than the use of a simulated gaze. Additionally, an increase in the perceived anthropomorphism is found when the robot uses the deictic gesture as a cue. These findings show the importance of social behaviors for people avoidance and suggest a paradigm of such behaviors in future humanoid robotic applications.

In biomedical engineering, robotic implants provide new methods to restore and improve bodily function, and regenerate tissue. A significant challenge with the design of these devices is to safely actuate them for weeks or months, while they are residing in a patient's body. Magnetic, and other force-at-distance actuation methods, allow mechanisms to be controlled remotely and without contact or line of sight to the device. In this paper, we present a novel magnetic field driven wireless motor. The motor drives a robotic implant for the treatment of long gap esophageal atresia and short bowel syndrome. The motor is equipped with two oppositely oriented permanent magnets which experience forces in opposite directions when a magnetic field is applied tangential to the magnets' directions. The implant can produce a force of 2 N. It is demonstrated with an ex vivo porcine esophagus.

Sensor-equipped unoccupied aerial vehicles (UAVs) have the potential to help reduce search times and alleviate safety risks for first responders carrying out Wilderness Search and Rescue (WiSAR) operations, the process of finding and rescuing person(s) lost in wilderness areas. Unfortunately, visual sensors alone do not address the need for robustness across all the possible terrains, weather, and lighting conditions that WiSAR operations can be conducted in. The use of multi-modal sensors, specifically visual-thermal cameras, is critical in enabling WiSAR UAVs to perform in diverse operating conditions. However, due to the unique challenges posed by the wilderness context, existing dataset benchmarks are inadequate for developing vision-based algorithms for autonomous WiSAR UAVs. To this end, we present WiSARD, a dataset with roughly 56,000 labeled visual and thermal images collected from UAV flights in various terrains, seasons, weather, and lighting conditions. To the best of our knowledge, WiSARD is the first large-scale dataset collected with multi-modal sensors for autonomous WiSAR operations. We envision that our dataset will provide researchers with a diverse and challenging benchmark that can test the robustness of their algorithms when applied to real-world (life-saving) applications. Link to dataset: https://sites.google.com/uw.edu/wisard/

This study presents a whole-body model predictive control (MPC) of robotic systems with rigid contacts, under a given contact sequence using online switching time optimization (STO). We treat robot dynamics with rigid contacts as a switched system and formulate an optimal control problem of switched systems to implement the MPC. We utilize an efficient solution algorithm for the MPC problem that optimizes the switching times and trajectory simultaneously. The present efficient algorithm, unlike inefficient existing methods, enables online optimization as well as switching times. The proposed MPC with online STO is compared over the conventional MPC with fixed switching times, through numerical simulations of dynamic jumping motions of a quadruped robot. In the simulation comparison, the proposed MPC successfully controls the dynamic jumping motions in twice as many cases as the conventional MPC, which indicates that the proposed method extends the ability of the whole-body MPC. We further conduct hardware experiments on the quadrupedal robot Unitree A1 and prove that the proposed method achieves dynamic motions on the real robot.

For achieving kinematically suitable configurations and highly dynamic task execution, an efficient way is to consider robot performance indices in the whole-body control (WBC) of robots. However, current WBC methods have not considered the intrinsic features of parallel robots, especially motion/force transmissibility (MFT). This paper proposes an MFT-enhanced WBC scheme for parallel-legged robots. Introducing the performance indices of MFT into a WBC is challenging due to the nonlinear relationship between MFT indices and the robot configuration. To overcome this challenge, we establish the MFT preferable space of the robot offline and formulate it as a polyhedron in the joint space at the acceleration level. Then, the WBC employs the polyhedron as a soft constraint. As a result, the robot possesses high-speed and high-acceleration capabilities by satisfying this constraint. The offline preprocessing relieves the online computation burden and helps the WBC achieve a 1kHz servo rate. Finally, we validate the performance and robustness of the proposed method via simulations and experiments on a parallel-legged bipedal robot.

Perceiving the environment through touch is important for robots to reach in cluttered environments, but devising a way to sense without disturbing objects is challenging. This work presents the design and modelling of whisker-inspired sensors that attach to the surface of a robot manipulator to sense its surrounding through light contacts. We obtain a sensor model using a calibration process that applies to straight and curved whiskers. We then propose a sensing algorithm using Bayesian filtering to localize contact points. The algorithm combines the accurate proprioceptive sensing of the robot and sensor readings from the deflections of the whiskers. Our results show that our algorithm is able to track contact points with sub-millimeter accuracy, outperforming a baseline method. Finally, we demonstrate our sensor and perception method in a real-world system where a robot moves in between free-standing objects and uses the whisker sensors to track contacts tracing object contours.

Lidar-based SLAM systems perform well in a wide range of circumstances by relying on the geometry of the environment. However, even mature and reliable approaches struggle when the environment contains structureless areas such as long hallways. To allow the use of lidar-based SLAM in such environments, we propose to add reflector markers in specific locations that would otherwise be difficult. We present an algorithm to reliably detect these markers and two approaches to fuse the detected markers with geometry-based scan matching. The performance of the proposed methods is demonstrated on real-world datasets from several industrial environments.

Current approaches in human-aware or social robot navigation address the humans that are visible to the robot. However, it is also important to address the possible emergences of humans to avoid shocks or surprises to humans and erratic behavior of the robot planner. In this paper, we propose a novel approach to detect and address these human emergences called ‘invisible humans’. We determine the places from which a human, currently not visible to the robot, can appear suddenly and then adapt the path and speed of the robot with the anticipation of potential collisions. This is done while still considering and adapting humans present in the robot's field of view. We also show how this detection can be exploited to identify and address the doorways or narrow passages. Finally, the effectiveness of the proposed methodology is shown through several simulated and real-world experiments.

This paper presents a novel method for smaller-sized humanoid robots to self-calibrate their foot force sensors. The method consists of two steps: 1. The robot is commanded to move along planned whole-body trajectories in different double support configurations. 2. The sensor parameters are determined by minimizing the error between the measured and modeled center of pressure (CoP) and ground reaction force (GRF) during the robot's movement using optimization. This is the first proposed autonomous calibration method for foot force-sensing devices in smaller humanoid robots. Furthermore, we introduce a high-accuracy manual calibration method to establish CoP ground truth, which is used to validate the measured CoP using self-calibration. The results show that the self-calibration can accurately estimate CoP and GRF without any manual intervention. Our method is demonstrated using a NAO humanoid platform and our previously presented force-sensing shoes.

Although many walking control frameworks have been developed to enable biped robots to walk stably on uneven terrain, the foot sole of the robot is also important. Inspired by that study, we have developed a Variable Stiffness Sole (VSS), which is able to adapt to the shape of the obstacles on the ground in Compliant Mode and provide robust support in Stiff Mode. Furthermore, we proposed a walking control framework on uneven terrain for a biped robot equipped with the developed VSS. The proposed walking control framework comprises a posture balance controller to stabilize the zero moment point from disturbances applied to the robot, an ankle torque/foot force controller for walking on uneven terrain, and a VSS controller to change the mode of the VSS. Finally, the proposed framework was verified through walking experiments of RoK-3 equipped with the VSS module on a single obstacle and uneven terrain with various obstacles.

This paper presents an easy-to-deploy, virtual reality-based teleoperation system for controlling a robot arm. The proposed system is based on a consumer-grade virtual reality device (Oculus Quest 2) with a low-cost robot arm (a LoCoBot) to allow easy replication and set up. The proposed Work-from-Home Virtual Reality (WFH-VR) system allows the user to feel an intimate connection with the real remote robot arm. Virtual representations of the robot and objects to be manipulated in the real-world are presented in VR by streaming data pertaining to orientation and poses. The user studies suggest that 1) the proposed telerobotic system is effective under conditions both with and without network latency, whereas a method that simply streams video does not. This design enables the system implemented at an arbitrary distance from the actual work site. 2) The proposed system allows novices to perform manipulation tasks requiring higher dexterity than traditional keyboard controls can support, such as setting tableware. All results, hardware settings, and questionnaire feedback can be obtained at https://arg-nctu.github.io/projects/vr-robot-arm.html.

This paper presents a method for learning logical task specifications and cost functions from demonstrations. Constructing specifications by hand is challenging for complex objectives and constraints in autonomous systems. Instead, we consider demonstrated task executions, whose logic structure and transition costs need to be inferred by an autonomous agent. We employ a spectral learning approach to extract a weighted finite automaton (WFA), approximating the unknown task logic. Thereafter, we define a product between the WFA for high-level task guidance and a labeled Markov decision process for low-level control. An inverse reinforcement learning (IRL) problem is considered to learn a cost function by backpropagating the loss between agent and expert behaviors through the planning algorithm. Our proposed model, termed WFA-IRL, is capable of generalizing the execution of the inferred task specification in a suite of MiniGrid environments.

Creating accurate maps of complex, unknown environments is of utmost importance for truly autonomous navigation robot. However, building these maps online is far from trivial, especially when dealing with large amounts of raw sensor readings on a computation and energy constrained mobile system, such as a small drone. While numerous approaches tackling this problem have emerged in recent years, the mapping accuracy is often sacrificed as systematic approximation errors are tolerated for efficiency's sake. Motivated by these challenges, we propose Voxfield, a mapping framework that can generate maps online with higher accuracy and lower computational burden than the state of the art. Built upon the novel formulation of non-projective truncated signed distance fields (TSDFs), our approach produces more accurate and complete maps, suitable for surface reconstruction. Additionally, it enables efficient generation of Euclidean signed distance fields (ESDFs), useful e.g., for path planning, that does not suffer from typical approximation errors. Through a series of experiments with public datasets, both real-world and synthetic, we demonstrate that our method beats the state of the art in map coverage, accuracy and computational time. Moreover, we show that Voxfield can be utilized as a back-end in recent multi-resolution mapping frameworks, producing high quality maps even in large-scale experiments. Finally, we validate our method by running it onboard a quadrotor, showing it can generate accurate ESDF maps usable for real-time path planning and obstacle avoidance.