IROS 2022 论文列表

Complex robots in challenging scenarios require constant monitoring of their state and adaptation of their behavior to ensure robustness, reliability and longevity. While known possible errors can be specifically surveilled, other prob-lems can be fully unforeseen, requiring detection systems able to identify novel faults. We detect possible faults as anomalies on various internal sensor data, utilizing unsupervised learning techniques. A bidirectional Wasserstein GAN approach for anomaly detection on multivariate, highly dependent time-series data is implemented and trained on a small amount of non-anomalous robot sensor data. This model is then used for inference on the on-board hardware of a robot without parallel processing units. We evaluate multiple variants of the architecture using manually introduced anomalies in the form of different weights attached to the robot's legs. Overall we are able to show that RoBiGAN is able to consistently detect and localize small anomalies in an online scenario, with little to no robot specific modeling needed.

In robot learning from demonstration (LfD), a visual representation of a cost function inferred from Inverse Reinforcement Learning (IRL) provides an intuitive tool for humans to quickly interpret the underlying objectives of the demonstration. The inferred cost function can be used by controllers, for example, Model Predictive Controllers (MPCs). In this work, we improve the recently developed IRL-MPC framework, by enhancing it in a risk-sensitive formulation to be more applicable for safety-critical applications like autonomous driving. Our risk-sensitive MPCs together with the distributional costmap demonstrate lower collision rates in the CARLA simulator for autonomous driving tasks compared to other learning-based baseline methods.

This paper contributes a novel strategy towards risk-aware motion planning for collision-tolerant aerial robots subject to localization uncertainty. Attuned to the fact that micro aerial vehicles are often tasked to navigate within GPS-denied, possibly unknown, confined and obstacle-filled environments the proposed method exploits collision-tolerance at the robot design level to mitigate the risks of collisions especially as their likelihood increases with growing uncertainty. Accounting for the maximum kinetic energy with which an impact is considered safe, alongside the robot dynamics, the planner builds a set of admissible uncertainty-aware and collision-inclusive paths over a horizon involving multiple motion steps. The first step of the best path is executed by the robot, while the procedure is then repeated in a receding horizon manner. Evaluated in extensive simulation studies and experimental results with a collision-tolerant flying robot, the planner successfully considers the interplay between uncertainty and the likelihood of a collision, balances the risks of possible impacts and enables to navigate safely within highly cluttered environments.

Motion planning in off-road environments re-quires reasoning about both the geometry and semantics of the scene (e.g., a robot may be able to drive through soft bushes but not a fallen log). In many recent works, the world is classified into a finite number of semantic categories that often are not sufficient to capture the ability (i.e., the speed) with which a robot can traverse off-road terrain. Instead, this work proposes a new representation of traversability based exclusively on robot speed that can be learned from data, offers interpretability and intuitive tuning, and can be easily integrated with a variety of planning paradigms in the form of a costmap. Specifically, given a dataset of experienced trajectories, the proposed algorithm learns to predict a distribution of speeds the robot could achieve, conditioned on the environment semantics and commanded speed. The learned speed distribution map is converted into costmaps with a risk-aware cost term based on conditional value at risk (CVaR). Numerical simulations demonstrate that the proposed risk-aware planning algorithm leads to faster average time-to-goals compared to a method that only considers expected behavior, and the planner can be tuned for slightly slower, but less variable behavior. Furthermore, the approach is integrated into a full autonomy stack and demonstrated in a high-fidelity Unity environment and is shown to provide a 30% improvement in the success rate of navigation.

This work presents a humanoid soft robotics finger design with rigid skeletons and proprioceptive sensors. This 4-DOFs dexterous finger has soft joints and rigid phalanxes, which is about the size of human hand. To enhance the overall stiffness and for human-like behavior and configuration, rigid-soft actuators which we called quasi-joints are introduced. Although their lengths are shortened in this design, the soft actuators can still bend over 90°, exhibiting joint-like flexion and abduction/adduction. Thus interphalangeal joints (IPs) and metacarpophalangeal joint (MCP) are realized. EGaIn soft sensors are embedded into the structure for bending detection. In addition, multi-step molding fabrication method is introduced for this complex multi-material structure. This rigid-soft finger is a preliminary work and modular part of a highly dexterous humanoid soft robotic hand.

We present a framework to generate periodic trajectory references for a 3D under-actuated bipedal robot, using a linear inverted pendulum (LIP) based controller with adaptive neural regulation. We use the LIP template model to estimate the robot's center of mass (CoM) position and velocity at the end of the current step, and formulate a discrete controller that determines the next footstep location to achieve a desired walking profile. This controller is equipped on the frontal plane with a Neural-Network-based adaptive term that reduces the model mismatch between the template and physical robot that particularly affects the lateral motion. Then, the foot placement location computed for the LIP model is used to generate task space trajectories (CoM and swing foot trajectories) for the actual robot to realize stable walking. We use a fast, real-time QP-based inverse kinematics algorithm that produces joint references from the task space trajectories, which makes the formulation independent of the knowledge of the robot dynamics. Finally, we implemented and evaluated the proposed approach in simulation and hardware experiments with a Digit robot obtaining stable periodic locomotion for both cases.

Cyber-attacks, failures, and implementation errors inside the controller of an autonomous system can affect its correct behavior leading to unsafe states and degraded performance. In this paper, we focus on such problems specifically on cyber-attacks that manipulate controller parameters like the gains in a feedback controller or that triggers different behaviors or block inputs based on specific values of the state and tracking error. If such attacks are undetected, they can lead to the partial or complete loss of system's control authority, resulting in a hijacking and leading the autonomous system towards unforeseen states. To deal with this problem, we propose a runtime monitoring and recovery scheme in which: 1) we leverage the residual between the expected and the received measurements to detect inconsistencies in the generated inputs and 2) provide a recovery method for counteracting the malicious effects to allow for resilient operations by manipulating the reference signal and state vector provided to the system to avoid the affected regions in the state and error space. We validate our approach with Matlab simulations and experiments on unmanned ground vehicles resiliently performing operations in the presence of malicious attacks to on-board controllers.

Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/

Deep reinforcement learning (RL) is a promising approach to solving complex robotics problems. However, the process of learning through trial-and-error interactions is often highly time-consuming, despite recent advancements in RL algorithms. Additionally, the success of RL is critically dependent on how well the reward-shaping function suits the task, which is also time-consuming to design. As agents trained on a variety of robotics problems continue to proliferate, the ability to reuse their valuable learning for new domains becomes increasingly significant. In this paper, we propose a post-hoc technique for policy fusion using Optimal Transport theory as a robust means of consolidating the knowledge of multiple agents that have been trained on distinct scenarios. We further demonstrate that this provides an improved weights initialisation of the neural network policy for learning new tasks, requiring less time and computational resources than either retraining the parent policies or training a new policy from scratch. Ultimately, our results on diverse agents commonly used in deep RL show that specialised knowledge can be unified into a “Renaissance agent”, allowing for quicker learning of new skills.

Assistive robot systems have been developed to help people accomplish daily manipulation tasks especially for those with disabilities, where scene understanding plays a crucial role in enabling robots to interpret the surroundings and behave accordingly. Most of the current systems approach scene understanding without considering the functional dependencies between objects. However, it is only valuable to interact with some objects when their function-relevant counterparts are considered. In this paper, we augment an assistive robotic arm system with an end-to-end semantic relationship reasoning model. It incorporates functional relationships between pairs of objects for semantic scene understanding. To ensure good generalization to unseen objects and relationships, the model works in a category-agnostic manner. We evaluate our design and three baseline methods on a self-collected benchmark with two levels of difficulty. To further demonstrate the effectiveness, the model is integrated with a symbolic planner for goal-oriented, multi-step manipulation task on a real-world assistive robotic arm platform.

Autonomous robots should operate in real-world dynamic environments and collaborate with humans in tight spaces. A key component for allowing robots to leave structured lab and manufacturing settings is their ability to evaluate online and real-time collisions with the world around them. Distance-based constraints are fundamental for enabling robots to plan their actions and act safely, protecting both humans and their hardware. However, different applications require different distance resolutions, leading to various heuristic approaches for measuring distance fields w.r.t. obstacles, which are computationally expensive and hinder their application in dynamic obstacle avoidance use-cases. We propose Regularized Deep Signed Distance Fields (ReDSDF), a single neural implicit function that can compute smooth distance fields at any scale, with fine-grained resolution over high-dimensional manifolds and articulated bodies like humans, thanks to our effective data generation and a simple inductive bias during training. We demonstrate the effectiveness of our approach in representative simulated tasks for whole-body control (WBC) and safe Human- Robot Interaction (HRI) in shared workspaces. Finally, we provide proof of concept of a real-world application in a HRI handover task with a mobile manipulator robot.

Remotely programming robots to execute tasks often relies on registering objects of interest in the robot's environment. Frequently, these tasks involve articulating objects such as opening or closing a valve. However, existing human-in-the-loop methods for registering objects do not consider articulations and the corresponding impact to the geometry of the object, which can cause the methods to fail. In this work, we present an approach where the registration system attempts to automatically determine the object model, pose, and articulation for user-selected points using nonlinear fitting and the iterative closest point algorithm. When the fitting is incorrect, the operator can iteratively intervene with corrections after which the system will refit the object. We present an implementation of our fitting procedure for one degree-of-freedom (DOF) objects with revolute joints and evaluate it with a user study that shows that it can improve user performance, in measures of time on task and task load, ease of use, and usefulness compared to a manual registration approach. We also present a situated example that integrates our method into an end-to-end system for articulating a remote valve.

Safety filters based on Control Barrier Functions (CBFs) have emerged as a practical tool for the safety-critical control of autonomous systems. These approaches encode safety through a value function and enforce safety by imposing a constraint on the time derivative of this value function. How-ever, synthesizing a valid CBF that is not overly conservative in the presence of input constraints is a notorious challenge. In this work, we propose refining a candidate CBF using formal verification methods to obtain a valid CBF. In particular, we update an expert-synthesized or backup CBF using dynamic programming (DP) based reachability analysis. Our framework, REFINECBF, guarantees that with every DP iteration the obtained CBF is provably at least as safe as the prior iteration and converges to a valid CBF. Therefore, REFINECBF can be used in-the-loop for robotic systems. We demonstrate the practicality of our method to enhance safety and/or reduce conservativeness on a range of nonlinear control-affine systems using various CBF synthesis techniques in simulation.

In this paper, a computationally efficient model predictive control (MPC) is proposed for the trajectory tracking of cable-driven robots subject to state/input constraints. While MPC has been an effective tool in dealing with various constraints, the primary drawback is the high computational load caused by the non-convexity of the corresponding optimization problem. In order to avoid the non-convexity, the prediction model in the proposed reference acceleration MPC (RA-MPC) is simplified into a linear one by assuming the reference accelerations being taken in the future horizon steps. As a result, RA-MPC only optimizes for the instantaneous joint accelerations and the corresponding actuator commands for the current step, resulting in a convex quadratic program that can be efficiently solved. It is further shown that by properly selecting parameters, RA-MPC can be interpreted as ‘soft-CTC’ and ‘soft-LQR’, where the joint acceleration is allowed to deviate from the corresponding desired value, computed from a PD gain or an LQR gain. The effectiveness of the proposed RA-MPC are demonstrated in both simulation and hardware experiment using cable-driven robots.

Whole-body control (WBC) with task priority transition is an important technology for robots to switch multiple behaviors, change different objectives, and adapt to various environments. Many methods have solved the problem of control continuity in the priority transition process. However, they either increased the computation consumption or sacrificed the accuracy of tasks in practical application. In this work, we propose a Recursive Hierarchical Projection (RHP) matrix and introduce it in Hierarchical Quadratic Programming (HQP). This RHP-HQP scheme can form continuously changing hierarchical projection and regard the WBC problem with task priority transition as a unified formulation. This unified formulation can be smoothly transitioned without increasing computation consumption and solved without losing task accuracy. The comparative simulations of the reactive collision avoidance verify that this priority transition scheme can guarantee high computational efficiency and task accuracy.

In this paper we consider the information path-planning problem for a single robot in a stochastic environment with static obstacles subject to a preassigned constraint on the distance it can travel. Given a set of candidate sampling locations, the objective is to determine a path for the robot that allows to visit as many sampling locations as possible to accurately reconstruct an unknown underlying scalar field while not exceeding the assigned travel budget. Starting from the assumption that the phenomenon being measured can be modeled by a Gaussian Process, our algorithm balances exploration and exploitation to determine a sequence of locations ensuring that a preassigned final site is reached before the budget is consumed. Using mutual information as a reward criterion, as well as a generative model to predict consumed energy, the algorithm iteratively determines where to sample next, and when to end the mission. Our findings are validated in simulation in various scenarios and lead to a better reconstruction with less failures when compared with other methods.

Anomaly detection is an important problem in computer vision; however, the scarcity of anomalous samples makes this task difficult. Thus, recent anomaly detection methods have used only “normal images” with no abnormal areas for training. In this work, a powerful anomaly detection method is proposed based on student-teacher feature pyramid matching (STPM), which consists of a student and teacher network. Generative models are another approach to anomaly detection. They reconstruct normal images from an input and compute the difference between the predicted normal and the input. Unfortunately, STPM does not have the ability to generate normal images. To improve the accuracy of STPM, this work uses a student network, as in generative models, to reconstruct normal features. This improves the accuracy; however, the anomaly maps for normal images are not clean because STPM does not use anomaly images for training, which decreases the accuracy of the image-level anomaly detection. To further improve accuracy, a discriminative network trained with pseudo-anomalies from anomaly maps is used in our method, which consists of two pairs of student-teacher networks and a discriminative network. The method displayed high accuracy on the MVTec anomaly detection dataset.

We investigated the use of impact sounds generated during exploratory behaviors in a robotic manipulation setup as cues for predicting object surface material and for recognizing individual objects. We collected and make available the YCB-impact sounds dataset which includes over 3,000 impact sounds for the YCB set of everyday objects lying on a table. Impact sounds were generated in three modes: (i) human holding a gripper and hitting, scratching, or dropping the object; (ii) gripper attached to a teleoperated robot hitting the object from the top; (iii) autonomously operated robot hitting the objects from the side with two different speeds. A convolutional neural network is trained from scratch to recognize the object material (steel, aluminium, hard plastic, soft plastic, other plastic, ceramic, wood, paper/cardboard, foam, glass, rubber) from a single impact sound. On the manually collected dataset with more variability in the speed of the action, nearly 60% accuracy for the test set (not presented objects) was achieved. On a robot setup and a stereotypical poking action from top, accuracy of 85% was achieved. This performance drops to 79% if multiple exploratory actions are combined. Individual objects from the set of 75 objects can be recognized with a 79% accuracy. This work demonstrates promising results regarding the possibility of using impact sound for recognition in tasks like single-stream recycling where objects have to be sorted based on their material composition.

Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross validation to ensure generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enable near-real time surgical activity recognition and prediction.