IROS 2023 Papers

The fingerprint effect describes the relationship between slip speed, fingerprint ridge spacing, and the frequency of vibrations created by the movement of a fingerprint across a surface. We have previously shown that the spacing between straight, parallel, evenly spaced ridges in fingerprint-like structures, and thus the vibrations produced by the fingerprint effect, are dependent on the orientation of the ridges with respect to the direction of movement. We also showed that, when ridge orientation is known, the fingerprint effect can be used to estimate slip speed in real-time. The physical processes behind the fingerprint effect also apply to the interaction between a surface and other, non-ridge, microstructures. It is, therefore, theoretically possible to use the fingerprint effect generated by these structures to estimate slip speed. However, it is first necessary to understand the nature of the fingerprint effect generated by these non-ridge structures. In this paper, we show that digitized structures, evenly spaced in columns and rows, have a more complex relationship to the fingerprint effect than ridges do. At most orientations, these structures produce vibrations amplified around four frequencies, each determined by a set of virtual ridges defined by the digitized structures. A sensor with \mathbf{100}\ \boldsymbol{\mu} \mathbf{m}\mathbf{100}\ \boldsymbol{\mu} \mathbf{m} tall, \mathbf{150}\ \boldsymbol{\mu} \mathbf{m}\mathbf{150}\ \boldsymbol{\mu} \mathbf{m} wide micropillars in evenly spaced rows and columns, with a spacing of \mathbf{300}\ \boldsymbol{\mu} \mathbf{m}\mathbf{300}\ \boldsymbol{\mu} \mathbf{m} center-to-center, is fabricated. This sensor is tested at angles between 0° and 90° by 15° increments. The results support our theoretical analysis.

Often in the field of haptic guidance, an important question is how the robotic device should assist some imperfect human movement. While many control strategies have been suggested to help improve human performance in particular tasks, structuring guidance in a generalizable way remains elusive. Many assistive controllers rely on predicting a user's goal movement or knowing some idealized trajectory a-priori but may fail to assist during an arbitrary task. In this study, we propose a ‘flipped’ approach to studying human-robot collaborative behavior - we ask humans to assist a robotic device whose movements are in some way imperfect. We conducted an experiment during which subjects assisted a haptic device performing a shape-following task autonomously but with different types of error-prone controllers. For each shape, we evaluated a simple trajectory-following controller as well as one with human-like motion constraints. We also evaluated the role of visual feedback on a user's ability to help the robot accomplish the unknown task. We found that the human was generally able to improve the robotic error in all trajectories when the robotic motion did not include gravity compensation; however, error reduction was primarily in the vertical direction. When the robotic controller included gravity compensation, the human user was not able to improve errors significantly, except for vertical errors when provided visual feedback. In the no visual feedback conditions, the human user contributed to significantly greater error for most paths compared to a robot with gravity compensation, indicating an inability to provide assistance in that case.

This work briefly covers our efforts to stabilize the flight dynamics of Northeatern's tailless bat-inspired micro aerial vehicle, Aerobat. Flapping robots are not new. A plethora of examples is mainly dominated by insect-style design paradigms that are passively stable. However, Aerobat, in addition for being tailless, possesses morphing wings that add to the inherent complexity of flight control. The robot can dynamically adjust its wing platform configurations during gaitcycles, increasing its efficiency and agility. We employ a guard design with manifold small thrusters to stabilize Aerobat's position and orientation in hovering, a flapping system in tandem with a multi-rotor. For flight control purposes, we take an approach based on assuming the guard cannot observe Aeroat's states. Then, we propose an observer to estimate the unknown states of the guard which are then used for closed-loop hovering control of the Guard-Aerobat platform.

Current parking slot detection in advanced driver-assistance systems (ADAS) primarily relies on ultrasonic sen-sors. This method has several limitations such as the need to scan the entire parking slot before detecting it, the incapacity of detecting multiple slots in a row, and the difficulty of classifying them. Due to the complex visual environment, vehicles are equipped with surround view camera systems to detect vacant parking slots. Previous research works in this field mostly use image-domain models to solve the problem. These two-stage approaches separate the 2D detection and 3D pose estimation steps using camera calibration. In this paper, we propose one-step Holistic Parking Slot Network (HPS-Net), a tailor-made adaptation of the You Only Look Once (YOLO)v4 algorithm. This camera-based approach directly outputs the four vertex coordinates of the parking slot in topview domain, instead of a bounding box in raw camera images. Several visible points and shapes can be proposed from different angles. A novel regression loss function named polygon-corner Generalized Intersection over Union (GIoU) for polygon vertex position optimization is also proposed to manage the slot orientation and to distinguish the entrance line. Experiments show that HPS-Net can detect various vacant parking slots with a F1-score of 0.92 on our internal Valeo Parking Slots Dataset (VPSD) and 0.99 on the public dataset PS2.0. It provides a satisfying generalization and robustness in various parking scenarios, such as indoor (F1: 0.86) or paved ground (F1: 0.91). Moreover, it achieves a realtime detection speed of 17 FPS on Nvidia Drive AGX Xavier. A demo video can be found at https://streamable.com/75j7sj.

The trend towards Industry 4.0 demands an increasing flexibility of system configurations including more agile collaborative human-robot systems that can quickly adapt to new tasks and missions. At the same time, state-of-the-art safety solution for many industrial robots is the use of barriers around the robot to limit the exposure to human co-workers. As these barriers cannot be reconfigured easily, it considerably limits the flexibility of many robotic solutions. Thus, new more agile safety solutions are required, and these require new, robust perception systems that can ensure detection of all safety relevant objects. In this paper, we propose a novel depth perception approach called HistoDepth, which addresses this need. It uses depth sensors that are mounted at fixed, static positions outside of the workspace and tracks each pixel measurement statistically over time. This enables our solution to differentiate static scene elements from dynamic (potentially hazardous) elements and to enforce a robot safety maneuver upon detection of a critical object. Moreover, it can adapt automatically to changing noise or scene configurations without user input or the need for setup changes. Using an UR5 robot arm, we demonstrate that this solution can enable new agile and flexible safety concepts for collaborative robots.

Safe autonomous driving critically depends on how well the ego-vehicle can predict the trajectories of neighboring vehicles. To this end, several trajectory prediction algorithms have been presented in the existing literature. Many of these approaches output a multimodal distribution of obstacle trajectories instead of a single deterministic prediction to account for the underlying uncertainty. However, existing planners cannot handle the multimodality based on just sample-level information of the predictions. With this motivation, this paper proposes a trajectory optimizer that can leverage the distributional aspects of the prediction in a computationally tractable and sample-efficient manner. Our optimizer can work with arbitrarily complex distributions and thus can be used with output distribution represented as a deep neural network. The core of our approach is built on embedding distribution in Reproducing Kernel Hilbert Space (RKHS), which we leverage in two ways. First, we propose an RKHS embedding approach to select probable samples from the obstacle trajectory distribution. Second, we rephrase chance-constrained optimization as distribution matching in RKHS and propose a novel sampling-based optimizer for its solution. We validate our approach with handcrafted and neural network-based predictors trained on real-world datasets and show improvement over the existing stochastic optimization approaches in safety metrics.

Over the years, soft growing robots that allow the feeding of new materials at their tips have attracted considerable attention owing to their unique locomotion characteristics. However, accessing targets over highly curved passages by steering compliant continuum bodies remains challenging. To this end, this study proposes a new tip steering mechanism that imparts soft growing robots with consecutive high-curvature steering ability. We place a hyper-redundant rolling contact joint and twisted string actuator (TSA) at the tip of a soft growing robot to provide high-curvature continuous steering with a smaller scale factor, allowing consecutive turns. The proposed small form factor continuous curvature tip steering mechanism allows the steering mechanism to stay at the tip during steering, improving accessibility by enabling consecutive turns. The improved accessibility of the proposed mechanism is demonstrated based on better confined space reachability compared with that of conventional whole-body steering soft growing robots, along with a consecutive high-curvature pipe navigation demonstration.

Lab automation has the potential to accelerate scientific progress in the natural sciences, allowing tedious experiments that would require many hours of human time to be automated, enabling higher accuracy, efficiency, and repeatability. Mobile manipulation robots have the potential to work in chemistry labs designed for humans to complete tasks for which setting up customized factory-scale automation is premature or infeasible. We present a new method to enable a mobile manipulation robot to automate injections, a common task in chemistry labs when using equipment such as gas chromatographs (GCs) for analyzing the contents of a sample mixture. This task is challenging for a mobile manipulation robot due to the need to navigate to the equipment in the lab and then achieve millimeter-scale accuracy required for the syringe positioning. Our approach leverages deep learning to create a model capable of localizing the syringe with high accuracy using cameras mounted on the chemistry equipment, and then uses a visual servoing approach based on the syringe's needle localization to achieve the injection. We demonstrate that our approach is robust to uncertainty in navigation as well as uncertainty in the grasping position and orientation of the syringe, achieving errors sufficiently small to enable the mobile manipulation robot to automate injections in real chemistry equipment.

Recognition, detection, and segmentation tasks in machine vision have focused on studying the physical and textural attributes of objects. However, robots and intelligent machines require the ability to understand visual cues, such as the visual affordances that objects offer, to interact intelligently with novel objects. In this paper, we present a large-scale multi-view RGBD visual affordance learning dataset a benchmark of 47,210 RGBD images from 37 object categories, annotated with 15 visual affordance categories and 35 cluttered/complex scenes. We deploy a Vision Transformer (ViT), called Visual Affordance Transformer (VAT), for the affordance segmentation task. Due to its hierarchical architecture, VAT can learn multiple affordances at various scales, making it suitable for objects of varying sizes. Our experimental results show the superior performance of VAT compared to state-of-the-art deep learning networks. In addition, the challenging nature of the proposed dataset highlights the potential for new and robust affordance learning algorithms. Our dataset is publicly available at https://sites.google.com/view/afaqshah/dataset.

Segmenting and recognizing surgical operation trajectories into distinct, meaningful gestures is a critical preliminary step in surgical workflow analysis for robot-assisted surgery. This step is necessary for facilitating learning from demonstrations for autonomous robotic surgery, evaluating surgical skills, and so on. In this work, we develop a hierarchical semi-supervised learning framework for surgical gesture segmentation using multi-modality data (i.e. kinematics and vision data). More specifically, surgical tasks are initially segmented based on distance characteristics-based profiles and variance characteristics-based profiles constructed using kinematics data. Subsequently, a Transformer-based network with a pre-trained ‘ResNet-18’ backbone is used to extract visual features from the surgical operation videos. By combining the potential segmentation points obtained from both modalities, we can determine the final segmentation points. Furthermore, gesture recognition can be implemented based on supervised learning. The proposed approach has been evaluated using data from the publicly available JIGSAWS database, including Suturing, Needle Passing, and Knot Tying tasks. The results reveal an average F1 score of 0.623 for segmentation and an accuracy of 0.856 for recognition. For more details about this paper, please visit our website: https://sites.google.com/view/surseg/home.

The safety-critical control of robotic systems often must account for multiple, potentially conflicting, safety constraints. This paper proposes novel relaxation techniques to address safety-critical control problems in the presence of conflicting safety conditions. In particular, Control Barrier Functions (CBFs) provide a means to encode safety as constraints in a Quadratic Program (QP), wherein multiple safety conditions yield multiple constraints. However, the QP problem becomes infeasible when the safety conditions cannot be simultaneously satisfied. To resolve this potential infeasibility, we introduce a hierarchy between the safety conditions and employ an additional variable to relax the less important safety conditions (Relaxed-CBF-QP). We also formulate a cascaded structure to achieve smaller violations of lower-priority safety conditions (Hierarchical-CBF-QP). The proposed approach, therefore, ensures the existence of at least one solution to the QP problem with the CBFs while dynamically balancing enforcement of additional safety constraints. Importantly, this paper evaluates the impact of different weighting factors in the Hierarchical-CBF-QP and, due to the sensitivity of these weightings in the observed behavior, proposes a method to determine the weighting factors via a sampling-based technique. The validity of the proposed approach is demonstrated through simulations and experiments on a quadrupedal robot navigating to a goal through regions with different levels of danger.

Many applications of imitation learning require the agent to generate the full distribution of behaviour observed in the training data. For example, to evaluate the safety of autonomous vehicles in simulation, accurate and diverse behaviour models of other road users are paramount. Existing methods that improve this distributional realism typically rely on hierarchical policies. These condition the policy on types such as goals or personas that give rise to multi-modal behaviour. However, such methods are often inappropriate for stochastic environments where the agent must also react to external factors: because agent types are inferred from the observed future trajectory during training, these environments require that the contributions of internal and external factors to the agent behaviour are disentangled and only internal factors, i.e., those under the agent's control, are encoded in the type. Encoding future information about external factors leads to inappropriate agent reactions during testing, when the future is unknown and types must be drawn independently from the actual future. We formalize this challenge as distribution shift in the conditional distribution of agent types under environmental stochasticity. We propose Robust Type Conditioning (RTC), which eliminates this shift with adversarial training under randomly sampled types. Experiments on two domains, including the large-scale Waymo Open Motion Dataset, show improved distributional realism while maintaining or improving task performance compared to state-of-the-art baselines.

Sequence models in reinforcement learning require task knowledge to estimate the task policy. This paper presents the hierarchical decision transformer (HDT). HDT is a hierarchical behavior cloning algorithm that improves the performance of transformer methods in imitation learning, improving their robustness to tasks with longer episodes and/or sparse rewards, without requiring task knowledge or user interaction currently present in the state-of-the-art. The high-level mechanism guides the low-level controller through the task by selecting sub-goals for the latter to reach. This sequence replaces the returns-to-go of previous methods, improving its performance overall, especially in tasks with longer episodes and scarcer rewards. We validate our method in multiple tasks of OpenAI Gym, D4RL, and RoboMimic benchmarks. Our method outperforms the baselines in twenty three out of thirty one settings of varied horizons and reward frequencies without prior task knowledge, showing the advantages of the hierarchical model approach for learning from demonstrations using a sequence model. We also evaluate the method on a reaching task on a physical robot.

The accurate prediction of the neighboring vehicles' trajectories affects the security of autonomous driving vehicles. However, it is challenging for existing methods to anticipating the trajectories of vehicles in the vicinity due to the uncertainty of driving behaviors and the complex interaction patterns of traffic flows. In this study, incorporating the planning information of the ego vehicle, we propose a novel trajectory prediction approach based on the hierarchical attention mechanism. Firstly, a spatio-temporary attention module is presented to extract the social interaction of surrounding vehicles and capture the temporal dependence of continuous frame historical information and planning information. Then, a hard-soft attention module is designed to perform two tasks: weighing the importance of both historical and future information, and learning different location information about the target vehicles. Our method is evaluated on two national highway datasets. The experimental results show that our algorithm achieves the state-of-the-art performance.

Despite the potential benefits of collaborative robots, effective manipulation tasks with quadruped robots remain difficult to realize. In this paper, we propose a hierarchical control system that can handle real-world collaborative manipulation tasks, including uncertainties arising from object properties, shape, and terrain. Our approach consists of three levels of controllers. Firstly, an adaptive controller computes the required force and moment for object manipulation without prior knowledge of the object's properties and terrain. The computed force and moment are then optimally distributed between the team of quadruped robots using a Quadratic Programming (QP)-based controller. This QP-based controller optimizes each robot's contact point location with the object while satisfying constraints associated with robot-object contact. Finally, a decentralized loco-manipulation controller is designed for each robot to apply manipulation force while maintaining the robot's stability. We successfully validated our approach in a high-fidelity simulation environment where a team of quadruped robots manipulated an unknown object weighing up to 18 kg on different terrains while following the desired trajectory.

Isolation wards operate in quarantine rooms to prevent cross-contamination caused by infectious diseases. Behind the benefits, medical personnel can have the infection risk from patients and the heavy workload due to the isolation. This work proposes a robot-assisted system to alleviate these problems in isolation wards. We conducted a survey about the medical staff's difficulties and envisioning robots. Using the investigation result, we devised three valuable services using two kinds of heterogeneous robots: telemedicine, emergency alert, and delivery services by care robots and delivery robots. Our system also provides user-interactive components such as a dashboard for medical staff and a patient app for inpatients. To manage the services efficiently, we suggest the robotic system based on a central control server and a hierarchical management architecture. Through a user study, we reviewed the usability of the developed system and its future directions.

We present an approach to task scheduling in heterogeneous multi-robot systems. In our setting, the tasks to complete require diverse skills. We assume that each robot is multi-skilled, i.e., each robot offers a subset of the possible skills. This makes the formation of heterogeneous teams (coalitions) a requirement for task completion. We present two centralized algorithms to schedule robots across tasks and to form suitable coalitions, assuming stochastic travel times across tasks. The coalitions are dynamic, in that the robots form and disband coalitions as the schedule is executed. The first algorithm we propose guarantees optimality, but its runtime is acceptable only for small problem instances. The second algorithm we propose can tackle large problems with short runtimes, and is based on a heuristic approach that typically reaches 1x-2x of the optimal solution cost.

One approach to the wireless actuation and gravity compensation of untethered helical magnetic devices (UHMD) is through swimming with a non-zero angle of attack (AoA). This configuration allows us to counteract gravity, so that for a given desired path, we can move the UHMD controllably without drifting downward under its own weight. This study seeks to investigate the use a reduced-order model of the complex 6-degrees-of-freedom model of UHMDs in low Reynolds-number regime. A one-dimensional model representing the relative position of the UHMD with respect to an actuator rotating permanent magnet is used to predict a gap which yields bounded behavior of the open-loop system. Using geometric representation of the reduced-order model, the local bounded behavior of the UHMD with near-zero AoA is attributed to periodic active magnetic suspension, which dominates near-zero AoA. Our numerical results are verified experimentally and bounded behavior of the UHMD demonstrates the capability to swim with near-zero AoA (6.3° ± 2.2°) without drifting downward. With this actuation strategy, it is unlikely that the orientation of the UHMD will be needed during noninvasive localization, making the control system dependent on only its position with respect to a prescribed trajectory. This strategy will also provide a computational advantage in adjusting the gap between the UHMD and a robotically controlled rotating permanent magnet actuator.

In the context of physical human-(tele)robot interaction, passivity-based stabilizers have been used to guarantee the physical or (tele) physical stability. In most of these examples, human biomechanics is considered an inherently passive system that dissipates energy. This assumption may not hold true when the interaction is implemented in the force-position domain, even though such a setting would be needed to boost positional accuracy and avoid the common kinematic drifts in the force-velocity domains. The aforementioned topic is examined in this paper using the concept of shortage versus excess of passivity index for human biomechanics in the force-position domain. We also investigate the compounding effect of the frequency of interaction. The outcomes of this paper will be imperative for the design of force-position domain pURI stabilizers when the classical assumption of passivity of human biomechanics can lead to serious safety issues. In this work, for the first time, we quantitatively present the passivity margin and, thus, the energetic behavior of the human arm's biomechanics under various interaction scenarios in the Force-Position domain. The outcome of this work includes a three-dimensional passivity index map (3DPiM) that is validated on five healthy participants. The goal is to illustrate the passivity margin of the human upper limb biomechanics for two distinct levels of muscle co-contractions, as indicated by the Electromyography (EMG) signal, across four interaction frequencies and eight geometric directions. This outcome enables the future development of biomechanics-aware stabilizers in the force-position domain, quantifying the passivity margin in real-time and thus significantly reducing the stabilizer's conservatism while ensuring the safety of human-robot interactions.