IROS 2023 Papers

For a class of biped robots with impulsive dynamics and a non-empty set of passive gaits (unactuated, periodic motions of the biped model), we present a method for computing continuous families of locally optimal gaits with respect to a class of commonly used energetic cost functions (e.g., the integral of torque-squared). We compute these families using only the passive gaits of the biped, which are globally optimal gaits with respect to these cost functions. Our approach fills in an important gap in the literature when computing a library of locally optimal gaits, which often do not make use of these globally optimal solutions as seed values. We demonstrate our approach on a well-studied two-link biped model.

Over the last decades, significant research effort has been put into creating Electromyography (EMG) based controllers for intuitive, hands-free control of robotic arms and hands. To achieve this, machine learning models have been employed to decode human motion and intention using EMG signals as input and to deliver several applications, such as prosthesis control using gesture classification. Despite the advances introduced by new deep learning techniques, real-time control of robot arms and hands using EMG signals as input still lacks accuracy, especially when a plethora of gestures are included as labels in the case of classification. This has been observed to be due to the noise and non-stationarity of the EMG signals and the increased dimensionality of the problem. In this paper, we propose an intuitive, affordances-oriented EMG-based telemanipulation framework for a robot arm-hand system that allows for dexterous control of the device. An external camera is utilized to perform scene understanding and object detection and recognition, providing grasping and manipulation assistance to the user and simplifying control. Object-specific Transformer-based classifiers are employed based on the affordances of the object of interest, reducing the number of possible gesture outputs, dividing and conquering the problem, and resulting in a more robust and accurate gesture decoding system when compared to a single generic classification model. The performance of the proposed system is experimentally validated in a remote telemanipulation setting, where the user successfully performs a set of dexterous manipulation tasks.

In this paper, we present AmbiSense, an acoustic field based sensing system that performs proximity detection and bearing estimation for safer physical human-robot interactions. A single low cost piezoelectric transducer is used to setup this novel acoustic sensing modality to create a blindspot-free sound field engulfing a robot arm. Two detection algorithms leveraging spectral information from reflected audio waves of objects entering the acoustic field are proposed to infer object presence and bearing. We also present a new receiver structure which improves signal to noise ratio (SNR). AmbiSense is paired with a collision avoidance inverse kinematic solver for real world deployment on a Kinova Gen3 robot. Validation is performed using ten test objects generating 2000 proximity and bearing estimation events in real world settings, we show that AmbiSense detects proximity with 93.8% sensitivity and 96.6 % specificity. It estimates bearing and maps it to three zones on a robot link with 100% sensitivity and specificity, while using fewer sensors than state of the art methods for similar coverage.

Mobile robots functioning in human environments should behave with a secure and socially-compliant manner. Although many studies have revealed the effectiveness of Deep Reinforcement Learning (DRL) in robot navigation, most of them can only handle the presence of human as independent individuals. Failing to consider groups may lead to the robot getting stuck or behaving rudely, and omitting to separately handle obstacles from pedestrians will cause low efficiency. In this work, we present a novel all-aware neural network that utilizes DRL to process groups, obstacles, and individuals simultaneously. The proposed solution employs a new Group–Robot Interaction (GRI) subnetwork to encode the mutual effects between groups and the robot, and a modified Obstacle–Robot Unilateral interaction (ORU) subnetwork is presented to avoid obstacle collisions caused by sensing noises or motion uncertainties. In addition, the influences of a pedestrian, obstacle, and group on other pedestrians or groups, that indirectly affect the robot, are also integrated into the Human–Robot Interaction (HRI) subnetwork or GRI subnetwork respectively by using map tensors. Finally, the GRI, ORU, and HRI subnetworks are aggregated into a planning subnetwork to train and derive an all-aware robot navigation policy based on DRL. Evaluation results in both real-world and simulation experiments show that the proposed approach outperforms the current cutting-edge methods.

This paper proposes an illumination-robust visual odometry (VO) system that incorporates both accelerated learning-based corner point algorithms and an extended line feature algorithm. To be robust to dynamic illumination, the proposed system employs the convolutional neural network (CNN) and graph neural network (GNN) to detect and match reliable and informative corner points. Then point feature matching results and the distribution of point and line features are utilized to match and triangulate lines. By accelerating CNN and GNN parts and optimizing the pipeline, the proposed system is able to run in real-time on low-power embedded platforms. The proposed VO was evaluated on several datasets with varying illumination conditions, and the results show that it outperforms other state-of-the-art VO systems in terms of accuracy and robustness. The open-source nature of the proposed system allows for easy implementation and customization by the research community, enabling further development and improvement of VO for various applications.

Line detection is widely used in many robotic tasks such as scene recognition, 3D reconstruction, and simultaneous localization and mapping (SLAM). Compared to points, lines can provide both low-level and high-level geometrical information for downstream tasks. In this paper, we propose a novel learnable edge-based line detection algorithm, AirLine, which can be applied to various tasks. In contrast to existing learnable endpoint-based methods, which are sensitive to the geometrical condition of environments, AirLine can extract line segments directly from edges, resulting in a better generalization ability for unseen environments. To balance efficiency and accuracy, we introduce a region-grow algorithm and a local edge voting scheme for line parameterization. To the best of our knowledge, AirLine is one of the first learnable edge-based line detection methods. Our extensive experiments have shown that it retains state-of-the-art-Ievel precision, yet with a 3-80\times3-80\times runtime acceleration compared to other learning-based methods, which is critical for low-power robots.

Reinforcement learning for swarms of flying robots is a challenging task that requires a large number of data samples. Moreover, the problem of sim-to-real transfer has long been a challenge in robotics algorithm deployment. To address these issues, we propose Air-M, a platform that facilitates large-scale drone swarm learning in a distributed docker container environment and deployment in a virtual reality setting. Air-M trains the policy network using physics engines and creates replicas of agents in docker containers, which helps amortize the computational cost. In addition, Air-M establishes an intermediate link between the simulation and the real world, allowing real drones to interact with virtual objects via virtual sensors. This enables the policy network to be trained using virtual agents and seamlessly transferred to real drones. Air-Mis highly scalable, accommodating hundreds of agents with dynamic models and virtual sensors. We evaluate the effectiveness of our approach by conducting experiments in three representative virtual scenarios with an increasing number of agents. Our results demonstrate that our method outperforms the state-of- the-art in terms of training efficiency and transferability, making it a promising platform for swarm robotics applications.

Autonomous drone racing is becoming an excellent platform to challenge quadrotors' autonomy techniques including planning, navigation and control technologies. However, most research on this topic mainly focuses on single drone scenarios. In this paper, we describe a novel time-optimal trajectory generation method for generating time-optimal trajectories for a swarm of quadrotors to fly through pre-defined waypoints with their maximum maneuverability without collision. We verify the method in the Gazebo simulations where a swarm of 5 quadrotors can fly through a complex 6-waypoint racing track in a 35m\times 35m35m\times 35m space with a top speed of 14m/s. Flight tests are performed on two quadrotors passing through 3 waypoints in a 4m\times 2m4m\times 2m flight arena to demonstrate the feasibility of the proposed method in the real world. Both simulations and real-world flight tests show that the proposed method can generate the optimal aggressive trajectories for a swarm of autonomous racing drones. The method can also be easily transferred to other types of robot swarms.

This paper presents a novel modular robot system that can self-reconfigure to achieve omnidirectional movements for collaborative object transportation. Each robotic module is equipped with a steerable omni-wheel for navigation and is shaped as a regular icositetragon with a permanent magnet installed on each corner for stable docking. After aggregating multiple modules and forming a structure that can cage a target object, we have developed an optimization-based method to compute the distribution of all wheels' heading directions, which enables efficient omnidirectional movements of the structure. By implementing a hierarchical controller on our prototyped system in both simulation and experiment, we validated the trajectory tracking performance of an individual module and a team of six modules in multiple navigation and collaborative object transportation settings. The results demonstrate that the proposed system can maintain a stable caging formation and achieve smooth transportation, indicating the effectiveness of our hardware and locomotion designs.

Efficient and safe multi-agent swarm coordination in environments where humans operate, such as warehouses, assistive living rooms, or automated hospitals, is crucial for adopting automation. In this paper, we augment the obstacle avoidance algorithm based on dynamical system modulation for a swarm of heterogeneous holonomic mobile agents. A smooth prioritization is proposed to change the reactivity of the swarm towards the specific agents. Further, a soft decoupling of the initial agent's kinematics is used to design an independent rotation control to ensure the agent reaches the desired position and orientation simultaneously. This decoupling allowed the introduction of a novel heuristic, the virtual drag. It minimizes the disturbance influence an agent has when moving through its surrounding. Additionally, the safety module adapts the velocity commands from the dynamical system modulation to avoid colliding trajectories between agents. The evaluation was performed in simulated assisted living and hospital environments. The prioritization successfully increased the minimum distance relative to a moving agent. The safety module is observed to create collision-free dynamics where alternative methods fail. Additionally, the repulsive nature of the safety module augments the convergence rate, thus making the proposed method better applicable to dense real-world scenarios.

This paper studies a novel application of an aerial manipulator (AM)-the contact-based landing on a mobile platform. An AM is inherently unstable, under-actuated, and usually loses some DOFs while contacting environments. Meanwhile, the AM's flight state is susceptible to uncertain movements of the mobile platform, such as acceleration, sudden stopping, and reversing. To accomplish the contact-based landing mission, a robust controller is first designed to maintain a steady contact-based flight. Then a hierarchical control framework is applied, integrating the controllers in free-flight and restricted-flight stages. An AM and a mobile platform are developed for contact-based flight experiments. The proposed scheme is reliable and has good repeatability in experiments. To the best of our knowledge, this is the first time an AM has been implemented to conduct a contact-based landing, which is also an innovative landing approach for rotorcraft UAVs.

Adversarial object rearrangement in the real world (e.g., previously unseen or oversized items in kitchens and stores) could benefit from understanding task scenes, which inherently entail heterogeneous components such as current objects, goal objects, and environmental constraints. The semantic relationships among these components are distinct from each other and crucial for multi-skilled robots to perform efficiently in everyday scenarios. We propose a hierarchical robotic manipulation system that learns the underlying relationships and maximizes the collaborative power of its diverse skills (e.g., PICK-PLACE, PUSH) for rearranging adversarial objects in constrained environments. The high-level coordinator employs a heterogeneous graph neural network (HetGNN), which reasons about the current objects, goal objects, and environmental constraints; the low-level 3D Convolutional Neural Network-based actors execute the action primitives. Our approach is trained entirely in simulation, and achieved an average success rate of 87.88% and a planning cost of 12.82 in real-world experiments, surpassing all baseline methods. Supplementary material is available at https://sites.google.com/umn.edu/versatile-rearrangement.

Autonomous vehicle (AV) evaluation has been the subject of increased interest in recent years both in industry and in academia. This paper focuses on the development of a novel framework for generating adversarial driving behavior of background vehicle interfering against the AV to expose effective and rational risky events. Specifically, the adversarial behavior is learned by a reinforcement learning (RL) approach incorporated with the cumulative prospect theory (CPT) which allows representation of human risk cognition. Then, the extended version of deep deterministic policy gradient (DDPG) technique is proposed for training the adversarial policy while ensuring training stability as the CPT action-value function is leveraged. A comparative case study regarding the cut-in scenario is conducted on a high fidelity Hardware-in-the-Loop (HiL) platform and the results demonstrate the adversarial effectiveness to infer the weakness of the tested AV.

Despite significant advances in the design of robotic lower-limb prostheses for individuals with impaired mobility, there is a need for further progress in improving the robustness, safety, and stability of these devices in a wide range of activities of daily living. Although powered prostheses have been able to adapt to different speeds, conditions, and rigid terrains, no control strategies have been proposed for addressing walking over compliant surfaces. This work proposes a continuous admittance controller that adjusts the ankle quasistiffness of a powered ankle-foot prosthesis and improves gait stability during locomotion over compliant terrain. The proposed controller is evaluated with walking experiments on an instrumented treadmill that can accurately change the walking surface stiffness. In these experiments, the proposed controller accurately changes the prosthesis ankle quasi-stiffness across a wide range of 10-20\frac{Nm}{deg}-(10-20\frac{Nm}{deg}-( while improving local dynamic stability compared to a standard phase-variable controller. The proposed controller can significantly improve the performance of lower-limb prostheses in dynamic and compliant environments frequently encountered in daily activities, resulting in improved quality of life for people with lower-limb amputation.

Binocular depth estimation is a fundamental problem in computer vision. Learning-based models have achieved significant performance improvements on public datasets in recent years. Our study finds that the performance of the current state-of-the-art deep learning-based models deteriorates significantly in distant areas. We point out that these deep learning-based models suffer from a scale shrinkage problem. Specifically, the predicted depth value ratio to the ground truth decreases as depth increases. Such a phenomenon is not conducive to the path planning and navigation of intelligent agents in outdoor scenes. We analyze the reasons for the scale shrinkage problem and give a simple and effective method. Our method employs a two-stage fine-tuning strategy and appropriately fuses the predictions of the two-stage models. The method does not reduce the prediction accuracy in close areas and significantly improves the accuracy of the models in distant areas. On the KITTI stereo 2015 dataset, our method can reduce the absolute relative difference by about 6% and the root-mean-square error (RMSE) by about 10%.

Drone grasping on power lines for recharging is challenging since it requires the gripper to be lightweight, carried by a drone, and efficient for a firm grasp. A deep understanding of the power line nature and its magnetic characteristic helps ease such challenges and bring new knowledge to gripper design. In this work, a novel adaptive, lightweight, and fail-safe magnetic gripper with a recharging feature is presented. The gripper exploits the radiated magnetic field of the lines for charging and holding the drone and can easily detach from the line. The gripper design has been validated in the lab and on a quadcopter with a real power line.

In this work, we use recent developments in the field of adaptive robust Model Predictive Control (MPC) to build a controller for bilateral teleoperation systems. To guarantee robust constraint satisfaction, we incorporate polytopic tube controllers in the MPC design. In addition, we use online learning methods to learn the environment model. Namely, we use set membership learning to learn the parametric uncertainty bounds and reduce the conservatism of the robust controller, and we combine it with least mean square method to learn a point estimate of the model parameters, which enhances the controller performance. Our simulation demonstrates the effectiveness of the proposed approach in maintaining robust constraint satisfaction and enhancing performance by learning during teleoperation tasks.

Local-remote systems allow robots to execute complex tasks in hazardous environments such as space and nuclear power stations. However, establishing accurate positional mapping between local and remote devices can be difficult due to time delays that can compromise system performance and stability. Enhancing the synchronicity and stability of localremote systems is vital for enabling robots to interact with environments at greater distances and under highly challenging network conditions, including time delays. We introduce an adaptive control method employing reinforcement learning to tackle the time-delayed control problem. By adjusting controller parameters in real-time, this adaptive controller compensates for stochastic delays and improves synchronicity between local and remote robotic manipulators. To improve the adaptive PD controller's performance, we devise a model-based reinforcement learning approach that effectively incorporates multi-step delays into the learning framework. Utilizing this proposed technique, the local-remote system's performance is stabilized for stochastic communication time-delays of up to 290ms. Our results demonstrate that the suggested model-based reinforcement learning method surpasses the Soft-Actor Critic and augmented state Soft-Actor Critic techniques. Access the code at: https://github.com/CAV-Research-Lab/Predictive-Model-Delay-Correction

Active Learning of Gaussian process (GP) surrogates is an efficient way to model unknown environments in various applications. In this paper, we propose an adaptive exploration-exploitation active learning method (ALX) that can be executed rapidly to facilitate real-time decision making. For the exploration phase, we formulate an acquisition function that maximizes the approximated, expected Fisher information. For the exploitation phase, we employ a closed-form acquisition function that maximizes the total expected variance reduction of the search space. The determination of each phase is established with an exploration condition that measures the predictive accuracy of GP surrogates. Extensive numerical experiments in multiple input spaces validate the efficiency of our method.