ICRA 2023 Papers

Mobile robots are becoming safer and more affordable, and their presence in the workspace is increasing. However, many tasks that involve reasoning, long-term planning or human preferences are still hard to automate. While some solutions in specialised areas slowly emerge, an alternative to full autonomy can be to actively leverage intuition and experience of human operators. To do this, suitable interfaces and modes of interaction have to be explored. Inspired by Real-Time Strategy games, we implement a Mixed Reality interface that can be used with either a Microsoft HoloLens 2 headset or a tablet. The interface allows users to interact with multiple mobile robots simultaneously. We conduct a user study to compare the headset and tablet versions of the interface in different scenarios inspired by a real-world construction setting. We show that, while performance and preference of interface are dependent on the task and the complexity of the required interaction, users are able to solve non-trivial tasks on both platforms using our system.

We study the problem of safe and intention-aware robot navigation in dense and interactive crowds. Most previous reinforcement learning (RL) based methods fail to consider different types of interactions among all agents or ignore the intentions of people, which results in performance degradation. In this paper, we propose a novel recurrent graph neural network with attention mechanisms to capture heterogeneous interactions among agents through space and time. To encourage longsighted robot behaviors, we infer the intentions of dynamic agents by predicting their future trajectories for several timesteps. The predictions are incorporated into a model-free RL framework to prevent the robot from intruding into the intended paths of other agents. We demonstrate that our method enables the robot to achieve good navigation performance and non-invasiveness in challenging crowd navigation scenarios. We successfully transfer the policy learned in simulation to a real-world TurtleBot 2i. Our code and videos are available at https://sites.google.com/view/intention-aware-crowdnav/home.

In this work, we present an Integrated Guidance and Controller (IGC) scheme to drive quadcopters in path-following tasks with obstacle avoidance and constant uncertainty rejection. This scheme is based on the combination of a time-varying artificial vector field and Backstepping with integral action control. The vector field switches between two behaviors: (i) path-following; and (ii) obstacle circumnavigation to allow collision avoidance. This vector field is then integrated into a nonlinear controller designed via Backstepping with Integral Action to deal with the quadcopter vehicle dynamics and reject constant uncertainties. The considered vehicle model is based on quaternion algebra. The control inputs are considered to be the total thrust and torques. Stability is proved by using Lyapunov's Theory and Matrosov's Theorem.

Generating high-quality instance-wise grasp con-figurations provides critical information of how to grasp specific objects in a multi-object environment and is of high importance for robot manipulation tasks. This work proposed a novel Single-Stage Grasp (SSG) synthesis network, which performs high-quality instance-wise grasp synthesis in a single stage: instance mask and grasp configurations are generated for each object simultaneously. Our method outperforms state-of-the-art on robotic grasp prediction based on the OCID-Grasp dataset, and performs competitively on the JACQUARD dataset. The benchmarking results showed significant improvements compared to the baseline on the accuracy of generated grasp configurations. The performance of the proposed method has been validated through both extensive simulations and real robot experiments for three tasks including single object pick-and-place, grasp synthesis in cluttered environments and table cleaning task.

To date, all previous research in the wireless magnetic actuation of untethered helical devices has achieved motion stability using feedback control in vitro. However, feedback control systems are likely to be affected by the increased sensory uncertainty during in vivo trials. In this study we investigate the input-output boundedness of an interconnection between a helical device and a single rotating magnet actuator in low-Reynolds-number regime. Using the resistive-force theory, the interconnection is expressed in terms of all possible input-output pairs. Inputs representing the actuation frequency, pitch angle, lateral speed, and field strength are analyzed numerically and experimentally. We demonstrate input-output boundedness of the states of the helical device during circular and straight runs in open-loop, and we demonstrate bounded input-output propulsion without orienting the angle of attack (the often used input to swim horizontally without vertical drift) of the helical device to counteract gravity. Our results are important for a number of minimally invasive applications and tasks requiring improved control authority for stable runs of helical devices without drift due to gravity and without feedback control and restricted configuration imposed on the helical device's motion.

Intelligent intersection managers can improve safety by detecting dangerous drivers or failure modes in autonomous vehicles, warning oncoming vehicles as they approach an intersection. In this work, we present FailureNet, a recurrent neural network trained end-to-end on trajectories of both nominal and reckless drivers in a scaled miniature city. FailureNet observes the poses of vehicles as they approach an intersection and detects whether a failure is present in the autonomy stack, warning cross-traffic of potentially dangerous drivers. FailureNet can accurately identify control failures, upstream perception errors, and speeding drivers, distinguishing them from nominal driving. The network is trained and deployed with autonomous vehicles in the MiniCity. Compared to speed or frequency-based predictors, FailureNet's recurrent neural network structure provides improved predictive power, yielding upwards of 84% accuracy when deployed on hardware.

Wearable devices have garnered widespread attention as a mobile solution, and various intelligent modules based on wearable devices are increasingly being integrated. Additionally, image captioning is an important task in computer vision that maps images to text. Existing image captioning achievements are based on high-quality visible images. However, higher target complexity and insufficient light can lead to reduced captioning performance and mistakes. In this paper, we present an infrared image captioning framework designed to solve the problem of invalid visible image captioning in special conditions. Remarkably, we integrate the infrared image captioning model into the wearable device. Volunteers perform offline and real-time environmental analysis tasks in the real world to evaluate the framework's effectiveness in multiple scenarios. The results indicate that both the accuracy of infrared image captioning and the feedback from wearable device users are promising.

In this paper, we develop an approach that enables autonomous robots to build and compress semantic environment representations from point-cloud data. Our approach builds a three-dimensional, semantic tree representation of the environment from raw sensor data which is then compressed by a novel information-theoretic tree-pruning approach. The proposed approach is probabilistic and incorporates the uncertainty in semantic classification inherent in real-world environments. Moreover, our approach allows robots to prioritize individual semantic classes when generating the compressed trees, so as to design multi-resolution representations that retain the relevant semantic information while simultaneously discarding unwanted semantic categories. We demonstrate the approach by compressing semantic octree models of a large outdoor, semantically rich, real-world environment. In addition, we show how the octree abstractions can be used to create semantically-informed graphs for motion planning, and provide a comparison of our approach with uninformed graph construction methods such as Halton sequences.

Multi-objective or multi-destination path planning is crucial for mobile robotics applications such as mobility as a service, robotics inspection, and electric vehicle charging for long trips. This work proposes an anytime iterative system to concurrently solve the multi-objective path planning problem and determine the visiting order of destinations. The system is comprised of an anytime informable multi-objective and multi-directional RRT* algorithm to form a simple connected graph, and a solver that consists of an enhanced cheapest insertion algorithm and a genetic algorithm to solve approximately the relaxed traveling salesman problem in polynomial time. Moreover, a list of waypoints is often provided for robotics inspection and vehicle routing so that the robot can preferentially visit certain equipment or areas of interest. We show that the proposed system can inherently incorporate such knowledge to navigate challenging topology. The proposed anytime system is evaluated on large and complex graphs built for real-world driving applications. C++ implementations are available at: https://github.com/UMich-BipedLab/IMOMD-RRTStar.

While in the design phase of a robotic system for the procedures performed in surgical confined spaces or hard-to-reach-deep surgical fields, designers can leverage a systematic method to compare the design alternatives for tele-surgical manipulators quantitatively. Unlike most of the work in the literature, we propose an approach for comparing design alternatives by considering the spurious motions along the length of the manipulator in lieu of existing approaches looking at only the end-effector dexterity measures. We propose a performance measure quantifying these spurious motions while the end-effector executes the application-critical tasks such as suturing and tying a knot. A good manipulator design should yield minimal swept volume along its length portions within the confined space. If informed about these spurious motions, that design would lead to reduced force on the internal organs, reducing the pain and discomfort as well as occurrences of extracorporeal inter-manipulator collisions. To validate the proposed approach, we present two illustrative simulation case studies: (1) two planar rigid link serial robots performing the task of following a desired trajectory and (2) two different architectures of tele-surgical manipulators performing the task of passing a circular suture needle under the fulcrum constraints. The results show the applicability of the proposed performance measure in determining the suitability of a particular design alternative for a given task. Although results are promising, using this measure alone for design optimization may compromise overall device dexterity. Therefore, this measure needs to be incorporated into a weighted optimization framework for robot design.

In this paper, we explore incremental few-shot object detection (iFSD), which incrementally learns novel classes using only a few examples without revisiting base classes. Previous iFSD works achieved the desired results by applying metalearning. However, meta-learning approaches show insufficient performance that is difficult to apply to practical problems. In this light, we propose a simple fine-tuning-based approach, the Incremental Two-stage Fine-tuning Approach (iTFA) for iFSD, which contains three steps: 1) base training using abundant base classes with the class-agnostic box regressor, 2) separation of the RoI feature extractor and classifier into the base and novel class branches for preserving base knowledge, and 3) fine-tuning the novel branch using only a few novel class examples. We evaluate our iTFA on the real-world datasets PASCAL VOC, COCO, and LVIS. iTFA achieves competitive performance in COCO and shows a 30% higher AP accuracy than meta-learning methods in the LVIS dataset. Experimental results show the effectiveness and applicability of our proposed method11Code is available at https://github.com/TMIU/iTFA.

Admittance and impedance controllers are often purely feedforward, using measured external force or motion, respectively, to generate a reference for an inner-loop controller. In this case, the range of dynamics which can be rendered is limited by the inner-loop, which causes, e.g. contact stability issues for low admittance industrial robots in stiff contact. When both position and force are measured, feedback control can be added to more flexibly reshape the rendered dynamics. This paper uses velocity feedback to increase the admittance of motion-controlled industrial robots in force control applications. This allows an industrial robot with a lower intrinsic admittance, which may be needed for payload, speed, or accuracy, to realize a higher admittance by control, allowing lighter manual guidance and safer contact. This is achieved by a modified disturbance observer, where an inverse dynamic model estimates external forces and amplifies them with positive feedback. This approach is compared with using positive velocity feedback with a shaping filter. Here, velocity reference calculated by the virtual admittance model is modified by the DOB (Dist-Add) or the positive velocity feedback (Vel-Add). When combined with an outer-loop admittance controller, these methods can render a higher admittance while maintaining contact stability compared to standard feedforward admittance control.

This paper proposes a novel approach to conduct routine calibration for the changing mechanical parameters over time of a vision-based tactile sensor, without disassembling its overall structure, i.e., in-situ mechanical calibration. Calibration for mechanical parameters, Young's modulus and Poisson's ratio, of a tactile sensor's sensing elastomer, is crucial for its force perception capabilities. However, there are few methods that can retrieve values of these parameters both accurately and conveniently. To address this problem, we propose an in-situ approach to calibrate mechanical parameters other than the verbose traditional evaluation process. This method incorporates the deformation sensing capability of the sensor, the accurate force sensing capability of a force/torque sensor, and most importantly, the deformation-force relation-ship for an indentation with embedded mechanical parameters of the elastomers. We also present the indentation test setup and the complete pipeline to extract Young's modulus and Poisson's ratio from experimental results. We validate the method by comparing the indentation depths simulated through finite element analysis (FEA) using the cali-brated parameters with the indentation depths measured in real experiments. Furthermore, superior contact force distribution can be achieved with the accurate mechanical parameters. The proposed method provides the theoretical basis for accurate, lifelong routine calibration, whether weekly or even daily, which can enhance the applications of tactile sensors in real manipulation scenarios.

Assistance during eating is essential for those with severe mobility issues or eating risks. However, dependence on traditional human caregivers is linked to malnutrition, weight loss, and low self-esteem. For those who require eating assistance, a semi-autonomous robotic platform can provide independence and a healthier lifestyle. We demonstrate an essential capability of this platform: safe, comfortable, and effective transfer of a bite-sized food item from a utensil directly to the inside of a person's mouth. Our system uses a force-reactive controller to safely accommodate the user's motions throughout the transfer, allowing full reactivity until bite detection then reducing reactivity in the direction of exit. Additionally, we introduce a novel dexterous wrist-like end effector capable of small, unimposing movements to reduce user discomfort. We conduct a user study with 11 participants covering 8 diverse food categories to evaluate our system end-to-end, and we find that users strongly prefer our method to a wide range of baselines. Appendices and videos are available at our website: https://tinyurl.com/btICRA.

This paper describes the development of BACH (Belt-Augmented Compliant Hand), a compliant robotic hand equipped with active surfaces. The hand can securely grasp an object using power grasp and simultaneously manipulate the grasped object. The hand consists of three identical fingers, each with an actuated timing belt wrapped around a Fin Ray based compliant finger backbone. Each finger is mounted on a compliant pivot joint allowing for further adaptability. The combination of compliant mechanisms and active surfaces allows the hand to perform dexterous in-hand manipulation with great robustness. Multiple analyses were conducted to optimize and validate the design of BACH. The hand was experimentally tested for grasping and manipulating objects of various geometries and sizes, and it demonstrated highly robust and efficient in-hand manipulation capabilities.

In this paper, we present our approach to efficiently leveraging GPU resources to improve the performance of local bundle adjustment for visual-inertial SLAM. We observe that for local bundle adjustment (i) the Schur complement method, a technique often used to speed up bundle adjustment, has the largest overhead when solving for the parameter update, and (ii) the workload consists of operations on small- to medium-sized matrices. Based on these observations, we develop and combine several techniques that efficiently handle small- to medium-sized matrices. We then implement these techniques as a drop-in replacement block solver for g2o, a library frequently used for bundle adjustment, and integrate it with ORB-SLAM3, a well-known open-source visual-inertial SLAM system. Our evaluation done with two popular datasets, EuRoC and TUM-VI, shows that we can reduce the time taken by local bundle adjustment by 13.81%-33.79% with our techniques across an embedded device and a desktop machine.

Predicting the future trajectories of robots' nearby objects plays a pivotal role in applications such as autonomous driving. While learning-based trajectory prediction methods have achieved remarkable performance on public benchmarks, the generalization ability of these approaches remains questionable. The poor generalizability on unseen domains, a well-recognized defect of data-driven approaches, can potentially harm the real-world performance of trajectory prediction models. We are thus motivated to improve models' generalization ability instead of merely pursuing high accuracy on average. Due to the lack of benchmarks for quantifying the generalization ability of trajectory predictors, we first construct a new benchmark called argoverse-shift, where the data distributions of domains are significantly different. Using this benchmark for evaluation, we identify that the domain shift problem seriously hinders the generalization of trajectory predictors since state-of-the-art approaches suffer from severe performance degradation when facing those out-of-distribution scenes. To enhance the robustness of models against domain shift problem, we propose a plug-and-play strategy for domain normalization in trajectory prediction. Our strategy utilizes the Frenét coordinate frame for modeling and can effectively narrow the domain gap of different scenes caused by the variety of road geometry and topology. Experiments show that our strategy noticeably boosts the prediction performance of the state-of-the-art in domains that were previously unseen to the models, thereby improving the generalization ability of data-driven trajectory prediction methods.

Autonomous navigation in crowded environments is an open problem with many applications, essential for the coexistence of robots and humans in the smart cities of the future. In recent years, deep reinforcement learning approaches have proven to outperform model-based algorithms. Nevertheless, even though the results provided are promising, the works are not able to take advantage of the capabilities that their models offer. They usually get trapped in local optima in the training process, that prevent them from learning the optimal policy. They are not able to visit and interact with every possible state appropriately, such as with the states near the goal or near the dynamic obstacles. In this work, we propose using intrinsic rewards to balance between exploration and exploitation and explore depending on the uncertainty of the states instead of on the time the agent has been trained, encouraging the agent to get more curious about unknown states. We explain the benefits of the approach and compare it with other exploration algorithms that may be used for crowd navigation. Many simulation experiments are performed modifying several algorithms of the state-of-the-art, showing that the use of intrinsic rewards makes the robot learn faster and reach higher rewards and success rates (fewer collisions) in shorter navigation times, outperforming the state-of-the-art.

Existing video super-resolution methods often utilize a few neighboring frames to generate a higher-resolution image for each frame. However, the abundant information in distant frames has not been fully exploited in these methods: corresponding patches of the same instance appear across distant frames at different scales. Based on this observation, we propose to improve the video super-resolution quality with long-term cross-scale aggregation that leverages similar patches (self-exemplars) across distant frames. Our method can be implemented as post-processing for any super-resolution methods to improve performance. Our model consists of a multi-reference alignment module to fuse the features derived from similar patches: we fuse the features of distant references to perform high-quality super-resolution. We also propose a novel and practical training strategy for reference-based super-resolution. To evaluate the performance of our proposed method, we conduct extensive experiments on our collected CarCam dataset, the Waymo Open dataset, and the REDS dataset, and the results demonstrate our method outperforms state-of-the-art methods.