ICRA 2023 Papers

As robots are widely used in assisting manual tasks, an interesting challenge is: Can mobile robots help create a labeled knowledge dataset that can be used for efficiently creating deep learning models for other sensors? This paper proposes an Unsupervised Person Labeling and Identification (UPLIFT) framework to automatically enlarge the labeled knowledge dataset. Typically, manual data labeling is very costly, especially when the user population is large and dynamic. To reduce the cost, we use a mobile robot to serve as a knowledge seed and to provide the pseudo-ground-truth for the system so that unlabeled images from other fixed surveillance cameras can be paired with the pseudo-ground-truth. Ultimately, the knowledge dataset can be generated via a system-to-system knowledge transfer process from the former to the latter and gradually expanded as the system operates longer. Experimental results in two environments indicate that UPLIFT achieves an accuracy of 94.1% on average to detect pedestrians' IDs every 10 seconds.

In this paper, we present a fast monocular depth estimation method for enabling 3D perception capabilities of low-cost underwater robots. We formulate a novel end-to-end deep visual learning pipeline named UDepth, which incorporates domain knowledge of image formation characteristics of natural underwater scenes. First, we adapt a new input space from raw RGB image space by exploiting underwater light attenuation prior, and then devise a least-squared formulation for coarse pixel-wise depth prediction. Subsequently, we extend this into a domain projection loss that guides the end-to-end learning of UDepth on over 9K RGB-D training samples. UDepth is designed with a computationally light MobileNetV2 backbone and a Transformer-based optimizer for ensuring fast inference rates on embedded systems. By domain-aware design choices and through comprehensive experimental analyses, we demonstrate that it is possible to achieve state-of-the-art depth estimation performance while ensuring a small computational footprint. Specifically, with 70 % −80 % less network parameters than existing benchmarks, UDepth achieves comparable and often better depth estimation performance. While the full model offers over 66 FPS (13 FPS) inference rates on a single GPU (CPU core), our domain projection for coarse depth prediction runs at 51.5 FPS rates on single-board Jetson TX2s. The inference pipelines are available at https://github.com/uf-robopi/UDepth.

This paper proposes a novel bin picking framework, two-stage grasping, aiming at precise grasping of cluttered small objects. Object density estimation and rough grasping are conducted in the first stage. Fine segmentation, detection, grasping, and pushing are performed in the second stage. A small object bin picking system has been realized to exhibit the concept of two-stage grasping. Experiments have shown the effectiveness of the proposed framework. Unlike traditional bin picking methods focusing on vision-based grasping planning using classic frameworks, the challenges of picking cluttered small objects can be solved by the proposed new framework with simple vision detection and planning.

Modern legged robot morphologies assign most of their actuated degrees of freedom (DoF's) to the limbs and designs continue to converge to twelve DoF quadrupeds with three actuators per leg and a rigid torso often modeled as a Single Rigid Body (SRB). This is in contrast to the animal kingdom, which provides tantalizing hints that core actuation of a jointed torso confers substantial benefit for efficient agility. Unfortunately, the limited specific power of available actuators continues to hamper roboticists' efforts to capitalize on this bio-inspiration. This paper presents the initial steps in a comparative study of the costs and benefits associated with a traditionally neglected torso degree of freedom: a twisting spine. We use trajectory optimization to explore how a one-DoF, axially twisting spine might help or hinder a set of axially-active (twisting) behaviors: trots, sudden turns while bounding, and parkour-style wall jumps. By optimizing for minimum electrical energy or average power, intuitive cost functions for robots, we avoid hand-tuning the behaviors and explore the activation of the spine. Initial evidence suggests that for lower energy behaviors the spine increases the electrical energy required when compared to the rigid torso, but for higher energy runs the spine trends toward having no effect or reducing the electrical work. These results support future, more bio-inspired versions of the spine with inherent stiffness or dampening built into their mechanical design.

Table-top robotic arms for education and research must be low-cost for availability and lightweight and soft for safety. Therefore, as such a robot, this study focuses on designing a plastic table-top cable-driven robotic arm with all motors located at the base link. However, locating all motors at the base link results in a significant distance between a driving motor and driven joint, increases the number of parts for the force transmission, and increases the risk of a cable loosening and coming off of a pulley. To overcome these issues, this study proposed a novel cable-driven robotic arm named Twist Snake. We designed a joint composition of Twist Snake to minimize the number of parts for the force transmission. In addition, it has a compact cable-pretension/termination-mechanism and covering parts to prevent the cable from loosening and coming off of the pulley. The arm comprised 475 mm long moving links with an 802 g. The feasibility of the arm was experimentally demonstrated by contact rich tasks, the insertion of a toy peg into a hole and swiping a whiteboard with a cleaner. The optimization of the proposed design and the development of a learning method for the arm that leverages contact will be investigated in future work.

Autonomous vehicles (AVs) need to reason about the multimodal behavior of neighboring agents while planning their own motion. Many existing trajectory planners seek a single trajectory that performs well under all plausible futures simultaneously, ignoring bi-directional interactions and thus leading to overly conservative plans. Policy planning, whereby the ego agent plans a policy that reacts to the environment's multimodal behavior, is a promising direction as it can account for the action-reaction interactions between the AV and the environment. However, most existing policy planners do not scale to the complexity of real autonomous vehicle applications: they are either not compatible with modern deep learning prediction models, not interpretable, or not able to generate high quality trajectories. To fill this gap, we propose Tree Policy Planning (TPP), a policy planner that is compatible with state-of-the-art deep learning prediction models, generates multistage motion plans, and accounts for the influence of ego agent on the environment behavior. The key idea of TPP is to reduce the continuous optimization problem into a tractable discrete Markov Decision Process (MDP) through the construction of two tree structures: an ego trajectory tree for ego trajectory options, and a scenario tree for multi-modal ego-conditioned environment predictions. We demonstrate the efficacy of TPP in closed-loop simulations based on real-world nuScenes dataset and results show that TPP scales to realistic AV scenarios and significantly outperforms non-policy baselines.

Stereo matching is a common technique used in 3D perception, but transparent objects such as reflective and penetrable glass pose a challenge as their disparities are often estimated inaccurately. In this paper, we propose transparency-aware stereo (TA-Stereo), an effective solution to tackle this issue. TA-Stereo first utilizes a semantic segmentation or salient object detection network to identify transparent objects, and then homogenizes them to enable stereo matching algorithms to handle them as non-transparent objects. To validate the effectiveness of our proposed TA-Stereo strategy, we collect 260 images containing transparent objects from the KITTI Stereo 2012 and 2015 datasets and manually label pixel-level ground truth. We evaluate our strategy with six deep stereo networks and two types of transparent object detection methods. Our experiments demonstrate that TA-Stereo significantly improves the disparity accuracy of transparent objects. Our project webpage can be accessed at mias.group/TA-Stereo.

Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e.g., which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e.g., grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7,600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models. We release code, datasets, and additional results, here: https://github.com/gtatiya/Implicit-Knowledge-Transfer.

Drone-to-drone detection using visual feed has crucial applications, such as detecting drone collisions, detecting drone attacks, or coordinating flight with other drones. However, existing methods are computationally costly, follow non-end-to-end optimization, and have complex multi-stage pipelines, making them less suitable for real-time deployment on edge devices. In this work, we propose a simple yet effective framework, TransVisDrone, that provides an end-to-end solution with higher computational efficiency. We utilize CSPDarkNet-53 network to learn object-related spatial features and VideoSwin model to improve drone detection in challenging scenarios by learning spatio-temporal dependencies of drone motion. Our method achieves state-of-the-art performance on three challenging real-world datasets (Average Precision@0.5IOU): NPS 0.95, FLDrones 0.75, and AOT 0.80, and a higher throughput than previous methods. We also demonstrate its deployment capability on edge devices and its usefulness in detecting drone-collision (encounter). Project: https://tusharsangam.github.io/TransVisDrone-project-page/

Radar semantic segmentation is a challenging task in environmental understanding, due as the radar data is noisy and suffers measurement ambiguities, which could lead to poor feature learning. To better tackle such difficulties, we present a novel and high-performance Transformer-based Radar Semantic Segmentation method, named TransRSS, to effectively and efficiently feature extraction for radar segmentation. Our approach first introduces the transformer into radar semantic segmentation and deeply integrates the merits of the Convolutional Neural Network (CNN) and transformer to extract more discriminative and global-level semantic features. On the one hand, it takes advantage of the CNN with flexible receptive fields to process images thanks to the shift convolution scheme. On the other hand, it takes advantage of the transformer to model long-range dependency with the self-attention mechanism. Meanwhile, we propose a Dual Position Attention module to aggregate rich context interdependencies between the multi-view features, which achieves an implicit mechanism for adaptively feature aggregation. Extensive experiments on the CARRADA dataset and RADIal dataset demonstrate that our TransRSS surpasses the state-of-the-art (SOTA) radar segmentation methods with remarkable margins.

A cuspidal serial robot can travel from one inverse kinematic solution to another without crossing a singularity. Cuspidal robots ask for extra care and caution in trajectory planning, as identifying an aspect related to one unique inverse kinematic solution is not possible. The issues related to motion planning with cuspidal robots are related to the inherent property arising from the geometric design of the robot. The cuspidality property has not been considered in recent industrial 6R robots with a non-spherical wrist. In this work, cuspidality is illustrated with the JACO robot (gen 2, non-spherical wrist), a serial arm by Kinova Robotics which is deployed in various applications and is cuspidal in nature. A nonsingular change of solutions for the robot is provided to highlight the effect of cuspidal robots on the interference with the environment. The pose with multiple inverse kinematic solutions in an aspect is presented. Problems in choosing the initial solution of the path in cuspidal robots, and its consequence, is illustrated with an example path in the workspace of the JACO robot. The paper presents the importance of cuspidality analysis of 6R robots and the implications of neglecting it.

UMA-2 is a wheeled mobile platform equipped with a vacuum adhesion system, eight actuated joints and four passive ones, designed to climb vertical and curved surfaces. The platform can perform maintenance tasks such as corrosion removal and cleaning with grinding while climbing. The quality of the repairing process is largely affected by grinding process parameters including tool forces, toolpath and the robot trajectory accuracy. The current work introduces a trajectory analysis and adaptation model to control the UMA-2 platform to ensure specific surface quality KPIs and incorporating the effects of robot compliancy. The proposed trajectory analysis has been extensively validated through experimental campaigns representative of maintenance in wind power industry.

Falls are the leading cause of fatal and non-fatal injuries, particularly for older persons. Imbalance can result from the body's internal causes (illness), or external causes (active or passive perturbation). Active perturbation results from applying an external force to a person, while passive perturbation results from human motion interacting with a static obstacle. This work proposes a metric that allows for the monitoring of the persons torso and its correlation to active and passive perturbations. We show that large changes in the torso sway can be strongly correlated to active perturbations. We also show that we can reasonably predict the future path and expected change in torso sway by conditioning the expected path and torso sway on the past trajectory, torso motion, and the surrounding scene. This could have direct future applications to fall prevention. Results demonstrate that the torso sway is strongly correlated with perturbations. And our model is able to make use of the visual cues presented in the panorama and condition the prediction accordingly.

The use of bidirectional propellers provides quadrotors with greater maneuverability which is advantageous in constrained environments. This paper addresses the development of a trajectory planning algorithm for quadrotors with bidirectional motors. Previous work has shown that the property of differential flatness can be leveraged for efficient trajectory planning. However, planners that leverage flatness for quadrotors fail at points where the acceleration of the center of mass is equal to gravity, i.e., when the vehicle experiences free fall. The central contribution of this paper is a flatness-based trajectory planning method that allows quadrotors to use bidirectional propellers and pass through the so-called free-fall singularity. We model our system as a differentially flat hybrid system with the aid of coordinate charts derived from the Hopf fibration and develop an algorithm that computes forward and reverse thrusts for each propeller, resulting in smooth trajectories everywhere in SE(3). We demonstrate the planner's versatility by planning knife-edge maneuvers and trajectories passing through the free-fall singularity, while transitioning from forward to reverse thrust.

Trajectory optimization is used to solve various planning tasks. In this paper we present a optimization-based method that solves a planning problem for multiple independent objects manipulated by a spatially continuous physical field. The field is generated and controlled (shaped) in real time by an array of actuators. In the paper we first formulate a trajectory optimization problem and a related initialization scheme, and then we demonstrate the proposed method using an experimental platform for distributed magnetic manipulation. The demonstrated task is that of planar reconfiguration of an ensemble of multiple objects, which significantly benefits from the inherent parallelism of the manipulation enabled by the array of actuators shaping the physical field. We show that the system can rearrange up to eight objects simultaneously while avoiding collisions.

Service robots have attracted extensive attention due to specially designed functions, such as mobile manipulators or robots with extra structures. For robots that have changing shapes, autonomous navigation in the real world presents new challenges. In this paper, we propose a trajectory optimization method for differential-drive mobile robots with controllable changing shapes in dense 3D environments. We model the whole-body trajectory as a polynomial trajectory that satisfies the nonholonomic dynamics of the base and dynamics of the extra joints. These constraints are converted into soft constraints, and an activation function for dense sampling is applied to avoid nonlinear mutations. In addition, we guarantee the safety of full shape by limiting the system's distance from obstacles. To comprehensively simulate a large extent of height and width changes, we designed a novel Shape-Changing Robot with a Differential Base (SCR-DB). Our global trajectory optimization gives a smooth and collision-free trajectory for SCR-DB at a low computational cost. We present vast simulations and real-world experiments to validate our performance, including coupled whole-body and independent differential-driven vehicle motion planning.

When a robot end-effector is attached to the arm via passive joints, undesirable end-effector sway will occur. In a forestry crane, such as the log-loading or harvesting machine, this sway is problematic as it hinders the efficiency and also can harm the machine and environment. Here, we tackle the sway problem of the forestry forwarder by proposing a methodology for generating anti-sway trajectories in fast maneuvers. We employ the dynamic programming algorithm, combined with a suitable linearization approach, the latter identified through a comparative study. The solution has low computational cost and provides excellent performance for residual sway damping. We demonstrate the dynamic programming solution on the virtual model of the forwarder by using a high-fidelity multibody-dynamics simulator to validate its performance. The results show our optimal trajectories can suppress the residual sway effectively to be, on average, less than 10% of the sway when using fifth order polynomial trajectories, in point-to-point maneuvers starting from rest or from initial sway conditions.

Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available atgithub.com/lis-epfl/apg_trajectory_tracking.

Precise pick-and-place is essential in robotic applications. To this end, we define an exact training method and an iterative inference method that improve pick-and-place precision with Transporter Networks [1]. We conduct a large scale experiment on 8 simulated tasks. A systematic analysis shows, that the proposed modifications have a significant positive effect on model performance. Considering picking and placing independently, our methods achieve up to 60% lower rotation and translation errors than baselines. For the whole pick-and-place process we observe 50% lower rotation errors for most tasks with slight improvements in terms of translation errors. Furthermore, we propose architectural changes that retain model performance and reduce computational costs and time. We validate our methods with an interactive teaching procedure on real hardware. Supplementary material is available at: https://gergely-soti.github.io/p3