ICRA 2022 Papers

Neural network used for the LiDAR semantic segmentation task needs the point-wise labeled point clouds for training, which is more expensive than bounding box annotations. Enhancing the diversity of training data through object insertion is an effective method to reduce labeling costs. The existing object insertion methods are mainly divided into two categories. First is “copy” the clusters from a LiDAR frame and “paste” it to other frames or positions. Second is inserting CAD models into the background then using LiDAR simulator to generate laser points of the inserted CAD models. “Copy-paste” method cannot generate realistic scanning lines and shadows, and the CAD models, especially the CAD models of flexible objects, are hard to obtain. We propose an object insertion based data augmentation method which can increase the performance of the semantic segmentation network remarkably. First, an object library is created by using the labeled LiDAR point clouds. Then, these objects are inserted into the LiDAR point clouds dynamically during the training. Finally, the realistic scanning lines and shadows are simulated according to the real LiDAR parameters. The experimental results show that the proposed augmentation method can increase the performance of different semantic segmentation frameworks remarkably.

Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through intelligent choice of action space. Operational Space Control (OSC) has been used as an effective task-space controller for manipulation. Nonetheless, its strength depends on the underlying modeling fidelity, and is prone to failure when there are modeling errors. In this work, we propose OSC for Adaptation and Robustness (OSCAR), a data-driven variant of OSC that compensates for modeling errors by inferring relevant dynamics parameters from online trajectories. OSCAR decomposes dynamics learning into task-agnostic and task-specific phases, decoupling the dynamics dependencies of the robot and the extrinsics due to its environment. This structure enables robust zero-shot performance under out-of-distribution and rapid adaptation to significant domain shifts through additional finetuning. We evaluate our method on a variety of simulated manipulation problems, and find substantial improvements over an array of controller baselines. For more results and information, please visit https://cremebrule.github.io/oscar-web/.

Freespace detection is an essential component of autonomous driving technology and plays an important role in trajectory planning. In the last decade, deep learning based freespace detection methods have been proved feasible. However, these efforts were focused on urban road environments and few deep learning based methods were specifically designed for off-road freespace detection due to the lack of off-road dataset and benchmark. In this paper, we present the ORFD dataset, which, to our knowledge, is the first off-road freespace detection dataset. The dataset was collected in different scenes (woodland, farmland, grassland and countryside), different weather conditions (sunny, rainy, foggy and snowy) and different light conditions (bright light, daylight, twilight, darkness), which totally contains 12,198 LiDAR point cloud and RGB image pairs with the traversable area, non-traversable area and unreachable area annotated in detail. We propose a novel network named OFF-Net, which unifies Transformer architecture to aggregate local and global information, to meet the requirement of large receptive fields for freespace detection task. We also propose the cross-attention to dynamically fuse LiDAR and RGB image information for accurate off-road freespace detection. Dataset and code are publicly available at https://github.com/chaytonmin/OFF-Net.

Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle perception. It contains over 70 interesting scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes, collected from 8 towns in CARLA and a digital town of Culver City, Los Angeles. We then construct a comprehensive benchmark with a total of 16 implemented models to evaluate several information fusion strategies (i.e. early, late, and intermediate fusion) with state-of-the-art LiDAR detection algorithms. Moreover, we propose a new Attentive Intermediate Fusion pipeline to aggregate information from multiple connected vehicles. Our experiments show that the proposed pipeline can be easily integrated with existing 3D LiDAR detectors and achieve outstanding performance even with large compression rates. To encourage more researchers to investigate Vehicle-to-Vehicle perception, we will release the dataset, benchmark methods, and all related codes in https://mobility-lab.seas.ucla.edu/opv2v/.

Inverse Reinforcement Learning (IRL) is attractive in scenarios where reward engineering can be tedious. However, prior IRL algorithms use on-policy transitions, which require intensive sampling from the current policy for stable and optimal performance. This limits IRL applications in the real world, where environment interactions can become highly expensive. To tackle this problem, we present Off-Policy Inverse Reinforcement Learning (OPIRL), which (1) adopts off-policy data distribution instead of on-policy and enables significant reduction of the number of interactions with the environment, (2) learns a reward function that is transferable with high generalization capabilities on changing dynamics, and (3) leverages mode-covering behavior for faster convergence. We demonstrate that our method is considerably more sample efficient and generalizes to novel environments through the experiments. Our method achieves better or comparable results on policy performance baselines with significantly fewer interactions. Furthermore, we empirically show that the recovered reward function generalizes to different tasks where prior arts are prone to fail.

This paper tackles the problem of nonprehensile object transportation through a legged manipulator. A whole-body control architecture is devised to prevent sliding of the object placed on the tray at the manipulator's end-effector and retain the legged robot balance during walking. The controller solves a quadratic optimization problem to realize the sought transportation task while maintaining the contact forces between the tray and the object and between the legs and the ground within their respective friction cones, also considering limits on the input torques. An extensive simulation campaign confirmed the feasibility of the approach and evaluated the control performance through a thorough statistical analysis conducted varying mass, friction, and the dimension of the transported object.

Jet-powered vertical takeoff and landing (VTOL) drones require precise thrust estimation to ensure adequate stability margins and robust maneuvering. Small-scale turbojets have become good candidates for powering heavy aerial drones. However, due to limited instrumentation available in these turbojets, estimating the precise thrust using classical techniques is not straightforward. In this paper, we present a methodology to accurately estimate the online thrust for the small-scale turbojets used on the iRonCub - an aerial humanoid robot. We use a grey-box method to capture the turbojet system dynamics with a nonlinear state-space model based on the data acquired from a custom engine test bench. This model is then used to design an extended Kalman filter that estimates the turbojet thrust only from the angular speed measurements. We exploited the parameter estimation algorithm to ensure that the EKF gives smooth and accurate estimates even at engine failures. The designed EKF was validated on the test bench where the mean absolute error in estimated thrust was found to be within 2% of rated peak thrust.

Contact adaptation is an essential capability when manipulating objects. Two key contact modes of non-prehensile manipulation are sticking and sliding. This paper presents a Trajectory Optimization (TO) method formulated as a Mathematical Program with Complementarity Constraints (MPCC), which is able to switch between these two modes. We show that this formulation can be applicable to both planning and Model Predictive Control (MPC) for planar manipulation tasks. We numerically compare: (i) our planner against a mixed integer alternative, showing that the MPCC planner converges faster, scales better with respect to the time horizon (TH), and can handle environments with obstacles; (ii) our controller against a state-of-the-art mixed integer approach, showing that the MPCC controller achieves improved tracking and more consistent computation times. Additionally, we experimentally validate both our planner and controller with the KUKA LWR robot on a range of planar manipulation tasks. See our accompanying video here: https://youtu.be/EkU6YHMhjto.

During postharvest storage, fruit firmness usually decreases due to respiration and bruise, the former of which indicates the fruit ripeness while the latter negatively influence consumers' taste preference. This paper presents a portable and low-cost device using vision-based tactile information to evaluate fruit firmness in a non-destructive manner. The device consists of a camera, LED lights, and a soft sensing layer with small bumps to capture detailed tactile information of the fruit. Two working modes are designed and a CNN-LSTM architecture is developed to relate the tactile information to fruit overall firmness or detect local firmness distortion. According to the experimental results, an R2 up to 92.9% was achieved for the evaluation of the overall firmness of Cuixiang kiwifruit, and accuracy of 98.0% was obtained for the detection of local firmness distortion of Fuji apples. These results demonstrate the efficacy of the proposed solution to evaluate fruit firmness featuring high precision, and its non-destructive and potable nature is also anticipated to be favorable by the fresh fruit market.

This paper presents a novel iterative closest points (ICP) variant, non-penetration iterative closest points (NPICP), which prevents interpenetration in 6DOF pose optimization and/or joint optimization of multiple object poses. This capability is particularly advantageous in cluttered scenarios, where there are many interactions between objects that constrain the space of valid poses. We use a semi-infinite programming approach to handle non-penetration constraints between complex, non-convex 3D geometries. NPICP is applied to a common use case for ICP as a post-processing method to improve the pose estimation accuracy of a rough guess. The results show that NPICP outperforms ICP, assists in outlier detection, and also outperforms the best result on the IC-BIN dataset in the Benchmark for 6D Object Pose Estimation.

We address the risk bounded trajectory optimization problem of stochastic nonlinear robotic systems. More precisely, we consider the motion planning problem in which the robot has stochastic nonlinear dynamics and uncertain initial locations, and the environment contains multiple dynamic uncertain obstacles with arbitrary probabilistic distributions. The goal is to plan a sequence of control inputs for the robot to navigate to the target while bounding the probability of colliding with obstacles. Existing approaches to address risk bounded trajectory optimization problems are limited to particular classes of models and uncertainties such as Gaussian linear problems. In this paper, we deal with stochastic nonlinear models, nonlinear safety constraints, and arbitrary probabilistic uncertainties, the most general setting ever considered. To address the risk bounded trajectory optimization problem, we first formulate the problem as an optimization problem with stochastic dynamics equations and chance constraints. We then convert probabilistic constraints and stochastic dynamics constraints on random variables into a set of deterministic constraints on the moments of state probability distributions. Finally, we solve the resulting deterministic optimization prob-lem using nonlinear optimization solvers and get a sequence of control inputs. To our best knowledge, it is the first time that the motion planning problem to such a general extent is considered and solved. To illustrate the performance of the proposed method, we provide several robotics examples.

Depth acquisition with the active stereo camera is a challenging task for highly reflective objects. When setup permits, multi-view fusion can provide increased levels of depth completion. However, due to the slow acquisition speed of high-end active stereo cameras, collecting a large number of viewpoints for a single scene is generally not practical. In this work, we propose a next-best-view framework to strategically select camera viewpoints for completing depth data on reflective objects. In particular, we explicitly model the specular reflection of reflective surfaces based on the Phong reflection model and a photometric response function. Given the object CAD model and grayscale image, we employ an RGB-based pose estimator to obtain current pose predictions from the existing data, which is used to form predicted surface normal and depth hypotheses, and allows us to then assess the information gain from a subsequent frame for any candidate viewpoint. Using this formulation, we implement an active perception pipeline which is evaluated on a challenging real-world dataset. The evaluation results demonstrate that our active depth acquisition method outperforms two strong baselines for both depth completion and object pose estimation performance.

Quadruped locomotion is rapidly maturing to a degree where robots now routinely traverse a variety of unstructured terrains. However, while gaits can be varied typically by selecting from a range of pre-computed styles, current planners are unable to vary key gait parameters continuously while the robot is in motion. The synthesis, on-the-fly, of gaits with unexpected operational characteristics or even the blending of dynamic manoeuvres lies beyond the capabilities of the current state-of-the-art. In this work we address this limitation by learning a latent space capturing the key stance phases of a particular gait, via a generative model trained on a single trot style. This encourages disentanglement such that application of a drive signal to a single dimension of the latent state induces holistic plans synthesising a continuous variety of trot styles. In fact properties of this drive signal map directly to gait parameters such as cadence, footstep height and full stance duration. The use of a generative model facilitates the detection and mitigation of disturbances to provide a versatile and robust planning framework. We evaluate our approach on a real ANYmal quadruped robot and demonstrate that our method achieves a continuous blend of dynamic trot styles whilst being robust and reactive to external perturbations.

Collaborative robots can help industry workers to improve their ergonomics. They can propose a safe and ergonomic posture to the workers to reduce the risk of musculoskeletal disorders. Proposing an ergonomic stance needs postural evaluation and optimization. To optimize the workers' posture, we need to run the optimization on a cost function representing the ergonomic status. The tabular ergonomic assessment methods are the most common methods used by ergonomists, but they are linear stepwise functions that are not differentiable and not suitable for optimization purposes. We propose NeuroErgo, a deep neural network model that can approximate the tabular ergonomic assessment methods more precisely than existing methods. By solving the task constraints optimization problem for any task in industry and NeuroErgo as posture cost function, a safe and ergonomic posture can be derived and recommended to the workers while accomplishing their job.

Neural Style Transfer (NST) refers to a class of algorithms able to manipulate an element, most often images, to adopt the appearance or style of another one. Each element is defined as a combination of Content and Style: the Content can be conceptually defined as the “what” and the Style as the “how” of said element. In this context, we propose a custom NST framework for transferring a set of styles to the motion of a robotic manipulator, e.g., the same robotic task can be carried out in an “angry”, “happy”, “calm”, or “sad” way. An autoencoder architecture extracts and defines the Content and the Style of the target robot motions. A Twin Delayed Deep Deterministic Policy Gradient (TD3) network generates the robot control policy using the loss defined by the autoencoder. The proposed Neural Policy Style Transfer TD3 (NPST33) alters the robot motion by introducing the trained style. Such an approach can be implemented either offline, for carrying out autonomous robot motions in dynamic environments, or online, for adapting at runtime the style of a teleoperated robot. The considered styles can be learned online from human demonstrations. We carried out an evaluation with human subjects enrolling 73 volunteers, asking them to recognize the style behind some representative robotic motions. Results show a good recognition rate, proving that it is possible to convey different styles to a robot using this approach.

We present a novel framework of motion tracking from event data using implicit expression. Our framework uses pre-trained event generation MLP called the implicit event generator (IEG) and carries out motion tracking by updating its state (position and velocity) based on the difference between the observed event and generated event from the current state estimation. The difference is computed implicitly by the IEG. Unlike the conventional explicit approach, which requires dense computation to evaluate the difference, our implicit approach realizes the update of the efficient state directly from sparse event data. Our sparse algorithm is especially suitable for mobile robotics applications in which computational resources and battery life are limited. To verify the effectiveness of our method on real-world data, we applied it to the AR marker tracking application. We have confirmed that our framework works well in real-world environments in the presence of noise and background clutter.

We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (∼5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/

This paper addresses the generation mechanism and avoidance method of negative stiffness during in-Hand manipulation with underactuated compliant hands. Firstly, a planar hand with two three-jointed fingers manipulating a rectangular is set, and a quasi-static underactuated operation model is established. Secondly, based on this simulation model, we investigated the stiffness evolution during in-hand manipulation, and analyze the influence factors of system stiffness. Finally, a stiffness regulation method is developed to avoid negative stiffness during in-hand manipulation. The method is validated by simulation. The research results are beneficial to improve the performance of underactuated in-hand manipulation.

Collision avoidance algorithms are of central interest to many drone applications. In particular, decentralized approaches may be the key to enabling robust drone swarm solutions in cases where centralized communication becomes computationally prohibitive. In this work, we draw biological inspiration from flocks of starlings (Sturnus vulgaris) and apply the insight to end-to-end learned decentralized collision avoidance. More specifically, we propose a new, scalable observation model following a biomimetic nearest-neighbor information constraint that leads to fast learning and good collision avoidance behavior. By proposing a general reinforcement learning approach, we obtain an end-to-end learning-based approach to integrating collision avoidance with arbitrary tasks such as package collection and formation change. To validate the generality of this approach, we successfully apply our methodology through motion models of medium complexity, modeling momentum and nonetheless allowing direct application to real world quadrotors in conjunction with a standard PID controller. In contrast to prior works, we find that in our sufficiently rich motion model, nearest-neighbor information is indeed enough to learn effective collision avoidance behavior. Our learned policies are tested in simulation and subsequently transferred to real-world drones to validate their real-world applicability.