ICRA 2022 Papers

Dynamic jumping with legged robots poses a challenging problem in planning and control. Formulating the jump optimization to allow fast online execution is difficult; efficiently using this capability to generate long-horizon motion plans further complicates the problem. In this work, we present a hierarchical planning framework to address this problem. We first formulate a real-time tractable trajectory optimization for performing omnidirectional jumping. We then embed the results of this optimization into a low dimensional jump feasibility classifier. This classifier is leveraged to produce geometric motion plans that select dynamically feasible jumps while mitigating the effects of the process noise. We deploy our framework on the Mini Cheetah Vision quadruped, demonstrating the robot's ability to generate and execute reliable, goal-oriented plans that involve forward, lateral, and rotational jumps onto surfaces as tall as the robot's nominal hip height. The ability to plan through omnidirectional jumping greatly expands the robot's mobility relative to planners that restrict jumping to the sagittal or frontal planes.

Panoptic segmentation of point clouds is one of the key challenges of 3D scene understanding, requiring the simultaneous prediction of semantics and object instances. Tasks like autonomous driving strongly depend on these information to get a holistic understanding of their 3D environment. This work presents a novel proposal free framework for lidar-based panoptic segmentation, which exploits three different point cloud representations, leveraging their strengths and compensating their weaknesses. The efficient projection-based range view and bird's eye view are combined and further extended by a point-based network with a novel attention-based neighborhood aggregation for improved semantic features. Cluster-based object recognition in bird's eye view enables an efficient and high-quality instance segmentation. Semantic and instance segmentation are fused and further refined by a novel instance classification for the final panoptic segmentation. The results on two challenging large-scale datasets, nuScenes and SemanticKITTI, show the success of the proposed framework, which outperforms all existing approaches on nuScenes and achieves state-of-the-art results on SemanticKITTI.

Modeling stochastic traffic dynamics is critical to developing self-driving cars. Because it is difficult to develop first principle models of cars driven by humans, there is great potential for using data driven approaches in developing traffic dynamical models. While there is extensive literature on this subject, previous works mainly address the prediction accuracy of data-driven models. Moreover, it is often difficult to apply these models to common planning frameworks since they fail to meet the assumptions therein. In this work, we propose a new stochastic traffic model, Recurrent Traffic Graph Neural Network (RTGNN), by enforcing additional structures on the model so that the proposed model can be seamlessly integrated with existing motion planning algorithms. RTGNN is a Markovian model and is able to infer future traffic states conditioned on the motion of the ego vehicle. Specifically, RTGNN uses a definition of the traffic state that includes the state of all players in a local region and is therefore able to make joint predictions for all agents of interest. Meanwhile, we explicitly model the hidden states of agents, “intentions,” as part of the traffic state to reflect the inherent partial observability of traffic dynamics. The above mentioned properties are critical for integrating RTGNN with motion planning algorithms coupling prediction and decision making. Despite the additional structures, we show that RTGNN is able to achieve state-of-the-art accuracy through comparisons with other similar works.

ROS is popular in robotic-software development, and thus detecting bugs in ROS programs is important for modern robots. Fuzzing is a promising technique of runtime testing. But existing fuzzing approaches are limited in testing ROS programs, due to neglecting ROS properties, such as multi-dimensional inputs, temporal features of inputs and the distributed node model. In this paper, we develop a new fuzzing framework named ROZZ, to effectively test ROS programs and detect bugs based on ROS properties. ROZZ has three key techniques: (1) a multi-dimensional generation method to generate test cases of ROS programs from multiple dimensions, including user data, configuration parameters and sensor messages; (2) a distributed branch coverage to describe the overall code coverage of multiple ROS nodes in the robot task; (3) a temporal mutation strategy to generate test cases with temporal information. We evaluate ROZZ on 10 common robotic programs in ROS2, and it finds 43 real bugs. 20 of these bugs have been confirmed and fixed by related ROS developers. We compare ROZZ to existing approaches for testing robotic programs, and ROZZ finds more bugs with higher code coverage.

We study a semantic SLAM problem where a robot is tasked with autonomous weeding under the corn canopy. The goal is to detect corn stalks and localize them in a global coordinate frame. This is a challenging scenario for existing algorithms because there is very little space between the camera and the plants, and the camera motion is primarily restricted to be along the row. To overcome these challenges, we present a multi-camera system where a side camera (facing the plants) is used for detection, whereas front and back cameras are used for motion estimation. Next, we show how semantic features in the environment (corn stalks, ground, and crop planes) can be used to develop a robust semantic SLAM solution and present results from field trials performed throughout the growing season across various cornfields.

Developing reusable software for mobile robots is still challenging. Even more so for swarm robots, despite the desired simplicity of the robot controllers. Prototyping and experimenting are difficult due to the multi-robot setting and often require robot-robot communication. Also, the diversity of swarm robot hardware platforms increases the need for hardware-independent software concepts. The main advantages of the commonly used robot software architecture ROS 2 are modularity and platform independence. We propose a new ROS 2 package, ROS2sWARM, for applications of swarm robotics that provides a library of ready-to-use swarm behavioral primitives. We show the successful application of our approach on three different platforms, the TurtleBot3 Burger, the TurtleBot3 Waffle Pi, and the Jackal UGV, and with a set of different behavioral primitives, such as aggregation, dispersion, and collective decision-making. The proposed approach is easy to maintain, extendable, and has good potential for simplifying swarm robotics experiments in future applications.

In a multirobot system, a number of cyber-physical attacks (e.g., communication hijack, observation per-turbations) can challenge the robustness of agents. This robust-ness issue worsens in multiagent reinforcement learning because there exists the non-stationarity of the environment caused by simultaneously learning agents whose changing policies affect the transition and reward functions. In this paper, we propose a minimax MARL approach to infer the worst-case policy update of other agents. As the minimax formulation is computationally intractable to solve, we apply the convex relaxation of neural networks to solve the inner minimization problem. Such convex relaxation enables robustness in interacting with peer agents that may have significantly different behaviors and also achieves a certified bound of the original optimization problem. We eval-uate our approach on multiple mixed cooperative-competitive tasks and show that our method outperforms the previous state of the art approaches on this topic.

The proposed RMS-FlowNet is a novel end-to-end learning-based architecture for accurate and efficient scene flow estimation which can operate on point clouds of high density. For hierarchical scene flow estimation, the existing methods depend on either expensive Farthest-Point-Sampling (FPS) or structure-based scaling which decrease their ability to handle a large number of points. Unlike these methods, we base our fully supervised architecture on Random-Sampling (RS) for multiscale scene flow prediction. To this end, we propose a novel flow embedding design which can predict more robust scene flow in conjunction with RS. Exhibiting high accuracy, our RMS-FlowNet provides a faster prediction than state-of-the-art methods and works efficiently on consecutive dense point clouds of more than 250K points at once. Our comprehensive experiments verify the accuracy of RMS-FlowNet on the established FlyingThings3D data set with different point cloud densities and validate our design choices. Additionally, we show that our model presents a competitive ability to generalize towards the real-world scenes of KITTI data set without fine-tuning.

Variable impedance control in operation-space is a promising approach to learning contact-rich manipulation behaviors. One of the main challenges with this approach is producing a manipulation behavior that ensures the safety of the arm and the environment. Such behavior is typically implemented via a reward function that penalizes unsafe actions (e.g. obstacle collision, joint limit extension), but that approach is not always effective and does not result in behaviors that can be reused in slightly different environments. We show how to combine Riemannian Motion Policies, a class of policies that dynamically generate motion in the presence of safety and collision constraints, with variable impedance operation-space control to learn safer contact-rich manipulation behaviors.

Wireless tags are increasingly used to track and identify common items of interest such as retail goods, food, medicine, clothing, books, documents, keys, equipment, and more. At the same time, there is a need for labelled visual data featuring such items for the purpose of training object detection and recognition models for robots operating in homes, warehouses, stores, libraries, pharmacies, and so on. In this paper, we ask: can we leverage the tracking and identification capabilities of such tags as a basis for a large-scale automatic image annotation system for robotic perception tasks? We present RF-Annotate, a pipeline for autonomous pixel-wise image annotation which enables robots to collect labelled visual data of objects of interest as they encounter them within their environment. Our pipeline uses unmodified commodity RFID readers and RGB-D cameras, and exploits arbitrary small-scale motions afforded by mobile robotic platforms to spatially map RFIDs to corresponding objects in the scene. Our only assumption is that the objects of interest within the environment are pre-tagged with inexpensive battery-free RFIDs costing 3–15 cents each. We demonstrate the efficacy of our pipeline on several RGB-D sequences of tabletop scenes featuring common objects in a variety of indoor environments.

Behavior trees (BTs) are hierarchical agent control architectures popular for robot task-level planning that can be autonomously learned from robot demonstrations via decision tree (DT) intermediaries, making them accessible to non-expert users. Conversion algorithms from DTs to BTs, such as the BT-Espresso algorithm, focus on replicating DT logic in a BT format but do not exploit the strengths of the BT architecture. We introduce the Representation Exploitation of BT-Espresso (RE:BT-Espresso) algorithm, which builds on BT-Espresso and improves the learned BT's interpretability and expressivity. RE:BT-Espresso improves interpretability by removing logical redundancies in the generated BTs and improves expressivity by exploiting desired BT structures, such as adding Inverter nodes, Repeater sequences, and Parallel Selector Action nodes that gives the user a choice of actions for state spaces that did not resolve to a concise action in the DT. The RE:BT-Espresso algorithm was evaluated against BT-Espresso using demonstration data synthesized by BTs. When compared to the synthesized BTs using graph edit distance (GED), RE:BT-Espresso outscored BT-Espresso on 54 subtrees, tied on 178, and lost on 2. Further, the proposed reduction strategies reduced the number of nodes in a generated tree by a median of 7.82%. The results validate improved interpretability and expressivity of learned RE:BT-Espresso task-level BT policies from robot demonstration.

Present-day Deep Reinforcement Learning (RL) systems show great promise towards building intelligent agents surpassing human-level performance. However, the computational complexity associated with the underlying deep neural networks (DNNs) leads to power-hungry implementations. This makes deep RL systems unsuitable for deployment on resource-constrained edge devices. To address this challenge, we propose a reconfigurable architecture with preemptive exits for effi-cient deep RL (RAPID-RL). RAPID-RL enables conditional activation of DNN layers based on the difficulty level of inputs. This allows to dynamically adjust the compute effort during inference while maintaining competitive performance. We achieve this by augmenting a deep Q-network (DQN) with side-branches capable of generating intermediate predictions along with an associated confidence score. We also propose a novel training methodology for learning the actions and branch confidence scores in a dynamic RL setting. Our experiments evaluate the proposed framework for Atari 2600 gaming tasks and a realistic Drone navigation task on an open-source drone simulator (PEDRA). We show that RAPID-RL incurs 0.34 × (0.25 ×) number of operations (OPS) while maintaining performance above 0.88 × (0.91 ×) on Atari (Drone navigation) tasks, compared to a baseline-DQN without any side-branches. The reduction in OPS leads to fast and efficient inference, proving to be highly beneficial for the resource-constrained edge where making quick decisions with minimal compute is essential.

In this paper, we propose a novel LiDAR-Inertial-Visual sensor fusion framework termed R3LIVE, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE consists of two subsystems, a LiDAR-Inertial odometry (LIO) and a Visual-Inertial odometry (VIO). The LIO subsystem (FAST-LIO) utilizes the measurements from LiDAR and inertial sensors and builds the geometric structure (i.e., the positions of 3D points) of the map. The VIO subsystem uses the data of Visual-Inertial sensors and renders the map's texture (i.e., the color of 3D points). More specifically, the VIO subsystem fuses the visual data directly and effectively by minimizing the frame-to-map photometric error. The proposed system R3LIVE is developed based on our previous work R2LIVE, with a completely different VIO architecture design. The overall system is able to reconstruct the precise, dense, 3D, RGB-colored maps of the surrounding environment in real-time (see our attached video 11https://youtu.be/j5fT8NE5fdg). Our experiments show that the resultant system achieves higher robustness and accuracy in state estimation than its current counterparts. To share our findings and make contributions to the community, we open source R3LIVE on our Github 22https://github.com/hku-mars/r31ive

Robotic engineers face major challenges to solve the complex actuation needs of Human-Robot Collaboration with existing act robotic gearboxes. Available technologies comprise high-ratio Planetary Gearheads, Cycloid Drives and Harmonic Drives, inherited from conventional industrial robotics. Alternative approaches include Direct-Drive and Quasi Direct-Drive actuation strategies, which propose to cancel or substantially reduce gear ratio, in order to minimize reflected inertia and attain enough backdrivability for collaborative tasks. This paper presents the proof-of-concept validation of a novel high-ratio, Wolfrom-based, gearbox technology that follows a different approach to attain the same objective. Testing five different gearbox prototypes, we confirm the ability of the R2poweR technology to improve efficiency and backdrivability while retaining the weight and control advantages derived from the use of high reduction ratios. The result is a highly efficient, backdrivable, high-ratio gearbox with exciting Huma-Robot Collaboration potential.

In autonomous vehicles or robots, point clouds from LiDAR can provide accurate depth information of objects compared with 2D images, but they also suffer a large volume of data, which is inconvenient for data storage or transmission. In this paper, we propose a Range image-based Point Cloud Compression method, R-PCC, which can reconstruct the point cloud with uniform or non-uniform accuracy loss. We segment the original large-scale point cloud into small and compact regions for spatial redundancy and salient region classification. Our range image-based method can keep and align all points from the original point cloud in the reconstructed point cloud, and the setting of the quantization module restricts the maximum reconstruction error. In the experiments, we prove that our easier FPS-based segmentation method can achieve better performance than instance-based segmentation methods such as DBSCAN, and our non-uniform compression framework shows a great improvement on the downstream tasks compared with the state-of-the-art large-scale point cloud compression methods. Our real-time method can achieve 40 × compression ratio without affecting downstream tasks, to act as a baseline for range image-based point cloud compression. The code is available on https://github.com/StevenWang30/R-PCC.git.

This paper presents QuadRunner, a transformable quasi-wheel legged robot that achieves both quadruped locomotion and wheel locomotion by exploiting a novel semicircular leg-wheel design with a Trotting Wheel gait. We built upon the Stanford Doggo open architecture platform and integrated it with a transformable leg-wheel design to enhance its locomotion capabilities. On its gait control, improvements were made to its trot gait kinematics with end-effector considerations as well as the design of a new trotting wheel gait. Our proposed locomotion strategy found that the robot's legged locomotion improves by an average speed of 10%. In addition, wheel to leg transition takes around 300 ms, and the speed of its wheel locomotion can reach more than five times its body-length/s (2.2 m/s) on flat terrain. Lastly, detailed experiments are conducted to observe the wheel-leg transition performance and gait verification under the absence of foot contact sensors.

PyROBOCOP is a Python-based package for control, optimization and estimation of robotic systems described by nonlinear Differential Algebraic Equations (DAEs). In particular, the package can handle systems with contacts that are described by complementarity constraints and provides a general framework for specifying obstacle avoidance constraints. The package performs direct transcription of the DAEs into a set of nonlinear equations by performing orthogonal collocation on finite elements. PyROBOCOP provides automatic reformulation of the complementarity constraints that are tractable to NLP solvers to perform optimization of robotic systems. The package is interfaced with ADOL-C [1] for obtaining sparse derivatives by automatic differentiation and IPOPT [2] for performing optimization. We evaluate PyROBOCOP on several manipulation problems for control and estimation.

In this paper, we study the problem of autonomously seating a teddy bear on a previously unseen chair. To achieve this goal, we present a novel method for robots to imagine the sitting pose of the bear by physically simulating a virtual humanoid agent sitting on the chair. We also develop a robotic system which leverages motion planning to plan SE(2) motions for a humanoid robot to walk to the chair and whole-body motions to put the bear on it. Furthermore, to cope with cases where the chair is not in an accessible pose for placing the bear, a human assistance module is introduced for a human to follow language instructions given by the robot to rotate the chair and help make the chair accessible. We implement our method with a robot arm and a humanoid robot. We calibrate the proposed system with 3 chairs and test on 12 previously unseen chairs in both accessible and inaccessible poses extensively. Results show that our method enables the robot to autonomously seat the teddy bear on the 12 previously unseen chairs with a very high success rate. The human assistance module is also shown to be very effective in changing the accessibility of the chair. Video demos and more details are available at https://chirikjianlab.github.io/putbearonchair/.

Efficient robotic manipulation of objects for sorting and searching often rely upon how well the objects are perceived and the available grasp poses. The challenge arises when the objects are irregular, have similar visual features (e.g., textureless objects) and the scene is densely cluttered. In such cases, non-prehensile manipulation (e.g., pushing) can facilitate grasping or searching by improving object perception and singulating the objects from the clutter via physical interaction. The current robotics literature in interactive segmentation focuses solely on isolated cases, where the central aim is on searching or singulating a single target object, or segmenting sparsely cluttered scenes, mainly through matching visual futures in successive scenes before and after the robotic interaction. On the other hand, in this paper, we introduce the first interactive segmentation model in the literature that can autonomously enhance the instance segmentation of such challenging scenes as a whole via optimising a Q-value function that predicts appropriate pushing actions for singulation. We achieved this by training a deep reinforcement learning model with reward signals generated by a Mask-RCNN trained solely on depth images. We evaluated our model in experiments by comparing its success on segmentation quality with a heuristic baseline, as well as the state-of-the-art Visual Pushing and Grasping (VPG) model [1]. Our model significantly outperformed both baselines in all benchmark scenarios. Furthermore, decreasing the segmentation error inherently enabled the autonomous singulation of the scene as a whole. Our evaluation experiments also serve as a benchmark for interactive segmentation research.