Look Before You Act: Boosting Pseudo-LiDAR with Online Semantic Embedding

Abstract

Vision-based 3D object detection is a research focus in the field of autonomous driving system. While recently proposed pseudo-LiDAR is a promising solution, its performance is severely restricted by the image-based depth estimator, leading to a considerable performance gap against the LiDAR-based counterparts. In this paper, substantial advances are developed along an orthogonal direction to the previous efforts in the pseudo-LiDAR pipeline. Concretely, we propose a plug- and-play module, called Online Semantic Embedding (OSE), aligning image semantics with the pseudo-LiDAR detection in an end-to-end manner. On the KITTI object detection benchmark, existing stereo-based baselines integrated with our approach show impressive improvements without bells and whistles. Furthermore, we emphasize that OSE works in retrieving the performance under geometric imperfection conditions.