Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention

Abstract

Event cameras provide sequential visual data with spatialsparsity and high temporal resolution, making them attractive for low-latency object detection. Existing asynchronous event-based neural net-works exploit this low-latency advantage by updating predictions eventby event, but still suffer from two bottlenecks: recurrent architecturesare difficult to train efficiently on long sequences, and improving accu-racy often increases per-event computation and latency. Linear atten-tion is appealing because it enables parallel training and recurrent infer-ence. However, its dense state updates make per-event computation scalewith the state size, yielding a poor accuracy-efficiency trade-off for ob-ject detection, where accurate localization requires fine-grained spatialstates. The key challenge is therefore to introduce sparse state activa-tion that exploits the spatial sparsity of events while preserving efficientparallel training. We propose Spatially-Sparse Linear Attention (SSLA),which introduces a mixture-of-spaces state decomposition and a scatter-compute-gather training procedure, enabling state-level sparsity as wellas training parallelism. Building on SSLA, we develop an end-to-endasynchronous linear attention model, SSLA-Det, for low-latency event-based object detection. On Gen1 and N-Caltech101, SSLA-Det achievesstate-of-the-art accuracy among asynchronous methods, reaching 0.375mAP and 0.515 mAP, respectively, while reducing per-event computationby over 20