Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline

Abstract

Sign Language Translation (SLT) is a core task in the field of AI-assisted disability. Traditional SLT methods are typically based on visible lightvideos, which are easily affected by factors such as lighting variations, rapidhand movements, and privacy concerns. This paper proposes the use of bio-inspired event cameras to alleviate the aforementioned issues. Specifically, weintroduce a new high-definition event-based sign language dataset, termed Event-CSL, which effectively addresses the data scarcity in this research area. Thedataset comprises 14,827 videos, 6,097 glosses, and 2,544 Chinese words in thetext vocabulary. These samples are collected across diverse indoor and outdoorscenes, covering multiple viewpoints, lighting conditions, and camera motions.We have also benchmarked existing mainstream SLT methods on this dataset tofacilitate fair comparisons in future research. Furthermore, we propose a novelevent-based, gloss-free sign language translation framework, termed EvSLT. Theframework first segments continuous video features into clips and employs aMamba-based memory aggregation module to compress and aggregate spatialdetail features at the clip level. Subsequently, these spatial features, along withtemporal representations obtained from temporal convolution, are then fused bya graph-guided spatiotemporal fusion module. Extensive experiments on Event-CSL, as well as other publicly available datasets, demonstrate the superior per-formance of our method. The dataset and source code are publicly available athttps://github.com/Event-AHU/OpenESL/tree/main/EvSLT.