The Language of Visual Attention: Modeling Scanpaths via Autoregressive Token Prediction

摘要

Understanding human visual attention on a scene over timehas applications in domains such as autonomous driving, interface de-sign, inferring cognitive states. Modeling human visual scanpaths has his-torically relied on specialized architectures with hand-crafted geometricpriors. While these architectures successfully model _x001C_xation sequences,their rigid structural biases restrict easy extendability and _x001D_exible con-ditioning. For instance, integrating task-speci_x001C_c instructions or adaptingto distinct viewer identities traditionally requires custom, disjoint ar-chitectural additions. We address this limitation by framing scanpathprediction purely as a discrete sequence modeling task. By mapping spa-tial coordinates into an autoregressive vocabulary, we leverage the pre-trained representations of Large Vision-Language Models (LVLMs). Thisformulation naturally absorbs diverse factors of variation: simple prompt-ing modi_x001C_cations allow for global conditioning, such as providing vieweridentities to capture personalized biases, or task-speci_x001C_c objectives (likefree-viewing versus visual search). The framework can also easily inte-grate the modeling of per-_x001C_xation attributes, such as individual _x001C_xa-tion durations alongside spatial locations. Crucially, this autoregressivealignment enables the scalable, exact computation of per-_x001C_xation log-likelihoods, directly equivalent to the commonly used Information Gain(IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003,a 46% improvement over matched-backbone DeepGaze III. This advan-tage persists even when baselines use identical high-capacity vision en-coders. Beyond predictive performance, our generative framework servesas a powerful computational tool for direct behavioral interventions, al-lowing for controlled in-silico simulations that would be experimentallydi_x001E_cult or impossible to conduct in vivo. We demonstrate this abilityby performing controlled interventions on the durations of pre-saccadic_x001C_xations, recovering known oculomotor phenomena purely from data.