Attention is Case-Sensitive

Abstract

In human visual perception, uppercase lettering serves as anatural salience cue that captures attention within lowercase text. Inthis paper, we present a systematic empirical characterization study re-vealing that Large Language Models (LLMs) exhibit an analogous prop-erty: letter casing modulates internal attention allocation. Through anal-ysis across 13 models—nine LLMs and four Vision-Language Models(VLMs)—with diverse tokenization schemes, we show that formattingtarget information in alternating or uppercase against a lowercase con-text concentrates attention on those textual spans. In text this e!ectis universal, holding across every evaluated non-reasoning model. Weframe it as a previously under-explored latent property of pretrainedtransformers rather than a prescriptive method. Our investigation re-veals a central attention–performance divergence: while this “casing ef-fect” robustly shifts attention, its impact on downstream accuracy isnon-trivial—increased concentration does not inherently improve taskaccuracy and, in high-entropy contexts like alternating case, can degradeit. We further identify a boundary condition: the deliberative “thinking”phase in reasoning models acts as a semantic bu!er that mitigates ty-pographic sensitivity in text. Extending the study to VLMs, we find thee!ect transfers partially: the same prompt-side casing reorganizes cross-modal attention along two coupled axes—predominantly a macroscopicdisengagement from the image toward the text prompt, and secondar-ily a concentration of the residual visual attention on the target region.By isolating casing as a zero-shot mechanism for attention steering thatrequires no model access or fine-tuning, we provide a new foundationalunderstanding of how pretraining internalizes typographic emphasis.