A Controlled Study on Long Context Extension and Generalization in LLMs

摘要

Achieving robust textual comprehension and in-context learning requires language models capable of interpreting entire document contexts. However, scaling these models directly to long contexts remains technically challenging, prompting a surge of “extension” strategies. To date, rigorous comparisons among these approaches have been complicated by inconsistent base models, training data, and evaluation metrics, limiting our understanding of how long-context performance may differ from standard benchmarks. In this work, we introduce a controlled extension protocol and a standardized evaluation pipeline, enabling an apples-to-apples comparison across diverse long-context methods. Through extensive experiments, we uncover three key insights: (1) perplexity emerges as a helpful (albeit imperfect) indicator for gauging model quality on lengthy-context tasks, (2) approximate attention mechanisms exhibit systematic performance deficits on long-context benchmarks, and (3) exact fine-tuning remains robust within its extension range, although extrapolation beyond that range continues to pose challenges. All codebases, trained models, and checkpoints will be released, fostering transparency and accelerating progress in this critical area of AI research. Our results not only help clarify the current landscape of long-context modeling but also offer guidance for building more capable, context-aware language models.