Memory Dial: A Training Framework for Controllable Memorization in Language Models

摘要

Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter, producing a family of models identical in architecture, data, and optimization, but varying in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1) reliably controls memorization, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.