NPUGen: NPU as KV Generator for Stable Long-Context Mobile LLM Inference

Abstract

As on-device large language model (LLM) applications like mobile agents become increasingly popular, long-context inference has become a key challenge. The key-value (KV) cache is the foundation of efficient autoregressive decoding, but its memory footprint grows rapidly with context length, often exceeding the memory limits of mobile devices. Existing KV cache management methods, such as compression and storage offloading, often lead to performance degradation or unpredictable stall latency. We find that in the prevalent CPU/GPU-decoding mobile pipelines, the NPU remains largely idle during decoding and can recompute missing KV cache more efficiently than storage offloading. We therefore propose NPUGen, whose KV recomputation engine generates missing KV cache on the idle NPU via a prefill-like forward pass, avoiding unstable storage I/O, CPU intervention, and unnecessary data movement. NPUGen further introduces a hidden-state anchor mechanism that eliminates redundant prefix recomputation, and turns KV recomputation into a schedulable background service to cope with resource contention and guarantee foreground QoS. Experiments on smartphones with Qualcomm SoCs across multiple long-context tasks show that NPUGen significantly reduces decoding latency and fluctuations, providing fast and stable long-context inference on resource-constrained mobile devices.