ExoMem: OS-Governed Memory for Local Large-LLM Inference on Mobile Edge Devices

摘要

High-quality large language model inference is increasingly used on mobile edge, from satellites to smart homes, where limited connectivity or privacy requirements motivate local model inference. In many such settings, token generation is inherently compute-limited and added latency may be acceptable, but output quality, memory feasibility and crash-free execution are essential. However, on today's mobile edge with unified-memory architectures, running a large model near the device limit can trigger non-deterministic out-of-memory failures. This is because deep learning frameworks retain freed buffers inside caching pools, thus continuing to occupy shared DRAM, which the OS cannot reclaim. During inference, fragmentation and conservative reuse can produce growing memory usage, eventually exceeding the memory availability. In this paper, we present ExoMem, an OS-governed middleware that restores OS authority over physical pages without sacrificing framework compatibility. ExoMem keeps model topology and operators in the framework, while anchoring weights in OS-managed mappings, and introduces a dormant state, whose virtual addresses remain stable for correctness but whose physical pages are immediately reclaimable when needed, leveraging standard OS page caching as an implicit multi-tier hierarchy. Implemented as a plug-and-play extension, ExoMem enables stable local inference for substantially larger LLMs on mobile edge, including 72B-class models on 64GB NVIDIA Jetson devices, while preserving headroom for concurrent workloads.