CFMoE: Cache-Friendly Mixture of Experts Model Serving System on Edge Devices

摘要

Deploying large language models (LLMs) on edge devices has garnered significant attention due to their inherent privacy and security benefits. However, the enormous parameter scale of modern LLMs poses fundamental challenges for resource-constrained edge devices. Mixture-of-Experts (MoE) models emerge as a promising solution by dynamically activating only a subset of experts during inference, and thus enhance computational efficiency. Nevertheless, the runtime uncertainty in expert activation still leads to high memory demands during inference. To address this, prior works concentrate on either compressing or offloading experts, all striving around the tradeoffs among the model precision, memory consumption, and serving throughput, and thus form an “impossible triangle”.