Prism: Efficient Zeroth-Order Fine-Tuning for On-Device LLMs

摘要

On-device fine-tuning of Large Language Models (LLMs) is becoming increasingly important for personalized AI assistants and adaptive agents. However, standard backpropagation imposes substantial training-state memory overheads that often exceed the stringent resource budgets of mobile platforms. To address this, Zeroth-order (ZO) optimization leverages forward-only probing instead of backpropagation, drastically reducing memory footprints and offering a compelling alternative for on-device adaptation. Nonetheless, scaling ZO fine-tuning to billion-scale trainable parameter spaces remains challenging, as the cost of recovering reliable update directions scales poorly with dimensionality under mobile constraints. In this paper, we present Prism, a mobile-native infrastructure for practical billion-scale on-device ZO fine-tuning. Prism re-architects the ZO loop around a scalable probing substrate, bringing both per-step training latency and memory overhead closer to the inference footprint. We implement Prism end to end on commodity smartphones and evaluate it across diverse tasks and multiple model scales. Prism achieves accuracy comparable to that of state-of-the-art ZO fine-tuning baselines while reducing per-step latency by 53.4%-76.3% and peak memory by 10.6%-13.8% relative to the best baseline in each setting. These results demonstrate the practical feasibility of billion-scale on-device ZO fine-tuning for LLMs under real mobile constraints. We will release our code to support future research.