nn-Live: Predicting Runtime Latency for Concurrent DNN Inference on Mobile Devices

摘要

Modern mobile applications (e.g., AR/XR, assistant robots) increasingly rely on multiple concurrent DNN models, yet their inference latency becomes highly unpredictable under dynamic resource contention from concurrent workloads. In mobile SoCs with heterogeneous CPU architectures, such as big. LITTLE designs, the latency is affected by a complex interplay of factors, including heterogeneous core performance, cache hierarchies and shared memory bandwidth. Existing latency prediction methods either fail to capture runtime dynamics or overlook the heterogeneity in mobile processors. In this paper, we propose nn-Live that can predict live inference latency for DNN models on mobile CPUs under dynamically changing operating conditions. nn-Live leverages per-core hardware counter statistics to quantify the interference from concurrent tasks, and utilizes a lightweight linear model to estimate each core's interference contribution to a specific task, enabling robust latency prediction across dynamic execution conditions. Given the large pool of DNN models and huge potential task placement configurations, nn-Live further adopts an adaptive hierarchical sampling strategy to reduce profiling cost while preserving configuration diversity and representativeness. We build a large-scale interference analysis dataset1 across three mainstream SoCs. Evaluation results show that nn-Live achieves high prediction accuracy under diverse runtime environments, with a clear advantage over existing methods. We further demonstrate the extensibility of nn-Live to mobile accelerators (e.g., GPU/NPU).