Optimizing Split Federated Learning through Adaptive Pipeline Parallelism

Abstract

Split federated learning (SFL) offers a promising solution for training large models on resource-constrained edge devices. However, existing synchronous and asynchronous SFL frameworks are plagued by a fundamental sequential execution pattern for communication and computation. This design creates a severe communication bottleneck, particularly for modern large models with substantial activation sizes, leading to significant resource underutilization and extended training time. To dismantle this bottleneck, we propose MicroSFL, a novel SFL framework with adaptive pipeline parallelism. MicroSFL enables workers to divide each of their training mini-batches into multiple small micro-batches and process them in a pipelined fashion. To counteract the challenges of system and statistical heterogeneity, MicroSFL adaptively assigns distinct micro-batch sizes to workers to accommodate their heterogeneous capacities, maximizing the temporal overlap between the computation and transmission of activations/gradients. Besides, to address statistical heterogeneity, MicroSFL employs an adaptive updating strategy. This strategy accumulates gradients at the server side to simulate updates on a large, balanced batch. Guided by a theoretical convergence analysis, the optimal updating weights are assigned to different workers to normalize their contributions for model updating. MicroSFL then dynamically determines the appropriate timing to apply the aggregated gradients, ensuring both stable convergence and high model accuracy. Extensive evaluations on a physical platform of 80 edge devices show that MicroSFL accelerates training by 3.64× to 5.67× and improves model accuracy by 3.6% to 6.1% compared to the baselines.