PhOrch: Proactive Phase-Level Flow Path Orchestration For Contention-Free LLM Training

摘要

The growing scale of large language models (LLMs) has made communication overhead a critical bottleneck in distributed training, primarily due to imbalanced traffic loads. Existing load balancing methods often lead to severe flow contention when handling low-entropy and high-volume LLM training flows. Motivated by the point-to-point pattern in each collective communication phase and the inherent periodicity of training traffic, we propose PhOrch, a proactive phase-level contention-free flow path orchestration framework tailored for LLM training workloads. We formulate the orchestration as an optimization problem, which is typically NP-hard. To tackle this problem, we develop a segmented edge coloring algorithm for bipartite multigraphs, which efficiently assigns flow paths while avoiding contention. Evaluation results demonstrate that PhOrch reduces the per-cycle communication time by 60% compared to the state-of-the-art methods and achieves contention-free training traffic in non-oversubscribed topologies, indicating that our method substantially mitigates the communication bottleneck during the LLM training process.