xCCLTuner: Treating xCCL as Black-Box and Automatically Tuning

摘要

The collective communication libraries (xCCL) are an important networking support for distributed training systems. Optimizing xCCL is crucial for reducing training costs while maintaining accuracy, but existing approaches often require extensive manual intervention, lack user-friendliness, and struggle to adapt to complex environments. Therefore, we introduce xCCLTuner, an out-of-the-box tuning system without needing to categorize or model parameters that automatically optimizes xCCL parameters for distributed training systems. xCCLTuner utilizes Bayesian optimization to navigate the high-dimensional parameter space of xCCL knobs in a black-box manner. It overcomes key challenges through three main designs: (1) Low-dimensional tuning to reduce the complexity of the parameter space, (2) xCCL-specific lookup table to identify and skip fatal parameter combinations, and (3) Bucketization techniques to conquer vast parameter domains. xCCLTuner rapidly delivers optimized training times across various topologies, showcasing its efficiency and ease of use. Our experiments demonstrate that xCCLTuner achieves up to a 5.87X improvement in throughput.