Frequent Checkpointing through Mergeable Delta Compression and Semi-Reliable Transmission

摘要

Checkpointing is a widely used technique to ensure fault tolerance in distributed machine learning systems. Frequent checkpointing can significantly reduce the wasted training time in the event of a failure. However, due to the large overhead of checkpoint storage and transmission, it is still challenging to achieve frequent checkpointing in practice. In this paper, we propose FlashCheck, a checkpointing framework that enables frequent checkpointing in distributed machine learning systems. FlashCheck first introduces the mergeable delta compression (MDC) technique that provides high compression ratios for checkpoints with low computational overhead. MDC allows for efficient merging of compressed checkpoints to mitigate the continuously increasing storage overhead. To further reduce the transmission overhead, FlashCheck employs a semi-reliable transport protocol (CSTP) that offers low tail transmission latency and fine-grained control over the reliability of checkpoint transmission. Through extensive experiments on real-world distributed machine learning workloads, we demonstrate that FlashCheck can conduct a significant checkpoint size reduction of over 99.17%. It can achieve over 4x speedup in the checkpointing frequency compared to existing solutions.