Zephon data loader gives the same results on 8 to 64 GPUs
A standard data loader changed the model score by 0.82 points when only the GPU count changed.
The authors of Zephon made a data loader. It keeps the sequence of training data the same when the GPU count changes. They used it in TorchTitan with a 1B-parameter model. With a standard data loader, the DCLM Core v1 score changed by 0.82 points between 8 and 64 GPUs. With Zephon, the change was 0.05 points.
Sources
Posted
