Eloi de Reynal

Affinity-Aware Sharding for Delayed Tensor Parallelism

It is shown that maximising the affinity between the KV heads and the FFN neurons co-located on a device, by permuting the dense model before sharding, speeds up the distillation or retraining process, and that the co-located affinity score at initialisation predicts the KL to the base model after training, across seve...