Distributed Training

Parallelism methodshuggingface

Amazing resource for hands-on training for data parallesism, tensor parallelism, model parallelism, ZeRo, etc.

GitHub - tunib-ai/large-scale-lm-tutorials: Large-scale language modeling tutorials with PyTorchGitHub

Transformer Math 101EleutherAI Blog