# Distributed Training

{% embed url="https://huggingface.co/docs/transformers/en/perf_train_gpu_many" %}

Amazing resource for hands-on training for data parallesism, tensor parallelism, model parallelism, ZeRo, etc.

{% embed url="https://github.com/tunib-ai/large-scale-lm-tutorials" %}

{% embed url="https://blog.eleuther.ai/transformer-math/" %}
