This book is a comprehensive guide to scaling training of models. It covers a wide range of topics, from hardware to software to model design, and provides a comprehensive overview of the current state of the art. It's a great read for anyone who wants to understand the challenges and solutions for scaling training of models. Its also got exercises and code examples to help you understand the concepts.
Picrotron provides a barebones implementation of Nvidia's Megatron-LM. It's a great read and a hackable starting point for anyone who wants to build their own language model.
Provides a comprehensive overview of the challenges and solutions for scaling training a model on 10K H100 GPUs. Soumith Chintala, from Meta AI, provides a detailed overview of the challenges and solutions for scaling training a model on 10K H100 GPUs. From the hardware to the software, it's a great read for anyone who wants to understand the challenges and solutions for scaling training a model on a large cluster.