Distributed training: 3 real scaling case studies
3 of the 73 engineering write-ups in our database are tagged with the technique "distributed training". They come from 1 company (Meta) and were published between 2024 and 2026. Stack that shows up most often in these stories: GPUs (2) and PyTorch (2). Often combined with hardware reliability (2) and networking (2). 2 of the 3 state concrete scale numbers, for example "gigawatt-scale supercluster" (Meta). Each card below links to our short summary and the original post.
Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster
Meta turned its New Albany, Ohio data center site from web traffic into Prometheus, a gigawatt-scale AI supercluster spanning multiple interconnected buildings that behaves as one machine. It runs NVL72 racks (Catalina on GB200 with 800G of scale-out per GPU, Clemente on GB300 in a single rack) on a dedicated Ethernet back-end network for GPU-to-GPU traffic. Reliability work pushed pre-training goodput above 85% and mean time between interruptions past 54K GPU hours, and the cluster is growing toward roughly 1 million GPUs.
Meta's Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI
Meta replaced separate per-surface ad prediction systems with GEM, one central recommendation foundation model trained across all of its ad surfaces. A new LLM-style architecture (Kunlun) showed LLM-like scaling with 2.5x more model complexity, while custom low-precision kernels and 5D parallelism doubled end-to-end training efficiency to 20-25% MFU at 4x the training FLOPs across thousands of GPUs. Post-training transfer (LoopFM) and elastic pre-computation (Solaris) pass GEM's gains to real-time serving models without extra latency.
How Meta trains large language models at scale
Meta covers the hardware, networking and reliability work needed to train LLMs across very large GPU clusters.