Queues: 3 real scaling case studies
3 of the 73 engineering write-ups in our database are tagged with the technique "queues". They come from 3 companies (Discord, GitHub and LinkedIn) and were published between 2015 and 2024. Stack that shows up most often in these stories: Kafka (2). 2 of the 3 state concrete scale numbers, for example "billions of messages" (Discord). Each card below links to our short summary and the original post.
GitHub · 2024
How we improved push processing on GitHub
GitHub broke one huge push-processing job into smaller, independently retried jobs driven by Kafka events.
Discord · 2017
How Discord Indexes Billions of Messages
Discord built message search on many small Elasticsearch clusters, sharding by guild and indexing lazily through a queue.
LinkedIn · 2015
Running Kafka At Scale
How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.
Get new queues scaling stories weekly
One email a week. No spam.