Rate limiting: 3 real scaling case studies
3 of the 73 engineering write-ups in our database are tagged with the technique "rate limiting". They come from 2 companies (OpenAI and Stripe) and were published between 2017 and 2026. Stack that shows up most often in these stories: Cosmos DB (2). Often combined with connection pooling (2). 2 of the 3 state concrete scale numbers, for example "70 million+ requests/sec" (OpenAI). Each card below links to our short summary and the original post.
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.
Scaling PostgreSQL to power 800 million ChatGPT users
OpenAI runs ChatGPT on a single-primary Postgres with nearly 50 global read replicas. Reads are offloaded, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe.
Scaling your API with rate limiters
The four kinds of rate limiters and load shedders Stripe uses to protect its API.