Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.
- Published
- 2026-09-11
- Scale
- 70 million+ requests/sec, 1 billion+ weekly users, 500 PB+, almost 40 regions
- Techniques
- connection pooling load balancing rate limiting change data capture rewrite
- Stack
- Python Rust Cosmos DB Envoy Rockset
Try this stack yourself
- Better Stack · Monitor uptime and logs with Better Stack, and get alerted fast if a migration or traffic spike goes sideways on your own system.
Better Stack isn't part of OpenAI's setup in this story. It's a tool for watching your own systems while you try ideas like these.
Some links above are affiliate links: if you sign up, How They Scaled may earn a commission at no extra cost to you. We only list tools that fit the stack in the story.