OpenAI · 2026
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.
70 million+ requests/sec1 billion+ weekly usersconnection poolingload balancingrate limitingPythonRustCosmos DB