Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
- Published
- 2026-10-23
- Scale
- 1B+ users across 40 regions, 500+ PB served by Habitat, largest Cosmos DB account: 10s of PB, 20M+ QPS, global read p50 ~120 ms to ~10 ms (12x), millions of CDC changes/sec
- Techniques
- multi-tenancy hot key throttling CDC cache invalidation geo-replication Rust rewrite
- Stack
- Azure Cosmos DB Valkey Kafka Rust Python