OpenAI architecture: how OpenAI scaled (3 case studies)
OpenAI runs ChatGPT on a single-primary PostgreSQL database on Azure with nearly 50 global read replicas, serving 800 million users and millions of QPS. Reads are offloaded to replicas, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe. Its Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second, later rewritten in Rust.
We track 3 public engineering write-ups by OpenAI, published in 2026. Techniques that come up more than once: connection pooling (2) and rate limiting (2). Stack mentioned most: Cosmos DB (2), Python (2) and Rust (2). 3 of the 3 state concrete scale numbers, for example "1B+ users across 40 regions". Each card below links to our short summary and the original post.
Tech used in OpenAI's scaling write-ups
OpenAI scaling timeline: every story, summarized
- 2026-01-22
Scaling PostgreSQL to power 800 million ChatGPT users
OpenAI runs ChatGPT on a single-primary Postgres with nearly 50 global read replicas. Reads are offloaded, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe.
- 2026-09-11
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.
- 2026-10-23
Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
FAQ
Does ChatGPT use PostgreSQL?
Yes. OpenAI's 2026 post says ChatGPT runs on a single-primary PostgreSQL with nearly 50 global read replicas, for 800 million users and millions of QPS, with write-heavy shardable workloads moved to Cosmos DB.
How does OpenAI protect its Postgres primary?
Per OpenAI's post: reads go to replicas, PgBouncer pools connections, cache leasing reduces load, multi-layer rate limits cap traffic, and write-heavy workloads that can be sharded move to Cosmos DB.
What is OpenAI's Habitat storage platform?
Habitat grew from a Python client library over Cosmos DB into a central storage service handling 70M+ requests per second, using Envoy for connection fan-in and a deliberately simple NoSQL API, before being rewritten in Rust. The post cites 1 billion+ weekly users and 500 PB+.