Discord · 2026
How Discord Automates ScyllaDB Clusters at Scale
Discord's 7-person persistence team replaced fragile Python and bash scripts with the Scylla Control Plane, a Rust framework of idempotent tasks with declared preconditions, composed into YAML workflows. Jobs batch nodes by availability zone with concurrency limits to protect quorum, store their state in SQLite so they resume after interruptions, and send a webhook alert on unrecoverable errors. Operations like standing up shadow clusters and rolling upgrades went from a day and a half of babysitting to under two hours.
dozens of clusters, hundreds of database nodes7-person persistence teamdatabase operations automationidempotent taskszone-aware rolloutsScyllaDBRustYAML
OpenAI · 2026
Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka