Dropbox

Scaling to exabytes and beyond

Dropbox built Magic Pocket, its own exabyte-class block store, and moved user data off third-party cloud storage.

Published
n.d.
Scale
500 PB
Techniques
custom storage migration off cloud erasure coding
Stack
Custom storage Rust Go

See also

Dropbox architectureGo vs Rust for backend services at scale

The best scaling write-ups, weekly

One email a week. No spam.

Subscribe

Related stories

Dropbox · 2019

How we optimized Magic Pocket for cold storage

Dropbox added a cold tier to Magic Pocket that cuts replication overhead for rarely read data.

tiered storageerasure codingCustom storage
Discord · 2020

Why Discord is switching from Go to Rust

Discord rewrote its Read States service from Go to Rust to get rid of garbage-collection latency spikes in a large in-memory cache.

rewritecachinglatency reductionRustGo
Dropbox · n.d.

Rewriting the heart of our sync engine

Dropbox rewrote its desktop sync engine in Rust with a new data model built for correctness and testability.

rewritetestingRustPython
OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.

1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka