How teams scaled Kafka: 7 real case studies
7 of the 73 engineering write-ups in our database mention Kafka in their stack. They come from 7 companies (Cloudflare, Datadog, GitHub, LinkedIn, Netflix and 2 more) and were published between 2015 and 2026. Techniques that show up most often alongside Kafka: multi-tenancy (2) and queues (2). It's often paired with Go (2). 4 of the 7 state concrete scale numbers, for example "1B+ users across 40 regions" (OpenAI). Each card below links to our short summary and the original post.
Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
Netflix moved older data in its multi-petabyte TimeSeries Abstraction out of hot Cassandra storage in two steps. First, Kafka and Flink streamed events to S3 as Parquet files, which a daily job compacted into small, time-sorted row groups with Bloom filters cached in EVCache (p50 ~150 ms, p99 ~500 ms). The second design skips that streaming pipeline: a ReadOnlyCompactionStrategy aligns immutable SSTables to token ranges, and a custom pluggable Memtable (Cassandra 4.1) reads directly from existing Cassandra backups in S3. It now holds over 15 PB of compressed cold data, cut disk usage on affected clusters from 80% to 7%, and improved p90 latency by ~30%.
Building and scaling Notion's data lake
Notion replaced slow warehouse syncs with a CDC pipeline (Debezium to Kafka to Hudi on S3) processed with Spark.
How we improved push processing on GitHub
GitHub broke one huge push-processing job into smaller, independently retried jobs driven by Kafka events.
Husky: Exactly-once ingestion and multi-tenancy at scale
A deep dive on how Husky routes, shards and ingests events exactly once across tenants.
HTTP Analytics for 6M requests per second using ClickHouse
Cloudflare rebuilt its HTTP analytics pipeline on ClickHouse, replacing a Postgres/Citus setup.
Running Kafka At Scale
How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.