GitHub

How we improved push processing on GitHub

GitHub broke one huge push-processing job into smaller, independently retried jobs driven by Kafka events.

Published
2024-06-11
Scale
Not stated in source
Techniques
queues job decomposition
Stack
Ruby on Rails Kafka

Try this stack yourself

  • Railway · Deploy something like this on Railway: spin up Ruby on Rails plus your app in a few clicks. New accounts get $20 in credit (applied when you pick a paid plan).
    Railway isn't part of GitHub's setup in this story. It's just an easy place to try the same building blocks.

Some links above are affiliate links: if you sign up, How They Scaled may earn a commission at no extra cost to you. We only list tools that fit the stack in the story.

See also

GitHub architecture

The best scaling write-ups, weekly

One email a week. No spam.

Subscribe

Related stories

LinkedIn · 2015

Running Kafka At Scale

How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.

800 billion messages/day175 TB/dayqueuesstreamingmulti-datacenterKafka
GitHub · 2021

Partitioning GitHub's relational databases to handle scale

GitHub defined schema domains and linted cross-domain queries so it could move table groups onto separate MySQL clusters.

vertical partitioningschema domainsonline migrationMySQLRuby on RailsVitess
OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.

1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka
Netflix · 2026

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

Netflix moved older data in its multi-petabyte TimeSeries Abstraction out of hot Cassandra storage in two steps. First, Kafka and Flink streamed events to S3 as Parquet files, which a daily job compacted into small, time-sorted row groups with Bloom filters cached in EVCache (p50 ~150 ms, p99 ~500 ms). The second design skips that streaming pipeline: a ReadOnlyCompactionStrategy aligns immutable SSTables to token ranges, and a custom pluggable Memtable (Cassandra 4.1) reads directly from existing Cassandra backups in S3. It now holds over 15 PB of compressed cold data, cut disk usage on affected clusters from 80% to 7%, and improved p90 latency by ~30%.

15+ PB compressed data in cold storagedisk usage cut from 80% to 7%tiered storagecompactioncost optimizationCassandraKafkaFlink