LinkedIn architecture: how LinkedIn scaled (2 case studies)
LinkedIn runs Kafka as its central data pipeline, with cluster tiers and mirroring between datacenters; its 2015 post cites 800 billion messages and 175 TB a day. LinkedIn also moved its feed to FollowFeed, a fan-out-on-read system that keeps member timelines in memory and on RocksDB.
We track 2 public engineering write-ups by LinkedIn, published between 2015 and 2016. They cover fan-out on read, multi-datacenter, partitioning and queues. Stack mentioned: Java, Kafka and RocksDB. 1 of the 2 state concrete scale numbers, for example "800 billion messages/day". Each card below links to our short summary and the original post.
Tech used in LinkedIn's scaling write-ups
| Tech | Where it comes up |
|---|---|
| Java | FollowFeed: LinkedIn's Feed Made Faster and Smarter (2016) |
| Kafka | Running Kafka At Scale (2015) |
| RocksDB | FollowFeed: LinkedIn's Feed Made Faster and Smarter (2016) |
LinkedIn scaling timeline: every story, summarized
- 2015-03-20
Running Kafka At Scale
How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.
- 2016-03-31
FollowFeed: LinkedIn's Feed Made Faster and Smarter
LinkedIn moved its feed to a fan-out-on-read system that keeps member timelines in memory and on RocksDB.
FAQ
How does LinkedIn use Kafka?
LinkedIn's 2015 post Running Kafka At Scale describes Kafka as its central data pipeline, with cluster tiers and mirroring between datacenters, at 800 billion messages and 175 TB per day.
How does LinkedIn's feed work?
LinkedIn's 2016 FollowFeed post describes a fan-out-on-read feed that keeps member timelines in memory and on RocksDB, written in Java.