Achieving relentless Kafka reliability at scale with the Streaming Platform

Datadog treats Kafka as commodity hardware behind a Streaming Platform control layer. Streams span multiple Kafka clusters, AZs, and Kubernetes clusters so producers and consumers talk to stable identifiers while the control plane redirects traffic to healthier topics in seconds—for incidents, rebalancing, decommissioning, or repartitioning—without app redeploys. A custom Assigner replaces Kafka's group coordinator for multi-cluster rebalancing in seconds; Stream lanes isolate QoS and park poison pills in a DLQ; an enriched commit log tracks multiple offset ranges plus timestamps so consumers can process live traffic while replaying backlogs and the control plane can compute wall-clock lag across millions of partitions. After zstd "bombs" crashed stock clients, Datadog wrote libstreaming in Rust (with Java/Go/Python bindings) for unified, high-throughput access without a proxy in the TB/s data path.

Published

2025-02-19

Scale

hundreds of trillions of observability events/day, hundreds of Kafka clusters, thousands of topics, millions of partitions, terabytes of traffic/sec, petabyte-scale pipelines

Techniques

multi-cluster failover custom coordinator commit log enrichment QoS lanes dead-letter queue

Stack

Apache Kafka Kubernetes Rust libstreaming

Read the original on Datadog's blog →