Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute
Datadog's eBPF-based file integrity monitoring sees more than 10 billion file events per minute, and at about 5 KB each, shipping them all would mean terabytes per second. Agent-side rules cut what goes upstream to roughly one million events per minute, and moving evaluation into the kernel stops the ring buffer from overflowing: static approvers compiled from detection rules let matching events through, while runtime discarders stored in LRU eBPF maps drop values no rule can match. Together they pre-filter up to 94% of events in the kernel with no dropped events.
- Published
- 2025-11-18
- Scale
- 10B+ file events/min across Datadog's infrastructure, ~5 KB per serialized event, up to 94% of events filtered in-kernel, ~1M events/min sent upstream
- Techniques
- in-kernel filtering edge filtering approvers and discarders LRU maps
- Stack
- eBPF Linux Datadog Agent