When failover isn't safe: Building high-availability PostgreSQL on Kubernetes
A gameday that injected network latency into one availability zone left several of Datadog's Kubernetes-based PostgreSQL clusters stuck: replicas lagged past Patroni's maximum_lag_on_failover, so no standby could be promoted safely. Datadog moved to a hybrid model: standbys in the leader pool take part in synchronous replication (one synchronous standby, synchronous_mode_strict, synchronous_commit=remote_apply) while read replicas stay asynchronous. pgbench showed remote_apply raised average latency by 53% and cut throughput by 34%, yet production clusters saw no measurable impact after a gradual rollout, and synchronous replication is now enabled in all of its clusters.
- Published
- 2026-06-04
- Scale
- +53% avg write latency with remote_apply (pgbench TPC-B), -34% tps with remote_apply (pgbench TPC-B)
- Techniques
- synchronous replication failover high availability chaos testing
- Stack
- PostgreSQL Kubernetes Patroni ZooKeeper
Try this stack yourself
- Railway · Deploy something like this on Railway: spin up PostgreSQL plus your app in a few clicks. New accounts get $20 in credit (applied when you pick a paid plan).
Railway isn't part of Datadog's setup in this story. It's just an easy place to try the same building blocks.
Some links above are affiliate links: if you sign up, How They Scaled may earn a commission at no extra cost to you. We only list tools that fit the stack in the story.