Field report

CloudNativePG failover drill: what we measure and what can go wrong

Versions in use on the drill date: PostgreSQL 18, CloudNativePG 1.28.

Planned database failover on 4 Oct 2026: we switched Fortlauf's product database to another replica while the service was running. The database cluster was fully back in service 19 s after the switch; writes paused for about 8 to 9 s, and no external request received a server error in that time. Not included: an unplanned outage.

This article shows how we prepared that drill, where each timer started and stopped, what the number does not tell you, and a checklist for running the same drill on your own CloudNativePG cluster. It also covers a mistake we made later the same day.

1. Why rehearse a planned switchover

CloudNativePG calls a planned change of primary a switchover and an unplanned one a failover. Both end the same way: a replica is promoted, the old primary rejoins as a replica, and the read-write service points at the new primary. A switchover is the part you can rehearse on purpose, in a window you choose, with a fresh backup behind you.

It is also not rare. The operator starts switchovers by itself during rolling updates, operator upgrades and node maintenance. If you have never watched one end to end, the first one you watch may be the one a node drain started at a bad moment.

A drill answers three questions you cannot answer from the manifests:

  • How long are writes really interrupted, and do your applications recover by themselves?
  • Does every replica follow the new primary, or does one silently stay behind?
  • Does continuous archiving carry on onto the new timeline?

2. Preparation: do not drill a degraded cluster

A switchover on a cluster that is already short of a healthy replica can leave you with no standby at all. So the drill starts with checks, and stops if any of them fails.

The cluster object is not enough. A cluster can report a healthy phase with every instance ready while one replica has stopped following the primary. Read three things: the phase, the timeline each instance reports, and each instance’s log.

# Phase, current primary and timeline, plus what each instance reports
kubectl -n app get clusters.postgresql.cnpg.io my-cluster \
  -o jsonpath='{.status.phase} | primary={.status.currentPrimary} TL={.status.timelineID}{"\n"}{.status.instancesReportedState}{"\n"}'

# Continuous archiving must be True
kubectl -n app get clusters.postgresql.cnpg.io my-cluster \
  -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}{"\n"}'

# No replica history errors and no isolation-check errors in the last hour, per instance
for p in $(kubectl -n app get pods -l cnpg.io/cluster=my-cluster -o name); do
  echo "$p: $(kubectl -n app logs "$p" --since=1h | grep -c -E "not in this server's history|connectivity error")"
done

# The disruption budget allows one instance to go
kubectl -n app get pdb my-cluster

What we required before the switch:

  • One healthy phase, all instances on one timeline, zero history errors in every instance log.
  • No operator restart in the previous 24 hours.
  • Every application that talks to the database running with at least two replicas.
  • If you use CloudNativePG’s primary isolation check with a NetworkPolicy, the instances must reach each other on the instance manager port, not only on 5432. Otherwise the check’s peer leg fails quietly, and a short API-server slowdown can be enough to kill the primary.
  • A fresh base backup, completed, and its last WAL file archived before the switch. The weekly backup was a week old; we did not want the drill to depend on it.
kubectl cnpg backup my-cluster -n app \
  --method plugin --plugin-name barman-cloud.cloudnative-pg.io   # Barman Cloud plugin
kubectl -n app get backups.postgresql.cnpg.io     # wait for the new backup to show "completed"

Pick the target on purpose. Choose the replica whose node has had no I/O stalls and whose log is clean, not whichever the tooling suggests. Use a kubectl cnpg plugin build that matches the operator version, and verify its checksum.

Agree the window and the roles. One person runs the commands. Someone else watches the application logs and the ingress. Write the window and the start time (UTC) in a run log before you begin.

3. What we measured, and how

The switch itself is one command:

kubectl cnpg promote my-cluster my-cluster-2 -n app

While it ran, a loop recorded the cluster status every 2 seconds with a UTC timestamp:

while true; do
  printf '%s ' "$(date -u +%T)"
  kubectl -n app get clusters.postgresql.cnpg.io my-cluster \
    -o jsonpath='{.status.phase} | primary={.status.currentPrimary} TL={.status.timelineID}{"\n"}'
  sleep 2
done | tee switchover-run.log

Timer definitions. These decide what the headline numbers mean:

Measure Starts Stops
Back in service the promote command the cluster reports a healthy phase with all instances ready
Write pause the first database error in the application logs (connections terminated by the server) the last refused connection before the new primary took over
External errors the first database error the end of the window; we counted 5xx status codes in the ingress access log, matching the status field itself, not any “5” in the line

What we saw, in seconds after the promote:

After What the status showed
+3 s switchover in progress, target named
+8 s the target is the current primary
+11 s the timeline number has gone up by one
+19 s healthy, all instances ready
  • Replicas. The old primary noticed it had diverged, needed no rewind, and was streaming from the new primary 17 s after the promote. The other replica restarted streaming on the new timeline at +19 s.
  • Writes. The first application error came 2 s after the promote, the last refused connection at +10 s, and the new primary was in place at +11 s. That is the write pause of about 8 to 9 s.
  • External requests. No 5xx at the ingress. The background workers retried and recovered on their own.
  • Archiving. The new timeline’s history file was archived 9 s after the promote, and the first full WAL segment on the new timeline about five minutes later, when that segment was closed.
  • The timeline in the status lags. The timeline each replica reports in the cluster status only caught up about 8.5 minutes later, at the replicas’ next restartpoint. Their logs showed the new timeline within seconds. If your check reads the status only, give it time or read the logs.

Check every criterion before you call it passed:

  • current primary = the target, and the timeline is one higher than before;
  • every instance reports that timeline;
  • no instance logs “not in this server’s history”;
  • archiving is still True, and a WAL file for the new timeline reaches the backup store;
  • every database client has no crash loops and no database errors after the window, and queues drain (the pending count falls; pod status alone is not proof).

We did not switch back afterwards. Each switchover is a risk of its own, and one clean drill is the evidence.

4. What the number does not cover

  • An unplanned outage. In a planned switchover the old primary shuts down cleanly and hands over. When a node dies, or the kubelet kills the primary after failed liveness probes, detection time and the primary’s shutdown behaviour add to the pause. Check two settings in your cluster spec: failoverDelay (default 0) is how long the primary must be unhealthy before the failover starts, and switchoverDelay is the timeout for the failing primary’s fast shutdown before an immediate shutdown follows. smartShutdownTimeout applies when a pod is deleted, not to the failover decision.
  • How every client reconnects. Our number is measured at the ingress, for requests from outside. A worker that holds idle pooled connections sees them terminated by the server. If its driver raises that as an unhandled error, the process exits and restarts. Nothing is lost if work sits in an outbox, but it is a restart you did not plan for. Test this per client.
  • Data under load. The drill ran in an agreed quiet window. A switchover at peak write load will show a different pause.
  • Restores. A switchover proves replication and promotion. It does not prove that you can restore from your backups. That needs its own drill.

5. Checklist for your own drill

Before:

  • Phase healthy, all instances on one timeline, zero history errors in every instance log.
  • Instances can reach each other on the instance manager port if a NetworkPolicy and the isolation check are in use.
  • No operator restarts in the last 24 h; the disruption budget allows one instance.
  • Every database client runs with at least two replicas.
  • Fresh base backup completed; its last WAL file archived.
  • kubectl cnpg plugin matches the operator version; checksum verified.
  • Target replica chosen on purpose; window, roles and start time written down.
  • No node cordoned, no drain running, no other change to the cluster in flight.

During:

  • One switchover, one command. If the target does not become primary within about two minutes, stop and read the operator log. Do not send a second order.
  • Status loop running every 2 s with UTC timestamps; application and ingress logs open.

After:

  • Current primary = target; timeline one higher; every instance on it (allow for the status lag, or read the logs).
  • Exactly one instance logged that it was promoting for this switchover.
  • Every replica is replaying, not just ready: its log shows streaming on the new timeline and none of “ahead of the WAL flush position”, “incorrect prev-link”, “invalid resource manager ID”, “terminating walreceiver process due to administrator command”.
  • Archiving True; a WAL file for the new timeline in the backup store.
  • Write pause and time to healthy recorded with their start and stop definitions.
  • Clients: no crash loops, queues drained.
  • No switch back just to restore the old placement.
# Exactly one promotion for this switchover (sum over all instances must be 1)
for p in $(kubectl -n app get pods -l cnpg.io/cluster=my-cluster -o name); do
  echo "$p: $(kubectl -n app logs "$p" --since=15m | grep -c "applying WALs and promoting")"
done

# Each replica is replaying on the new timeline, with none of the fork symptoms
kubectl -n app logs my-cluster-1 --since=10m \
  | grep -E "streaming|prev-link|invalid resource manager|ahead of the WAL flush|terminating walreceiver"

6. What went wrong later the same day

The drill was clean. Later on 4 Oct 2026, during node maintenance, we changed the primary of two clusters again: the same product database cluster, and a second database cluster on the same platform. Both went wrong in the same way.

  • Two orders, under a second apart. Cordoning the node that ran the primary made the operator start its own switchover to a replica it picked. That is CloudNativePG working as designed: node maintenance triggers a switchover. Less than a second later, a manual change set a different replica as the target.
  • Both replicas promoted. Each took the order and promoted itself, and both picked the same new timeline number. The operator kept one and demoted the other.
  • The rewind saw nothing to do. pg_rewind compares timelines. Both forks carried the same number, so it reported that no rewind was required, and the demoted instance kept WAL the real primary never wrote.
  • The fork reached the backup store. The losing instance archived its version of one WAL segment. In one of the two clusters, another replica restored that segment from the archive and replayed it. The real primary overwrote the object about five minutes later, when it closed its own segment.
  • The cluster still said healthy. The affected replicas were ready and the cluster reported a healthy phase. They had stopped replaying. Only their logs showed it.

What we did. We rebuilt the affected replicas from the current primary the same day: delete the replica’s volume claim and pod, and let the operator clone a fresh one. A restart does not repair a forked history. The application was unaffected. The losing instances were writable for a fraction of a second. Applications reach the database only through the service that points at the primary, so a lost write is very unlikely, but the logs we had could not rule it out completely.

What we changed in our switchover runbook:

  • One switchover at a time. After a cordon, wait until the cluster reports the new primary and a healthy phase before any further change. If the result is not the instance you wanted, run a second, separate switchover. Prefer the plugin’s promote command to editing the cluster status by hand.
  • After every switchover, check that exactly one instance promoted and that every replica is replaying, not just ready (the last part of the checklist above).
  • If two instances promoted, rebuild the losing one at once, and any replica that restored a WAL segment from the archive in the minutes before the real primary’s next segment switch.
  • For point-in-time restores, treat the minutes after a double promotion as suspect, because the archive briefly held the wrong segment.
  • An alert on “replica timeline differs from the primary’s” would not have fired here: every instance reported the same timeline number. Watch replay progress instead: a replica whose replay position stays flat while the primary’s advances.

Where this fits

We build and run Kubernetes platforms with PostgreSQL on CloudNativePG, and we rehearse restores and switchovers as part of the handover: Kubernetes platforms, built and run. Where the data behind this drill lives, and our other dated measurements: Where your data lives.

Want this measured on your own cluster?

The Disaster Recovery Drill runs a planned switchover or a restore of one agreed system with your team, times every step and ends in a written report with the gaps and a prioritised fix list.