OpenEverest exposes automatic failover of MySQL Galera cluster without data loss

What Percona tested, and why it matters

On November 12, 2026, Percona published the second part of a series on OpenEverest, its Kubernetes operator for open source databases. The author, Arunjith Aravindan, had already set up a three-node K3s cluster in Part 1 and built on it a three-replica Percona XtraDB Cluster (PXC) called mysql-vjg. In this second part, the test is more interesting for those who code against the database than for those who only administer it: it kills the writer pod while a client inserts one row per second, and observes what the application sees during the switch.

For those who build systems that depend on MySQL on Kubernetes, this is the question everyone avoids until the incident happens in production: when the pod that receives writes dies, does the client see an error, see slowness, or not notice at all? Percona's article answers with real numbers from a lab cluster, not with marketing promises.

How to read the health of a Galera cluster

PXC replicates data using the Galera library, which exposes its state through wsrep_* variables (from write-set replication). A write-set is the package of row changes generated by a transaction; on commit, it is sent to all nodes and goes through certification before being confirmed. Running SHOW STATUS LIKE '%wsrep%stat%' is, in Galera, the equivalent of checking SHOW SLAVE STATUS in traditional replication.

A detail every dev who's been bitten by a "deadlock" in Galera should know: wsrep_local_cert_failures counts transactions that failed certification because another node changed the same rows first. That's Galera refusing a concurrent write, not an application bug. wsrep_local_bf_aborts, in turn, shows local transactions forcibly aborted by a replicated write-set (the so-called brute-force abort): under heavy, concurrent write loads, these two counters rise together and indicate that the table's access pattern needs to change, not that the cluster is broken.

The state machine of a node joining or leaving

Every node that joins the cluster goes through a fixed sequence of states, visible in wsrep_local_state:

1. Joining: the node has entered the group and is requesting the data it's missing.
2. Donor/Desynced: the node is sending a state transfer to another that is joining; HAProxy takes it out of rotation while this happens.
3. Joined: the node has received the data and applies the write-sets accumulated in the queue.
4. Synced: the node is up to date; only nodes in this state receive traffic from HAProxy.

How the node recovers what it lost is the point that most affects production operation. If the gcache (the circular buffer of recent write-sets) of a donor node still contains everything the returning node lost, the recovery is an IST (Incremental State Transfer): a few seconds, little extra load. If it doesn't, Galera falls back to an SST (State Snapshot Transfer): a full copy of the database via Percona XtraBackup, which can take a long time and overloads the donor node. Sizing the gcache correctly is, in practice, the difference between a pod that leaves and comes back in seconds and one that stalls the cluster for minutes.

The test: killing the writer with real traffic flowing

Percona set up the test across three simultaneous terminals. A MySQL client runs inside the cluster itself, not through an SSH tunnel, precisely so that an external connection drop isn't mistaken for the failover itself:

while true; do
  out=$(mysql -h mysql-vjg-haproxy -uroot --connect-timeout=2 -Nse "
    INSERT INTO demo.failover_test (served_by) VALUES (@@hostname);
    SELECT @@hostname, LAST_INSERT_ID();" 2>&1 | tr '\n\t' '  ')
  echo "$(date +%T)  $out"
  sleep 1
done

Before the failover, every printed line shows mysql-vjg-pxc-0, the current writer, inserting one row per second without failures. That's the baseline: the article doesn't start in chaos, it starts by proving the normal flow works.

With the write loop running, the command that triggers the test is almost brutally simple:

kubectl delete pod mysql-vjg-pxc-0 -n everest

This sends a SIGTERM to the pod, simulating planned maintenance or a node restart. The StatefulSet that manages it immediately recreates a pod with the same name, but it joins as a new node: it will go through Joining, get an IST (or an SST, if the gcache is too small) and only start receiving traffic again once it reaches Synced.

What the application sees during the switch

The promise the article makes right at the opening is the one that matters most to those who write code against this database: HAProxy switches the writer, the downed pod rejoins, and the three nodes end up with identical data. The check used to prove this is the same one seen in Section A, run after the test: comparing wsrep_last_committed across the three pods via kubectl exec and confirming the number matches across all three.

This doesn't mean zero impact. During the window between the writer pod's death and HAProxy redirecting connections, the test's write loop logs timeouts or connection errors (the --connect-timeout=2 exists precisely to keep these attempts from hanging). What the test proves isn't the absence of transient errors: it's that no committed write is lost, and that the cluster resumes accepting writes on its own, without manual intervention on the database.

Where this helps, and where it doesn't solve things alone

For those deciding whether a Galera cluster orchestrated by Kubernetes is worth the complexity, three caveats apply, imposed by Galera's own behavior:

• Quorum requires a majority: a three-node cluster tolerates losing one; losing two at once takes the rest out of the Primary state and it stops accepting writes until quorum returns. This is protection against split-brain, not a bug, but it means planning for an odd number of nodes and distribution across availability zones.
• Flow control isn't free: if a node falls behind in the apply queue (wsrep_local_recv_queue_avg rising), Galera pauses writes across the entire cluster until it recovers. Very heavy write loads on uneven hardware suffer more from this than from the failover itself.
• The operator automates provisioning, not diagnosis: OpenEverest takes care of spinning up the StatefulSet, HAProxy, and the Kubernetes secrets, but understanding whether an incident is a certification failure, a slow SST, or flow control still requires reading the wsrep_* variables by hand, as Part 2 of the article teaches you to do.

For teams that already have this operational knowledge, OpenEverest's real gain is reducing the manual work of orchestrating PXC on Kubernetes, not eliminating the need to understand Galera underneath. Those looking for a managed database with no learning curve in synchronous replication are probably better served by a fully managed service; those who need fine-grained control over failover and topology will find, in this material from Percona, a reproducible playbook for validating that resilience before trusting it in production.