PolarFS Implements Disk-Paxos Lease to Prevent Split-Brain on Shared Disk

Sharing a single disk across multiple machines is a common architecture in databases that separate compute from storage: PolarDB, Amazon Aurora, and similar systems follow this design. The gain is elasticity, because replicas read the same block without copying data. The risk is the opposite of the gain: if two nodes decide, at the same time, that they own the write, the physical page gets overwritten by two different sources of truth, and there is no possible recovery after that.

A post published on the Awide Labs blog on October 24, 2026, and republished by Planet PostgreSQL, details how PolarFS, the user-space file system behind PolarDB, closed a real gap on this point. The post is signed by NeilXP and describes the mechanism called the disk-paxos lease: a lock written to the shared disk itself, renewed every second and protected by a watchdog that kills the node when renewal stops.

What a PBD Is, and Why the Local Lock Wasn't Enough

In PolarFS vocabulary, a PBD is the shared virtual disk: a single block accessible by multiple compute nodes. Each PBD can be mounted in read-write (RW) or read-only (RO) mode, reproducing PostgreSQL's classic primary-and-replicas topology. The rule is absolute: the engine assumes each page has exactly one authoritative writer, and all the failure-recovery and replication logic depends on that.

The original code already prevented a duplicate RW mount, but only within the same server. The function paxos_hostid_local_lock opens a file in /var/run/pfs/ and takes an fcntl on it; two processes on the same host competing for the same PBD get blocked. The problem is what falls outside that boundary: two different hosts, each convinced it owns the write, have no barrier between them at all. That is exactly the split-brain that any distributed systems architect fears, and its first symptom is usually corruption already done, not a timely alert.

The Awide Labs article points out that the skeleton of the cross-host protocol already existed in the code, but was never finished: the mnt_host_generation field was initialized to zero and never incremented, the per-host sectors in the paxos file were reserved but never written, and the pfs_log_paxos_try_release function was a stub that just returned success. The timing machinery was ready; the on-disk protocol was not.

The Disk-Paxos Lease, Sector by Sector

The fix allocates a 512-byte sector per host in the paxos file, using a structure called pfs_host_record_t:

struct pfs_host_record {
 uint32_t hr_magic; /* 0xcafebabe, or 0 when empty */
 uint32_t hr_flags; /* FL_RW, FL_PREPARE */
 uint32_t hr_host_id;
 uint32_t hr_generation; /* prev+1 after crash */
 uint64_t hr_timestamp; /* heartbeat */
 uint64_t hr_mbal; /* promised ballot */
 uint64_t hr_bal; /* accepted ballot */
 uint8_t hr_unused[468];
 uint32_t hr_checksum;
};

Three fields carry all the logic. hr_flags marks the protocol phase (FL_PREPARE for "I'm trying," FL_RW for "I hold the lease"). hr_generation distinguishes a live mount from the record left behind by an instance that crashed, incrementing with every new mount. hr_timestamp is the heartbeat, written using CLOCK_REALTIME, which is why hosts need to keep their clocks synchronized via NTP: a clock jump can make a live holder look expired, or a dead record look fresh. The protocol tolerates this within a configurable paxos_clock_skew_max (2 seconds by default).

The RW mount runs a two-phase protocol inside pfs_leader_load:

1. Check: reads every host's sector; if any RW record is alive, it fails with -EBUSY.
2. Prepare: writes FL_PREPARE with its own ballot, announcing intent.
3. Verify: rereads all sectors; any fresh RW record from another host, or a higher promised ballot, makes this host back off.
4. Acquire: writes FL_RW with its ballot, claiming the lease.
5. Conflict check: rereads again to catch the case where two hosts went through prepare at the same time; the higher ballot wins.

The ballot is calculated as generation * num_hosts + host_id, which guarantees that a remount after a crash always outranks the record left by its own previous instance. This is the reduction of disk Paxos to its single-writer form: each host writes its own slot, reads the others, and the freshest record decides (a live RW beats any ballot, and among live contenders the higher ballot wins).

Takeover: Wait, Then Verify

When a new node finds a live lease, it doesn't fail right away. It polls the holder's sector every second, for up to a full lease window (30 seconds by default, plus the clock skew margin), and observes three possible outcomes: the timestamp keeps advancing (live holder, -EBUSY); the sector gets zeroed out (holder released cleanly, proceed to acquire); or the timestamp stops moving for the entire window (holder presumed dead, proceed to takeover).

This design has a direct, measurable cost: failover from a stuck node can take up to 30 seconds plus the skew margin before another node takes over the write. For anyone designing a high-availability SLA, this number has to be accounted for: it's the price of split-brain safety, not an implementation detail.

Autofencing: Killing Itself Before Risking a Second Write

The most relevant part of the design, from the perspective of anyone building distributed systems, isn't the holder election, it's what happens when the holder can no longer prove it's alive. The lease is continuously renewed by the log thread, every log_paxos_lease seconds (one second by default), independent of data traffic, so an idle database doesn't let the lease expire for lack of writes.

When renewal fails, the node fences itself off in layers:

• Abort: after a full lease window without renewal, the thread itself calls abort().
• Timer-kill: a POSIX timer armed at acquisition fires SIGUSR2 and then SIGKILL, working even if the process is stuck somewhere else. If this timer can't be armed, the mount simply refuses the lease.
• Kernel watchdog: with paxos_watchdog_enable on, the node feeds /dev/watchdog on every renewal; if the entire kernel freezes, the watchdog reboots the machine.

This chain is the real guarantee: a holder that can't renew gets killed by the mechanism itself, and a dead holder doesn't write. It's the same principle behind techniques like STONITH in traditional clusters, applied inside the shared disk itself instead of relying on an external orchestrator.

What Still Depends on Assumptions, and Why That Matters

The Awide Labs article is honest about the design's limits: the first version of the protocol had flaws that follow-up work fixed, such as a verify step that only compared ballots (and so couldn't see a live holder with a lower ballot), corrupted sectors treated as empty, and an unsigned age calculation vulnerable to a fast clock. With the fixes, the guarantee holds under two assumptions that remain explicit: host clocks stay within the configured skew margin, and a fenced holder actually stops writing.

For anyone designing or operating distributed databases, the central lesson isn't the PolarFS detail itself, it's the pattern: an application lock is only worth something if it's tied to a mechanism that prevents the physical write once the lock should have expired.

A lease in etcd or ZooKeeper without a fencing token on the protected resource's side has exactly the same gap that PolarFS had before this fix: the old holder can keep writing after the coordinator has already decided it lost. It's worth checking, in any in-house system that shares storage across replicas, whether something equivalent to the autofencing described here exists, or whether mutual exclusion depends solely on the control layer's goodwill.