Postgres recovery_prefetch parameter can lock synchronous replicas for hours

A replication setting enabled by default in recent versions of PostgreSQL can turn sequential writes into random reads and lock up an entire cluster with synchronous replication, according to an investigation published by Jeremy Schneider on October 20.

Jeremy Schneider, who writes the Ardent Performance Computing blog, published a technical report on October 20 linking a seemingly harmless PostgreSQL parameter to real production incidents: recovery_prefetch. The text is not speculation. Schneider describes how he consistently reproduced a complete write lockup that can last from 30 minutes to several hours in clusters with synchronous replication.

The problem remained invisible for years because each isolated symptom seemed too small to investigate thoroughly. Putting the pieces together, Schneider shows that the combination of three factors common in production (Full Page Writes enabled, synchronous replication, and a prefetch parameter active by default) forms a recipe for an outage.

What recovery_prefetch does and why it exists

recovery_prefetch was introduced in PostgreSQL to speed up WAL replay on standbys. The idea is simple: before applying a WAL record that requires reading a block from disk, the replay process issues a posix_fadvise call for that block in advance, taking advantage of I/O time in parallel instead of blocking serially on every read.

By default, the feature is enabled whenever the operating system supports fadvise. For replicas that need to catch up to the primary after falling behind, this reduces catch-up time. So far, the rationale is defensible: less delayed replay time means less lag visible to whoever monitors the replica.

Where prefetch breaks: FPI and IOPS amplification

The first sign of trouble came from within the community itself, even before the feature was merged. In February 2021, Stephen Frost raised a concern on the Postgres hackers mailing list about prefetch behavior when Full Page Writes (FPW) are enabled, which today is the vast majority of production installations, since it's the only native Postgres protection against torn writes on operating system restarts.

Issuing unnecessary prefetches for blocks we've already sent a prefetch for is arguably a concern even if FPWs are off but the benefit of doing the prefetching almost certainly will outweight that and mean that finding a way to address it is something we could certainly do later as a future improvement. I wouldn't have any issue with that. Just doesn't seem as clear-cut to me when thinking about the FPW-enabled case. Ultimately, if you, Andres and Munro are all not concerned about it and no one else speaks up then I'm not going to pitch a fuss over it being committed, but, as you said above, it seemed like a good point to raise for everyone to consider.

Stephen Frost

The concern was noted, but the feature moved forward without changes on this point. Four years later, in March 2025, Michael Zhilin, from Postgres Pro, reported in another thread a real customer case: a physical standby on powerful hardware accumulating hours of replay lag, with the startup process consuming CPU mostly in kernel space.

One customer complained that physical standby has significant replay lag (few hours) on modern powerful bare metal servers. According to output of "top", startup process and few kernel workers utilized CPU, mostly in kernel space. The incident has been resolved by disabling recovery_prefetch after quick analysis of perf and strace outputs. Actual XLog prefetching code doesn't consider that shared buffer may be loaded from WAL FPI rather than from disk and makes fadvise calls in such cases.

Michael Zhilin, Postgres Pro

Schneider argues that both reports captured real symptoms, but not the complete cause. The third element, which he says had not been diagnosed before, is that prefetch also turns large sequential I/O on the primary into 8 KB random reads on the standby. For the same number of bytes read, this can amplify IOPS consumption by up to 16 times on the replica, according to the numbers presented in the report.

Why this becomes an outage, not just slowness

The devastating effect appears when this IOPS pattern meets synchronous replication. With synchronous_commit depending on at least one healthy standby, the primary doesn't confirm any transaction without remote confirmation. It's the price of guaranteeing durability without relying on proprietary storage replication.

The core of the behavior lies in the state machine defined in src/backend/access/transam/xlogrecovery.c, responsible for deciding where the standby reads the next WAL from:

/*-------
 * Standby mode is implemented by a state machine:
 *
 * 1. Read from either archive or pg_wal (XLOG_FROM_ARCHIVE), or just
 * pg_wal (XLOG_FROM_PG_WAL)
 * 2. Check for promotion trigger request
 * 3. Read from primary server via walreceiver (XLOG_FROM_STREAM)
 * 4. Rescan timelines
 * 5. Sleep wal_retrieve_retry_interval milliseconds, and loop back to 1.
 * Failure to read from the current source advances the state machine to
 * the next state.
 */

When walreceiver drops, due to network instability, for example, the standby switches to local WAL restore mode and only resumes accepting streaming after fully catching up on the lag. If, in the meantime, a write burst (a mass UPDATE, a heavy VACUUM) pushed amplified IOPS to the replica, the time for it to catch up to the primary can stretch to hours, with the primary blocked the whole time.

The reproduction: from 5 TPS to zero in a few minutes

Schneider published a reproduction script in the ardentperf-labs/walreceiver-outage-repro repository on GitHub. The steps described in the report are:

1. Spin up Postgres with synchronous replication to a standby, on a small server with cheap cloud disk (500 IOPS, 2 CPUs, 8 GB memory).
2. Initialize pgbench with scale 640, about 10 GB of data.
3. Disable autovacuum on pgbench_accounts so it doesn't interfere with the experiment.
4. Run pgbench at a fixed 5 TPS in a visible terminal.
5. Update 5% of the table (UPDATE pgbench_accounts WHERE aid % 20 = 0), generating a write burst; wait for it to finish.
6. Kill the walreceiver process to simulate a failure or network instability.

With recovery_prefetch at default, TPS drops to zero and writes remain locked for more than 30 minutes, potentially reaching hours with a larger table. Disk graphs show the primary at around 300 IOPS during the burst, while the standby stays pinned at the 500 IOPS ceiling for 40 minutes processing the same volume of bytes.

With recovery_prefetch = off (and FPW still enabled), the same test shows very different behavior:

Schneider sums up the practical conclusion: "Right now, I'm leaning toward disabling the recovery_prefetch setting across the board, but I want to see where the discussions go," meaning he himself is already leaning toward recommending disabling the parameter across the board, but is waiting to see how the community discussion unfolds before making a definitive recommendation.

How to diagnose this in a real environment

For those who administer Postgres with synchronous replication, it's worth putting together an observation checklist before waiting for an incident:

• Correlate replay_lag spikes in pg_stat_replication with windows of mass UPDATE/DELETE or heavy VACUUM runs.
• Run iostat on the standby during those windows and check whether the disk device hits the contracted IOPS ceiling, even with a byte volume comparable to the primary's.
• Use top and strace on the standby's startup process when there is persistent lag, replicating the diagnosis Michael Zhilin applied in the Postgres Pro case: CPU time concentrated in kernel space is a sign of excessive fadvise calls.
• Deliberately test a walreceiver drop in a staging environment with a realistic write load, using the script published by Schneider as a starting point.

What to do and what it costs

The most direct mitigation is to disable recovery_prefetch (recovery_prefetch = off) on instances running synchronous replication over storage with limited IOPS, a common case in Kubernetes with cloud disks. The trade-off is losing part of the feature's original benefit: standbys that fall behind for other reasons (maintenance, slow network) will take longer to catch up, because they go back to depending on serial reads without prefetch.

Schneider also suggests revisiting the replication topology instead of just tweaking the GUC. Running with two standbys and quorum commit (ANY 1) instead of depending on a single synchronous standby reduces the chance of the primary getting completely blocked when a walreceiver drops: it only takes one of the two to reconnect in time. The cost is operating with three replicas in total instead of two, which weighs on the infrastructure budget.

While the Postgres community hasn't settled on a definitive path forward for prefetch behavior, the decision falls to each team: measure your own write pattern, check the IOPS ceiling of the standbys' storage, and decide whether the catch-up gain from recovery_prefetch is worth the risk of an hours-long lockup on the next write burst combined with a slow network reconnection.