OS patching without downtime in PostgreSQL: why HA must cover the entire stack

A database went down because of a security patch, and PostgreSQL was never the target of that patch. It's with this concrete case that Umair Shahid opens his article on Planet PostgreSQL, published by Stormatics. The team needed to update the kernel, the kind of update that only takes effect after a reboot, and did what almost everyone does the first time: scheduled a maintenance window, took the database down, applied the patch, rebooted, and brought it back. A planned, clean outage for something that had nothing to do with the database.

Shahid's point is what matters for anyone operating PostgreSQL at scale: availability isn't a property of the database process, it's a property of the entire stack it runs on. And a large share of teams build HA to survive a Postgres crash, but never to survive the ground Postgres is standing on.

The database doesn't hold itself up

PostgreSQL runs on top of an operating system, which runs on top of a kernel. In most modern environments this also means a container runtime, a storage layer, a network layer, and, underneath everything, a hypervisor or physical host. Postgres sits at the top of that stack and trusts everything below it.

Each layer affects uptime, and each one ships patches on its own schedule:

• The kernel gets a security fix that requires a reboot.
• glibc gets an update.
• The container runtime changes version.
• The storage driver has a CVE.

None of these is a Postgres problem, and all of them can take Postgres down if the only failure you've protected against is the database process crashing. As Shahid sums it up, that's the trap:

Teams build high availability to survive a database crash, a node dying, a disk filling up. Then a routine glibc update comes along, someone reboots the machine, and the database goes down with it. The outage gets filed away as "planned maintenance," everyone moves on, and nobody notices that the availability design never covered the ground the database was standing on.

>

-- Umair Shahid, Stormatics

Why the maintenance window becomes the default

The maintenance window is an easy habit to fall into because it works, more or less. You announce it, absorb the impact, apply the patch, and life goes on. The model stops working once the business runs 24/7: a payments platform has no quiet hour, a bank has a batch close that can't slip, a SaaS with customers across every time zone has no 2 a.m. that's 2 a.m. for everyone.

Worse: in a regulated environment, OS patching isn't optional. When a kernel CVE appears, security and compliance want the fix applied immediately, not at the next window. That creates two forces pulling in opposite directions: patching fast to stay secure, and staying up to meet the SLA. The window model forces a choice. A cluster built for rolling operation removes the choice.

Rolling patching, one node at a time

In a well-built HA cluster, OS patching is a rolling operation: you update one node at a time while the database stays up the entire time, and Postgres itself is never touched. The approach Shahid proposes is straightforward:

1. Build a new node image with the OS already patched. Same Postgres version, same configuration, everything identical above the OS. The only change is the layer being patched.
2. Apply it to the replicas first. Stop Patroni on a replica, bring the node up with the new image, and let it rejoin the cluster and catch up on replication.
3. With all replicas patched and healthy, trigger a controlled switchover. The primary role moves to a node that's already patched.
4. Patch the former primary the same way. It comes back as a replica, already running the new image.

What nobody feels is an outage: the only perceived interruption is a switchover measured in seconds, at a moment you choose. Compare that to a full reboot of a single primary, where you're down for however long the machine takes to come back, at a time the patch schedule chose for you.

Notice what didn't happen: Postgres stayed untouched. Same binaries, same config, same data files. The entire operation swapped out the ground under the database. This isn't a database upgrade, it's infrastructure maintenance the database sails through cleanly.

What actually makes rolling patching safe

The four steps look easy on paper. What makes them safe in production is the operational rigor around them, and that's where the difference between a real HA cluster and a hopeful one shows up. Shahid lists five conditions:

• Connections need to follow the switchover. If applications connect to a fixed primary address, the switchover just moves the outage instead of removing it. You need a routing layer in front: a load balancer, a pooler like PgBouncer, or a virtual IP that tracks the leader. That way, when the primary role moves, new connections land on the new leader without anyone reconfiguring anything.
• Replication needs to be caught up before the switch. Promoting a lagging replica means switching over to a slow database, or worse. Check replication lag before promoting, and know whether you run synchronous or asynchronous replication, because that determines how much in-flight data a switchover can cost you.
• Health checks and quorum need to be honest. Patroni decides based on what the cluster reports about itself. If a node comes back with the new image but the health check is lying, the cluster will trust it anyway. Watch the rejoin, confirm the node is actually streaming and caught up, and keep enough healthy nodes to never lose quorum mid-patch.
• The node image needs to be reproducible. The whole model depends on the patched image being identical to the old one, except for the OS layer. If the images drift (a different extension version, a different config), you find out during a switchover, the worst possible moment. Always build images from the same definition.
• Test the switchover before you need it. A switchover you've never run is a plan, not a capability. Run a controlled one during a quiet period, see how the application handles it, measure the blip, and fix whatever surprises you. Then patch night is just a repeat of something you already know works.

Where it gets complicated, and when it doesn't hold

The author avoids selling rolling patching as perfectly invisible. A switchover is not zero impact: in-flight transactions on the old primary are dropped, and there's a short window, typically a few seconds, where writes pause while the role migrates. For most workloads that's a blip; for something extremely latency-sensitive, you plan around it. Either way, it's far better than a full outage.

Some underlying-layer patches call for more than a reboot: a storage layer migration or a change to the underlying host can involve moving data, not just cycling an image, and that requires its own plan. The rolling model still applies, but the steps get bigger.

And there's the honest prerequisite: all of this assumes there's a cluster. A single primary with no replica can't do any of this. If the database is one node, OS patching genuinely is a maintenance window, and the real fix is the HA design, not the patching procedure. As Shahid puts it, rolling patching is something you build over time, and it's one of the clearest reasons to set up real HA from the start.

The final takeaway is for any DBA who still treats OS patching as database maintenance: the last thing that took your database down probably wasn't the database, it was the ground it was standing on. Building the cluster for this once (reproducible images, leader-following routing, healthy replication, a tested switchover) takes that entire category of outage off the table.