Green backup doesn't guarantee restoration: the test most PostgreSQL teams never ran

A backup job that finishes without error proves far less than it seems. The only thing that actually proves recovery is restoring, bringing up the cluster, and validating the data at a point in the past.

A backup job that finishes without error proves far less than it seems. The only thing that actually proves recovery is restoring, bringing up the cluster, and validating the data at a point in the past.

I've lost count of how many teams I've found with an impeccable backup dashboard and no idea how long it would take to get back online after a real incident. That gap between "the job ran" and "I can actually restore" is, in my experience as a DBA, the most expensive blind spot in database operations. And it's rare for anyone to measure it before the day they actually need to.

Stormatics, a PostgreSQL-focused consultancy, recently published an account by Semab Tariq that describes exactly this pattern with uncomfortable precision. I agree with almost the entire argument, and that's why I want to take it further: it's not enough to say that a green backup isn't a restore, we need to explain why even mature teams, with automated verification tools, still fall into this trap.

An incident that serves as a portrait, not an exception

Tariq's piece lays out a timeline that he himself describes as a recurring pattern across clients, not the case of one specific company. It's worth reproducing because it's instructive.

• 09:38: a new feature drops a column that still held live data.
• 09:44: support tickets start coming in reporting incomplete records.
• 10:05: after trying point fixes, the team decides to restore to the moment before the incident.
• 10:20: they discover there's no restore host ready. No one had ever needed one.
• 13:50: the restore finishes, but recovery stops before the expected point because a WAL segment is missing from the archive.
• 16:30: they give up on the exact point, recover a copy from six days earlier, and spend two days manually rebuilding the difference.

The business expected one hour of downtime and zero data loss. It took more than two days, with guaranteed loss. The backup, over the previous fourteen months, had always been green.

What a successful backup actually proves

Here's the core of the argument, and it's where I agree without reservation with Tariq: a green backup job is a much smaller claim than it seems. It proves that a process started and finished without error, that bytes were read from one place and written to another, that the backup catalog gained a new entry.

None of those things prove that the restore actually works. A green job doesn't say whether all the files outside the data directory (roles, certificates, extensions, configuration files) are in fact preserved, whether the WAL needed to bring that backup to a consistent state still exists, whether the pages inside the backup are healthy, or whether you can stop at a specific point in the past. And it says nothing about whether all of this fits within the time you promised the business.

A backup is only useful if you know you can actually restore it when you need it.

Semab Tariq, Stormatics

That sentence should be stuck to every DBA's screen. Not because it's some theoretical novelty, but because it's exactly the kind of thing everyone knows and almost no one practices.

The corruption that the backup copies perfectly

The example that stuck with me most in the piece is silent corruption. A cluster created years ago with data_checksums disabled migrates across storage generations. A report that runs only a few times a year starts failing, because a data page has been damaged for months, in a part of the table that normal usage never touches.

When checking the backups, the problem becomes clear: all of them copied the damaged page perfectly. The oldest backup still within the retention window already contains the corruption. There's no clean copy left to go back to.

This completely changes how I read a retention policy. Thirty days of retention sound like thirty days of protection, but against silent corruption it's the opposite: it's thirty days to notice the problem, after which the last clean copy has already expired and the damage is present in every copy you keep. It's a strong argument for enabling data_checksums from the moment the cluster is created, and for periodic integrity audits, not just periodic backups.

The archiving gap nobody monitors

The second scenario is even more insidious because it fails without triggering any obvious alarm. WAL archiving to object storage is configured and working, until a credential is rotated and archiving starts failing.

It fails silently: a broken archiving WAL doesn't bring down the database or the backup job. PostgreSQL simply retains the segments and tries again, and the first visible symptom is usually disk pressure in pg_wal, whose natural reaction is to free up space, which turns a temporary problem into a permanent one. The alert was watching the backup job; archiving is a different mechanism, with a different failure mode, and nobody was keeping an eye on it.

Tariq's practical recommendation is to monitor three things separately: the count of archiving failures, the age of the most recently successfully archived segment, and the size of pg_wal. I fully agree: treating archiving and backup as the same checklist item is the most common observability configuration mistake I see.

"But my tools already check everything": the counterargument that deserves an answer

Here comes the most reasonable objection, and it's fair to address it. Modern PostgreSQL backup tools usually bring verification layers beyond the simple "the job finished without error," and a team that operates disciplined using those layers can argue that manual restore testing is redundant.

The counterargument is valid up to a point, but not all the way. Integrity verification of backup files proves that the written bytes didn't get corrupted in transit or at rest; it doesn't prove that PITR stops at the right second, that the cluster comes up with the right roles and extensions, that the application reads consistent data after replay, or that the team has somewhere to restore to within the promised RTO. These are different layers of guarantee, and automation solves the first one without touching the other three.

In short: a mature backup tool reduces the risk of corruption in the archive, but it doesn't replace the operational proof that someone, with a clean machine and a reference timestamp, can bring the database back within the time agreed with the business.

What I'd demand from any team before sleeping soundly

If I were auditing a PostgreSQL environment tomorrow, the list would be short, and I'd insist on it before accepting any other improvement item:

1. Confirm that WAL archive monitoring is separate from backup job monitoring.
2. Check whether the retention window covers backup and WAL together, not just one of the two.
3. Document everything that doesn't live in the data directory (roles, certificates, repository encryption keys) and where it's stored.
4. Pick a database and restore it to a timestamp from last week, on a clean machine, with someone other than whoever built the routine at the keyboard.

The goal isn't to feel confident about the backup. It's to be able to show someone why that confidence exists, and to know the date it was last proven. A green backup without a tested restore isn't security: it's an assumption that hasn't been called in yet.