/notes / 13 Oct 2026

A backup you haven't restored is a hope, not a backup

What recovering a corrupted HR system with zero data loss in a day and a half taught me — and the HA/DR checklist I now use for every design.

· 2 min read

A dark data centre with one open server rack and a glowing cyan progress bar of a data restore in progress

Every HA/DR design I review now starts with one question: when did you last restore this? Here is the incident that taught me to ask it.

What happened

An ERP upgrade at an enterprise customer corrupted their HR data. That alone is recoverable — it's what backups are for. Then the backup failed validation on restore. Payroll and HR records for the whole organisation were at risk, and there was no clean restore point.

How we recovered

We stopped looking for a single fix and split the response into three parallel streams, each with a named owner:

  1. Backup restore — in case a partial or older restore could still be salvaged.
  2. Vendor escalation — getting the product's own engineers working on the corruption.
  3. Transaction-log recovery — rebuilding the data from the database's own logs.

Leadership got updates on a fixed clock, whether or not there was news. Nobody had to ask for status, which kept the technical teams focused.

The transaction-log path worked. We recovered everything — zero data lost, in a day and a half — and reconciled the data before handing the system back. The customer later sent a thank-you letter, which in our line of work is rarer than an outage.

The root cause

It wasn't really the upgrade. Upgrades fail; that's normal. The root cause was a backup that had never been restore-tested. It existed, it ran every night, it showed green — and it was useless on the one day it mattered.

The checklist I use now

  • Restore, don't just back up. Schedule restore tests into an isolated environment, and time them. The restore time is your real RTO.
  • Agree RPO and RTO per system, in writing, with the business owner — not just the infrastructure team.
  • Keep the logs. Transaction, redo and archive logs, retained and shipped off the host. They are your second line of defence.
  • Recover in parallel streams with owners. Don't run your hopes one after another.
  • Communicate on a clock. Every two hours, even when the update is "no change".
  • Rehearse failover. A DR site that has never been failed over to is a theory, not a capability.
  • Reconcile before you declare victory. "The database is up" and "the data is right" are different statements.

In the cloud this is easier — and easier to skip

Managed backups, point-in-time recovery and cross-region replication remove a lot of the heavy lifting. They also make it very easy to assume the problem is solved. Defaults are not a strategy:

  • Test restores of managed backups too.
  • Protect backups from the same credentials that run production — ransomware goes after backups first.
  • Keep at least one copy in another region, and know how long it takes to bring it back.

The one-line version

A backup you haven't restored is a hope, not a backup.

HA/DRResilienceBackupIncident responsePostmortem

More notes