# A backup you haven't restored is a hope, not a backup

> What recovering a corrupted HR system with zero data loss in a day and a half taught me — and the HA/DR checklist I now use for every design.

- Author: Maged Zamzam (https://magedzamzam.ae/profile)
- Published: 2026-10-13
- Topics: HA/DR, Resilience, Backup, Incident response, Postmortem
- Canonical: https://magedzamzam.ae/notes/a-backup-you-havent-restored-is-a-hope

---
Every HA/DR design I review now starts with one question: *when did you last restore this?* Here is the incident that taught me to ask it.

## What happened

An ERP upgrade at an enterprise customer corrupted their HR data. That alone is recoverable — it's what backups are for. Then the backup failed validation on restore. Payroll and HR records for the whole organisation were at risk, and there was no clean restore point.

## How we recovered

We stopped looking for a single fix and split the response into **three parallel streams, each with a named owner**:

1. **Backup restore** — in case a partial or older restore could still be salvaged.
2. **Vendor escalation** — getting the product's own engineers working on the corruption.
3. **Transaction-log recovery** — rebuilding the data from the database's own logs.

Leadership got updates **on a fixed clock**, whether or not there was news. Nobody had to ask for status, which kept the technical teams focused.

The transaction-log path worked. We recovered everything — **zero data lost, in a day and a half** — and reconciled the data before handing the system back. The customer later sent a thank-you letter, which in our line of work is rarer than an outage.

## The root cause

It wasn't really the upgrade. Upgrades fail; that's normal. The root cause was a backup that **had never been restore-tested**. It existed, it ran every night, it showed green — and it was useless on the one day it mattered.

## The checklist I use now

- **Restore, don't just back up.** Schedule restore tests into an isolated environment, and time them. The restore time *is* your real RTO.
- **Agree RPO and RTO per system, in writing,** with the business owner — not just the infrastructure team.
- **Keep the logs.** Transaction, redo and archive logs, retained and shipped off the host. They are your second line of defence.
- **Recover in parallel streams with owners.** Don't run your hopes one after another.
- **Communicate on a clock.** Every two hours, even when the update is "no change".
- **Rehearse failover.** A DR site that has never been failed over to is a theory, not a capability.
- **Reconcile before you declare victory.** "The database is up" and "the data is right" are different statements.

## In the cloud this is easier — and easier to skip

Managed backups, point-in-time recovery and cross-region replication remove a lot of the heavy lifting. They also make it very easy to assume the problem is solved. Defaults are not a strategy:

- Test restores of managed backups too.
- Protect backups from the same credentials that run production — ransomware goes after backups first.
- Keep at least one copy in another region, and know how long it takes to bring it back.

## The one-line version

> A backup you haven't restored is a hope, not a backup.

---

Source: https://magedzamzam.ae/notes/a-backup-you-havent-restored-is-a-hope. Views are the author's own.
