Metrics
1.5 daysto full recovery0data lostThank-youletter from the customer
Context
An ERP upgrade corrupted HR data at a Saudi enterprise, and the backup turned out to be unusable.
Impact
Payroll and HR records for the whole organisation were at risk. There was no clean restore point.
Timeline
- Corruption detected after the upgrade; backup restore fails validation.
- Response split into three parallel streams with named owners.
- Transaction-log recovery path proves viable; the other streams are kept warm.
- Data recovered and reconciled; service handed back to the business.
Actions
- Split the response into three parallel recovery streams: backup restore, vendor escalation, and transaction-log recovery
- Ran timed executive updates so leadership never had to ask for status
- Kept every stream alive until one was proven, not just promising
- Reconciled the recovered data before handing the system back
Root cause
The upgrade corrupted HR tables, and the backup that should have been the safety net had never been restore-tested.
Outcome
Recovered through the transaction logs with zero data lost in 1.5 days. The customer sent a thank-you letter.
Learned
“A backup you haven't restored is a hope, not a backup.”