Backups you have actually restored
For two years the nightly job reported success. The first real restore failed in the fourth minute: the archive was fine, the credentials to open it were stored on the machine that had just died.
The monthly drill
- Pick one service at random.
- Restore the latest backup to a scratch machine using only the written procedure.
- Start the service and run its smoke test.
- Write down every step that needed knowledge not present in the procedure.
Checklist
| Item | Kept where |
|---|---|
| Archive | Off-site object storage, versioned |
| Encryption key | Printed copy and a password manager |
| Procedure | This page and a copy next to the archive |
Two mistakes found only by restoring
Database dumps were taken without the roles they depend on, and file ownership was lost because the archive was created by an unprivileged user. Both are one-line fixes once seen.
Previous: A local DNS cache that does not lie.