This test is not the automatic check: what it adds
The guide on the 3-2-1 strategy with Restic and S3 describes an automatic check: a sentinel file restored every week, which raises an alert when the backup chain breaks. It is useful and worth keeping. But it answers "can the repository be read?", not "does the client's site come back, and how fast?". Restoring a file of a few bytes tells you nothing about a database, uploaded files, the expected PHP version or missing secrets.
The exercise in this guide restores an entire site, by hand, on a new machine, and produces three measurements: a duration, the age of the data and a list of anomalies. The two mechanisms do not compete. The automatic check detects the failure every week; the quarterly exercise proves the recovery, with a signature and a date. That is the proof a client, a tender or an audit ends up asking for.
Nor should it be confused with the restore check in the monthly security audit checklist for a client VPS: that one verifies each month that a file or a database restores; the quarterly exercise in this guide brings a whole site back up, stopwatch in hand.
What the exercise proves that the backup alone does not
- The repository can be read without the original server — the restore starts from the remote repository, with secrets kept elsewhere, never from the site's disk.
- The site really starts again — the home page, the admin login and the latest content in the database respond on the new machine.
- The RTO is measured, not assumed — an observed duration replaces the estimate "about an hour".
- The RPO keeps its promise — the gap between the simulated incident and the restored snapshot is compared with what the contract states.
- The access sits in the right hands — the test reveals who really holds the repository password and keys.
- A proof exists — a dated, signed log handed to the client works as a sales argument and as protection in a dispute.
Preparing the test: scope, disposable machine, effort
Pick one site per client per quarter, alternating profiles: a site backed by a database, another with large uploaded files, a third with scheduled tasks. Before you start, gather three things outside the site's server: access to the repository, the password or key that opens it, and the list of the application's secrets.
Then rent a disposable test VPS with the same system as production, and plan to destroy it at the end of the session. For the effort, count as an editorial order of magnitude half a day per quarter for a typical site, more for a large repository, less once the procedure is routine. Write down the start time before the first command, never after.
The procedure in nine steps
Set the simulated incident
Write down the exact time of the simulated incident, for example the last known write on the site, and the scope to restore: files, database, configuration. This time feeds the RPO calculation.
Create the test VPS
Create a new machine with the same operating system as production. No site file reaches it from the original disk. Write down the start time: it starts the RTO stopwatch.
Install the tool and open the repository
Install only the backup tool and the site's stack (web server, PHP, database). Fetch the credentials from your password vault, never from the failed server.
restic -r <repository> snapshotsmust list the restore points.Choose the snapshot
Pick the last snapshot before the simulated incident and note its identifier and timestamp. It fixes the observed RPO.
Restore the files
restic -r <repository> restore <id> --target /srv/restore
Then move the content to the web directory. With Borg, first move into the target folder, because extraction writes to the current directory:cd /srv/restore && borg extract <repository>::<archive>.Restore the database
pg_restore --create --dbname=postgres backup.dump
For a plain-text dump,psql -f backup.sql. Then count the rows of a witness table (SELECT count(*) FROM orders;) and compare with the value read in production before the test.Rebuild the configuration
Put back the
.envfile, the secrets, the vhost and the scheduled tasks from the vault, then align the PHP and database versions with production.Check the application
Open the home page, log in to the admin area and find the latest order or article in the database. Without touching DNS:
curl -k --resolve your-domain.com:443:<TEST_VPS_IP> https://your-domain.com/.Stop the stopwatch and destroy the machine
Stop at the first successful application request, not when the restore command finishes. Calculate the RTO and the RPO, fill in the log, then destroy the test VPS.
The stopwatch: observed RTO and RPO
The observed RTO is the time the verified restore ends minus the time the test started. The observed RPO is the gap between the time of the simulated incident and the timestamp of the restored snapshot. Take a fictional example: simulated incident at 14:00, last snapshot taken at 03:00 the same day, so 11 hours of data lost. If the contract promises 24 hours, it complies; if it promises 4, it does not, and the log must say so as it is.
An RTO measured in a test remains an optimistic floor: new machine, nobody waiting, no pressure. It still beats a back-of-the-envelope calculation, especially with a large repository: on a modest connection, download time alone makes a theoretical RTO unrealistic. Measure the real throughput during the restore and note it in the log.
The test log: fields to fill in
Scroll the table
| Field | What to record | Example |
|---|---|---|
| Date and site | Day of the test and the client site restored | DD/MM/YYYY, demo site |
| Tool and snapshot | Backup tool, snapshot identifier and timestamp | Restic, short identifier, full timestamp |
| Duration and RTO | Start time, verified end time, observed duration | Start 09:00, end 10:40, RTO 1 h 40 |
| Observed RPO | Gap between simulated incident and snapshot, compared with the target | 11 h observed for 24 h promised |
| Anomalies | Everything that blocked or slowed the recovery | Uploads folder missing from the scope |
| Fix and deadline | Action, owner and completion date | Add the folder to the repository before the next test |
| Signatory | Person who attests to the test | Name, role, signature |
| Out of scope | What was not tested | Email, DNS, certificates, third-party services |
Reference commands by tool
Scroll the table
| Tool | List restore points | Restore and check |
|---|---|---|
| Restic | `restic -r <repository> snapshots` | `restic -r <repository> restore <id> --target /srv/restore` |
| Restic, repository integrity | `restic -r <repository> check` | `check` verifies the structure; `--read-data` rereads the whole repository, `--read-data-subset=10%` a random share |
| Borg | `borg list <repository>`, then `borg list <repository>::<archive>` | `borg extract --list <repository>::<archive>` from the target folder; `--dry-run` reads and verifies without writing |
| PostgreSQL, custom dump | `pg_restore --list backup.dump` | `pg_restore --create --dbname=postgres backup.dump`, then count a witness table |
| PostgreSQL, plain dump | Check the file size and date | `psql -f backup.sql`, then count a witness table |
| Uploaded files | Compare the file count in the repository with production | `find /srv/restore -type f | wc -l`, then open three files at random |
Restore pitfalls, with their symptom
- Repository key stored on the failed server — the new machine cannot open the repository and shows
wrong password or no key found; keep the key in a separate vault, with two holders. - Database restored, uploaded files missing — the site displays but images and attachments return 404; the scope covered only the database or only the code.
- Different PHP or database version — blank page, syntax or migration error; record each site's versions in the log.
- Secrets and
.envnot backed up — the application starts but no longer reaches the database, the mail server or third-party APIs. - Repository too large for the bandwidth — the theoretical RTO looks fine, the real RTO exceeds the promise; measure throughput before promising.
- Truncated dump — the row count of the witness table does not match production; rerun the backup and check its size.
- Scheduled tasks not recreated — the site responds but no longer sends emails or reminders; inventory the cron jobs before the test.
Troubleshooting: errors seen during an exercise
Fatal: wrong password or no key found — Restic finds no key that opens with the password supplied. Frequent causes to rule out one by one: a password with special characters altered by the shell, a password read from standard input, an S3 repository URL in the wrong style, a damaged key file. Try --password-file or the RESTIC_PASSWORD variable, check the repository URL, then restic key list. If the password is truly lost, the repository data is lost with it: record it as a critical anomaly.
No recent restore point — restic snapshots lists nothing from the past week: the backup job stops silently. Fix the timer and add an alert.
pg_restore ends with ignored errors — by default it continues and prints a count at the end. Rerun with --exit-on-error to stop at the first error, and read it before declaring the test a success.
The files are there but the site returns 403 — restoring as root left the files with the wrong owner. Give them back to the web server user.
The repository password never lives alone on the server
Keep it in a team vault, with a sealed copy held by a second person in charge. The exercise starts by opening the repository from that vault: if someone has to log in to the site's server to find the password, the test has already failed, and that is the most useful anomaly it can reveal.
The log handed to the client: one page, and what was not tested
The log fits on one page: a header with client, site and date; the table of eight fields; a one-sentence conclusion, compliant or not with the RPO and RTO targets; a dated corrective plan; a signature. Reserve a final box "Not tested in this exercise": mailboxes and deliverability, DNS and delegation, TLS certificates, third-party services such as payment or a CRM, and any other site on the account.
A log that does not state its limits reads as "everything is backed up", and that is a promise nobody can keep. Keep the logs of every quarter: one test proves one day, a series proves a practice. Also record each site's PHP version, as in the guide on the version inventory of a client portfolio: it is the most frequent trap when you replay a restore.
On the contract side: what the backup clause must name
A useful clause names seven points: the scope (files, databases, configuration), the frequency, the retention, the RPO and RTO targets, the frequency and proof of tests, the liability if the restore fails, and the person who holds the repository access. This is not legal advice: have the text reviewed by a lawyer. And never promise what the underlying infrastructure does not guarantee.
On that point, ServOrbit.com backups are weekly on the entry level, daily on Pro and on all reseller hosting offers. The floor RPO follows from that frequency: up to seven days with a weekly backup, up to one day with a daily one. An RPO under one day therefore requires the agency's own application backup, in addition. As for restore time, no commitment follows from a frequency: it is measured, with the exercise above.
Making the test billable: a selling point, not a burden
The objection is familiar: the client does not ask for it, and the time is not billed. Two answers. Build a line "quarterly restore test, log delivered" into the maintenance plan, or bill the half day separately. And treat the log as a deliverable: after an incident it shows what had been tested and fixed; during a sale it answers "prove it" before the question is asked.
To keep the pace across a portfolio, plan a rotation: one site per client per quarter, with the calendar set at the start of the year. The test VPS is rented and destroyed on demand; the terms are on the offer page.
