Why hosting snapshots are not enough: three concrete limits
A hosting snapshot is convenient, but it shares the same failure plane as your VM. First limit: if the VM is deleted — by mistake, by your provider, or after a payment failure — the associated snapshots disappear with it. Second limit: if an attacker compromises your hosting account, they can delete snapshots and VM in seconds via the API or dashboard. Third limit: datacenter snapshots never test the restore. A 'consistent' snapshot can contain a corrupt filesystem or a database in an inconsistent state; you will only find out when you need it.
What a 3-2-1 strategy guarantees, and what a snapshot does not
- Three copies: the original on local disk, one on hosting snapshots, one off-site on S3 — no single point of failure can erase everything.
- Two distinct media: your VPS NVMe disk and an object storage bucket at another provider are physically and logically separated.
- Encryption before transfer: Restic encrypts client-side with AES-256; the S3 storage provider sees only opaque blobs, unreadable without your key.
- Restore testing: an untested backup is a promise, not a guarantee. A systemd timer can automatically verify that a sentinel file can be restored.
- Provider independence: your B2 or R2 bucket cannot be deleted from your VPS hosting provider's dashboard.
- A hosting snapshot does not encrypt before transfer: data is readable by the provider.
- A hosting snapshot never verifies the application-level consistency of your databases.
The three copies explained: local, hosting, off-site
The 3-2-1 rule is not a rigid recipe — it is a risk diversification principle.
Copy 1 — your VPS local disk. This is your first line: the original, on NVMe. It serves fast restores of a file accidentally deleted, with no network latency.
Copy 2 — hosting snapshots. The automatic backup managed by your VPS provider covers configuration accidents and accidental deletions at the system level. It is your on-site safety net.
Copy 3 — off-site object storage, encrypted with Restic. This is the link that most teams forget, and the only one that survives the complete loss of the hosting account. Restic encrypts data before any transfer, deduplicates it to minimise cost and sends it to an S3-compatible bucket at a third-party provider. This third copy is what this guide builds.
Backblaze B2 vs Cloudflare R2: choosing your S3 backend
| Criterion | Backblaze B2 | Cloudflare R2 |
|---|---|---|
| Storage | Public per-GB rate | Free up to 10 GB/month, then public per-GB rate |
| Egress (outbound traffic) | Free to Cloudflare and select CDN partners; billed outside partners | Free without limit — no egress cost |
| S3 compatibility | Native S3-compatible API; endpoint `s3.us-west-004.backblazeb2.com` | Native S3-compatible API; endpoint `<account>.r2.cloudflarestorage.com` |
| `RESTIC_REPOSITORY` variable | `s3:https://s3.us-west-004.backblazeb2.com/bucket-name` | `s3:https://<account>.r2.cloudflarestorage.com/bucket-name` |
| Access keys | B2 Application Key (Key ID + Application Key) | R2 API Token with Object Read & Write permissions |
| Recommended use case | High volume with limited egress or from Cloudflare infrastructure | Frequent egress or regular restore tests from anywhere |
Complete implementation: Restic + S3 on your VPS
Export environment variables to a secured file
Create /root/.restic-env with the required variables. For Backblaze B2:
export RESTIC_REPOSITORY="s3:https://s3.us-west-004.backblazeb2.com/your-bucket"
export RESTIC_PASSWORD="your-repository-password"
export AWS_ACCESS_KEY_ID="your-b2-key-id"
export AWS_SECRET_ACCESS_KEY="your-b2-application-key"For Cloudflare R2, replace the repository URL and keys:
export RESTIC_REPOSITORY="s3:https://<account-id>.r2.cloudflarestorage.com/your-bucket"
export RESTIC_PASSWORD="your-repository-password"
export AWS_ACCESS_KEY_ID="your-r2-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-r2-secret-access-key"Immediately restrict permissions: chmod 600 /root/.restic-env. This file must never be committed to a Git repository or included in a backup accessible without authentication.
Initialise the Restic repository on S3
Load the environment, then initialise the encrypted repository on your bucket:
source /root/.restic-env
restic initRestic creates the repository structure and seals the encryption with your password. The operation outputs a repository ID — note it. Store the repository password outside the VPS: in a password manager, an encrypted vault on a separate machine, or a secret manager. Without it, no restore is possible, even if you have full access to the bucket.
Run a first backup to S3
Test the complete path with a manual backup:
source /root/.restic-env
restic backup /etc /var/www /opt/docker-dataFor databases, generate a dump first or use stdin mode. PostgreSQL example:
pg_dump -U postgres mydb | restic backup --stdin --stdin-filename mydb.sqlVerify the snapshot was created: restic snapshots.
Define the retention policy
The forget command removes references to old snapshots; --prune physically frees orphaned blocks in the backend:
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --pruneRationale for the values. Seven daily snapshots cover a full week — enough to detect a silent corruption that only appears a few days after the incident. Four weekly snapshots offer a one-month window to detect a slow application-level issue. Three monthly snapshots allow a restore to a state prior to the current quarter — useful after a failed database migration. --prune is essential: without it, forget marks snapshots for deletion but does not free space in the bucket, and the storage bill continues to grow.
Distinguish restic check from restic check --read-data
These two commands do not verify the same thing.
restic check validates repository metadata — pack structure, index consistency, pointer integrity. Fast (seconds to a few minutes depending on repository size). Run it after every forget --prune.
restic checkrestic check --read-data downloads and verifies every data blob by comparing each to its cryptographic hash. Slow (proportional to repository volume) and potentially costly in egress if your provider bills outbound traffic. Reserve it for a monthly check or after doubts about bucket integrity.
restic check --read-dataA weekly metadata check is sufficient to catch common corruptions without transfer overhead.
Create the systemd backup unit (.service)
Create /etc/systemd/system/restic-backup.service:
[Unit]
Description=Restic backup to S3
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/root/.restic-env
ExecStartPre=/bin/sh -c 'curl -sf --max-time 10 https://one.one.one.one > /dev/null || exit 1'
ExecStart=/usr/bin/restic backup /etc /var/www /opt/docker-data
ExecStartPost=/usr/bin/restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
ExecStartPost=/usr/bin/restic check
StandardOutput=journal
StandardError=journalExecStartPre checks network connectivity before attempting the backup — if the VPS is isolated or the bucket is unreachable, the service fails cleanly with an actionable status rather than hanging or writing a silent error to the logs.
Create the systemd scheduling unit (.timer)
Create /etc/systemd/system/restic-backup.timer:
[Unit]
Description=Daily timer for restic-backup.service
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaySec=900
[Install]
WantedBy=timers.targetPersistent=true ensures that if the VPS is off at 03:00, the backup runs at the next boot. Enable and start:
systemctl daemon-reload
systemctl enable --now restic-backup.timerVerify the timer status: systemctl status restic-backup.timer and last run logs: journalctl -u restic-backup.service.
Create an automated restore test with a sentinel file
Create the sentinel file and include it in your backup:
echo "sentinel-$(date +%s)" > /opt/restic-sentinel.txtCreate /etc/systemd/system/restic-restore-test.service:
[Unit]
Description=Restic restore test
After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/root/.restic-env
ExecStart=/bin/sh -c '
rm -rf /tmp/restic-test-restore && \
restic restore latest --target /tmp/restic-test-restore && \
test -f /tmp/restic-test-restore/opt/restic-sentinel.txt && \
echo "Restore OK: $(cat /tmp/restic-test-restore/opt/restic-sentinel.txt)" || \
(echo "FAILED sentinel restore" && exit 1)
'
StandardOutput=journal
StandardError=journalCreate the weekly timer /etc/systemd/system/restic-restore-test.timer:
[Unit]
Description=Weekly Restic restore test
[Timer]
OnCalendar=Sun 04:00:00
Persistent=true
[Install]
WantedBy=timers.targetEnable: systemctl enable --now restic-restore-test.timer.
Restic secrets: isolated .env file, never hard-coded in the service
Never place RESTIC_PASSWORD, AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY directly in a systemd unit definition, a versioned script or a Dockerfile. The /root/.restic-env file with chmod 600 is the right approach: it is loaded by EnvironmentFile= without ever appearing in systemctl show or logs. Add /root/.restic-env to your .gitignore or .dockerignore if your /root directory is versioned. For multi-user environments, prefer a secret manager or systemd credentials (LoadCredential=) over a plain file.
Troubleshooting: four common errors
Fatal: unable to open config file on restic init or first restic backup. Restic cannot reach the bucket. Check that variables are loaded (echo $RESTIC_REPOSITORY) and credentials are correct. Verify IAM permissions on your B2 or R2 key: it needs at minimum read, write and list rights on the bucket. On B2, an Application Key restricted to a specific bucket must explicitly include listBuckets. On R2, the token must have Object Read and Object Write permissions on the target bucket.
S3 bucket: insufficient permissions. If restic init succeeds but restic backup fails with an authorisation error, a bucket policy is likely overriding the key's permissions. On B2, check that no denyUpload bucket rule is active. On R2, check that the bucket is not in public mode with write restrictions.
systemd timer not triggering. Diagnose with three commands: systemctl status restic-backup.timer, systemctl list-timers --all | grep restic, journalctl -u restic-backup.service --since today. If the timer is active but the service has not run, verify that OnCalendar is syntactically valid: systemd-analyze calendar '*-*-* 03:00:00' should return a next trigger date.
restic check --read-data too slow. On a repository of several tens of GB, --read-data can take hours and generate significant egress costs on B2. Use --read-data-subset=10% for a random sample check on each weekly run, and reserve the full verification for a scheduled monthly maintenance window.
Recovery plan: how long to restore from B2 or R2?
Restore time depends on three factors: data volume, available bandwidth between your VPS and the bucket, and any egress costs.
As an estimate: a 20 GB Restic repository on B2 (already deduplicated and compressed data) restores in roughly 15 to 30 minutes with a standard datacenter connection at 1 Gbps. On R2, egress is free — no financial pressure to stagger the restore. On B2, if your VPS is outside the Backblaze partner network, the first GB are free then billed; a full restore of a large repository may incur non-trivial costs.
Two practices reduce recovery time: first, maintain a list of your critical directories separate from cache or log directories (do not back up /tmp, /proc, unused Docker volumes) — a smaller repository restores faster. Second, regularly test the partial restore of a single directory (restic restore latest --target /tmp/test --include /etc) to calibrate the real duration on your infrastructure, rather than discovering it during an incident. restic stats gives the repository size for planning.