Paj Digital Labs
All articles
Hosting & Server Infrastructure

Backups That Actually Restore: 3-2-1 With restic, rclone and Real Drills

We pair restic and rclone with automated restore drills to build a 3-2-1 backup pipeline that guarantees data actually recovers.

Paj Digital Labs 8 min read

Unverified backups are just expensive random number generators. We learned this three years ago when an incident required restoring an 800GB PostgreSQL cluster from what we thought was a healthy S3 bucket. The backup scripts had returned exit code 0 for eight months straight. What they hadn't reported was that a silent path truncation in the backup script was skipping the database dump directory entirely. We were backing up empty folders with perfect reliability.

Since that post-mortem, we treat backups as untrusted until proven otherwise by an automated restore drill. Our current infrastructure relies on a strict 3-2-1 strategy powered by restic for client-side encrypted, deduplicated snapshots, and rclone for repository replication across heterogeneous cloud targets.

Here is how we build, schedule, and continuously validate that pipeline.

The Architecture: 3-2-1 Without Enterprise Bloat

The 3-2-1 rule demands three copies of data, across two different media types, with one copy stored offsite. Many teams overcomplicate this with proprietary agents that break on kernel updates. We use two open-source binaries that run anywhere: restic handles local snapshotting and encryption; rclone handles object store transport.

text
+---------------------+
| Production Server   |
| (Source Data)       |
+----------+----------+
           |
           | 1. restic backup
           v
+---------------------+
| Local ZFS Volume    |  <-- Copy 1 (Fast local restore target)
| /var/backups/restic |
+----------+----------+
           |
           | 2. rclone sync
           +-----------------------+
           |                       |
           v                       v
+---------------------+ +---------------------+
| Hetzner Storage Box | | Backblaze B2 S3     |  <-- Copies 2 & 3 (Offsite/Cloud)
| (SFTP / Append-Only)| | (Object Lock API)   |
+---------------------+ +---------------------+

1. Local Repository (Copy 1): restic backs up production state directly to an isolated local NVMe or ZFS mount. This gives us sub-minute restore speeds for accidental file deletions or operator error. 2. Offsite Primary (Copy 2): An append-only SFTP mount on a remote server (Hetzner Storage Box). If the local node is compromised, an attacker with local root cannot delete historical remote snapshots because the remote user only has write/append permissions. 3. Offsite Secondary (Copy 3): An object storage bucket (Backblaze B2 or AWS S3) synchronized via rclone with Object Lock enabled in compliance mode.

Step 1: Initializing the Restic Repository

restic encrypts everything by default using AES-256 in counter mode and Poly1305 for authentication. Never pass passwords on the command line; use a secure file readable only by root with 0400 permissions.

We start by initializing the local repository:

bash
class="text-code-comment">#!/usr/bin/env bash
set -euo pipefail

export RESTIC_REPOSITORY=class="text-code-string">"/var/backups/restic-repo"
export RESTIC_PASSWORD_FILE=class="text-code-string">"/etc/restic/repo-password.key"
class="text-code-comment">
# Initialize local repository with AES-256 encryption
if [ ! -f class="text-code-string">"${RESTIC_REPOSITORY}/config" ]; then
    restic init
fi

To take a consistent backup of filesystem state, system configurations, and application mounts, we use explicit exclude patterns to eliminate socket files, transient caches, and dynamic logs:

bash
class="text-code-comment">#!/usr/bin/env bash
set -euo pipefail

export RESTIC_REPOSITORY=class="text-code-string">"/var/backups/restic-repo"
export RESTIC_PASSWORD_FILE=class="text-code-string">"/etc/restic/repo-password.key"
class="text-code-comment">
# Read paths to exclude
cat << class="text-code-string">'EOF' > /etc/restic/excludes.txt
/dev/*
/proc/*
/sys/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/cache/*
/var/tmp/*
/var/lib/docker/*
*.log
EOF
class="text-code-comment">
# Perform the backup with cgroup I/O limits
nice -n 19 ionice -c 3 restic backup \
    --one-file-system \
    --exclude-file=/etc/restic/excludes.txt \
    --tag class="text-code-string">"scheduled-daily" \
    / /etc /var/www /var/vmail

Using --one-file-system prevents restic from crossing filesystem boundaries into network mounts or pseudo-filesystems like /proc.

Step 2: Offsite Transport with Rclone

restic has native S3 and SFTP backends, but using rclone to mirror the local repository files gives us finer control over throughput, retry logic, and bandwidth shaping. Because restic stores data in immutable content-addressable pack files, an rclone sync operation on a local repo is lightweight and safe.

Our /etc/rclone.conf configures two remote targets:

ini
[hetzner-sftp]
type = sftp
host = u123456.your-storagebox.de
user = u123456
port = 23
key_file = /etc/restic/id_ed25519_backup
md5sum_command = none
sha1sum_command = none

[backblaze-b2]
type = b2
account = 002a1b2c3d4e5f60000000001
key = K002abcdef1234567890abcdef123456
hard_delete = false

We sync the repository offsite immediately after a snapshot completes. The sync command transfers missing pack files without re-reading the source dataset:

bash
class="text-code-comment">#!/usr/bin/env bash
set -euo pipefail

LOCAL_REPO=class="text-code-string">"/var/backups/restic-repo"
class="text-code-comment">
# Sync to Hetzner SFTP
rclone sync class="text-code-string">"${LOCAL_REPO}" hetzner-sftp:/backups/restic-repo \
    --fast-list \
    --transfers 8 \
    --checkers 16 \
    --bwlimit class="text-code-string">"02:00,100M 08:00,10M" \
    --log-level INFO \
    --log-file /var/log/rclone-sync.log
class="text-code-comment">
# Sync to Backblaze B2 Cloud Object Storage
rclone sync class="text-code-string">"${LOCAL_REPO}" backblaze-b2:my-company-backup-bucket/restic-repo \
    --fast-list \
    --transfers 16 \
    --checkers 32 \
    --b2-hard-delete=false \
    --log-level INFO \
    --log-file /var/log/rclone-b2-sync.log

The --bwlimit flag limits network consumption to 10 MB/s during business hours (08:00) while allowing up to 100 MB/s during night windows (02:00).

Step 3: Production Scheduling and Resource Control

Backups running during peak hours can cause I/O spikes that stall web servers or increase database query latency. We enforce strict resource limits using systemd unit files rather than unmanaged crontabs.

Create /etc/systemd/system/restic-backup.service:

systemd
[Unit]
Description=Restic Backup Service
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/run-restic-backup.sh
ExecStartPost=/usr/local/bin/run-rclone-sync.sh
class="text-code-comment">
# Resource control settings
CPUSchedulingPolicy=batch
CPUShares=102
IONiceClass=best-effort
IONicePriority=7
IOWeight=10
MemoryMax=2G
MemoryHigh=1.5G

Create /etc/systemd/system/restic-backup.timer:

systemd
[Unit]
Description=Run restic backup daily at 02:30 UTC

[Timer]
OnCalendar=*-*-* 02:30:00 UTC
Persistent=true
RandomizedDelaySec=900

[Install]
WantedBy=timers.target

The RandomizedDelaySec=900 parameter prevents "thundering herd" bottlenecks on shared storage hardware or cloud API rate limits when managing hundreds of servers.

Step 4: Maintenance, Pruning, and Lock Management

Without a retention policy, repositories will eventually consume all available storage. However, running restic prune incorrectly can create race conditions with active backups because pruning requires an exclusive lock on the repository.

We separate the quick snapshot creation (daily) from snapshot pruning (weekly). Here is our retention policy script:

bash
class="text-code-comment">#!/usr/bin/env bash
set -euo pipefail

export RESTIC_REPOSITORY=class="text-code-string">"/var/backups/restic-repo"
export RESTIC_PASSWORD_FILE=class="text-code-string">"/etc/restic/repo-password.key"
class="text-code-comment">
# Apply retention policy: keep 7 daily, 4 weekly, 12 monthly, and 2 yearly snapshots
restic forget \
    --tag class="text-code-string">"scheduled-daily" \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 12 \
    --keep-yearly 2 \
    --prune \
    --max-unused 5%

Passing --max-unused 5% ensures that restic only rewrites pack files if it can free at least 5% of repository space. This prevents expensive I/O operations and API charge spikes on small amounts of dead data.

Step 5: Real Restore Drills

This is where standard operational advice usually stops, and where true disaster recovery planning begins. You do not have a working backup system until you execute automated restore tests that spin up service instances, write data into them, and run assertions against that data.

We run an automated verification script every Sunday night on an isolated staging worker. The script performs the following operations:

1. Connects to the offsite rclone destination (verifying offsite accessibility). 2. Pulls the latest snapshot index. 3. Restores a target database volume to a temporary directory. 4. Boots an ephemeral Docker container using that restored data volume. 5. Runs functional SQL assertions against the active container. 6. Cleans up state and sends a heartbeat to our monitoring platform.

Here is the Python implementation of our automated database restore drill:

python
class="text-code-comment">#!/usr/bin/env python3
import os
import subprocess
import sys
import time
import json

RESTIC_REPO = class="text-code-string">"s3:https://s3.us-west-002.backblazeb2.com/my-company-backup-bucket/restic-repo"
PASSWORD_FILE = class="text-code-string">"/etc/restic/repo-password.key"
RESTORE_PATH = class="text-code-string">"/tmp/restore-drill-pg"
CONTAINER_NAME = class="text-code-string">"drill-postgres-verify"

env = os.environ.copy()
env[class="text-code-string">"RESTIC_REPOSITORY"] = RESTIC_REPO
env[class="text-code-string">"RESTIC_PASSWORD_FILE"] = PASSWORD_FILE
env[class="text-code-string">"AWS_ACCESS_KEY_ID"] = os.getenv(class="text-code-string">"B2_KEY_ID", class="text-code-string">"")
env[class="text-code-string">"AWS_SECRET_ACCESS_KEY"] = os.getenv(class="text-code-string">"B2_APPLICATION_KEY", class="text-code-string">"")

def run(cmd, shell=False):
    res = subprocess.run(cmd, shell=shell, env=env, capture_output=True, text=True)
    if res.returncode != 0:
        print(fclass="text-code-string">"Error executing: {cmd}\nSTDOUT: {res.stdout}\nSTDERR: {res.stderr}")
        sys.exit(1)
    return res.stdout

def main():
    print(class="text-code-string">"[1/5] Fetching latest snapshot ID...")
    snapshots_json = run([class="text-code-string">"restic", class="text-code-string">"snapshots", class="text-code-string">"--json", class="text-code-string">"--latest", class="text-code-string">"1"])
    snapshots = json.loads(snapshots_json)
    if not snapshots:
        raise Exception(class="text-code-string">"No snapshots found in repository!")
    
    latest_id = snapshots[0][class="text-code-string">"short_id"]
    print(fclass="text-code-string">"Targeting snapshot: {latest_id}")

    print(class="text-code-string">"[2/5] Restoring database snapshot to local temporary disk...")
    run([class="text-code-string">"rm", class="text-code-string">"-rf", RESTORE_PATH])
    os.makedirs(RESTORE_PATH, exist_ok=True)
    run([class="text-code-string">"restic", class="text-code-string">"restore", latest_id, class="text-code-string">"--target", RESTORE_PATH, class="text-code-string">"--include", class="text-code-string">"/var/lib/postgresql/data"])

    print(class="text-code-string">"[3/5] Starting transient PostgreSQL container...")
    run([class="text-code-string">"docker", class="text-code-string">"rm", class="text-code-string">"-f", CONTAINER_NAME], shell=False)
    docker_cmd = [
        class="text-code-string">"docker", class="text-code-string">"run", class="text-code-string">"-d",
        class="text-code-string">"--name", CONTAINER_NAME,
        class="text-code-string">"-v", fclass="text-code-string">"{RESTORE_PATH}/var/lib/postgresql/data:/var/lib/postgresql/data",
        class="text-code-string">"-e", class="text-code-string">"POSTGRES_PASSWORD=drill_pass",
        class="text-code-string">"postgres:15-alpine"
    ]
    subprocess.run(docker_cmd, check=True)

    print(class="text-code-string">"[4/5] Waiting for engine initialization and running validation SQL...")
    time.sleep(10)
    
    sql_check = [
        class="text-code-string">"docker", class="text-code-string">"exec", CONTAINER_NAME,
        class="text-code-string">"psql", class="text-code-string">"-U", class="text-code-string">"postgres", class="text-code-string">"-c",
        class="text-code-string">"SELECT count(*) FROM information_schema.tables;"
    ]
    
    output = run(sql_check)
    print(fclass="text-code-string">"Verification output:\n{output}")
    
    if class="text-code-string">"count" not in output:
        print(class="text-code-string">"CRITICAL: Verification SQL query failed!")
        sys.exit(1)

    print(class="text-code-string">"[5/5] Restore Drill Successful. Tearing down resources...")
    run([class="text-code-string">"docker", class="text-code-string">"rm", class="text-code-string">"-f", CONTAINER_NAME])
    run([class="text-code-string">"rm", class="text-code-string">"-rf", RESTORE_PATH])

if __name__ == class="text-code-string">"__main__":
    main()

If this script exits with status code 1, our alerting engine triggers an P1 incident for the infrastructure on-call engineer. We do not wait for a datacenter outage to discover that a schema migration broke our database restore process.

Performance and Cost Tradeoffs

Choosing remote storage engines requires balancing latency, transfer limits, and object locking capability. Below is our evaluation matrix across four standard offsite targets:

| Metric / Backend | Local ZFS NVMe | Hetzner Storage Box | Backblaze B2 | AWS S3 Standard | | :--- | :--- | :--- | :--- | :--- | | Primary Role | Copy 1 (Fast local) | Copy 2 (Offsite primary) | Copy 3 (Offsite immutable) | Copy 3 (Alternative) | | Protocol Support | Local filesystem | SFTP / SMB / WebDAV | S3 API / Native B2 | S3 API | | Storage Cost/TB | $0 (Owned hardware) | ~$3.50/month | $6.00/month | $23.00/month | | Egress Cost/TB | $0 | $0 | $0 (via Fastly/Bandwidth Alliance) | $90.00/month | | Immutable Object Lock | No (ZFS Snapshots help) | No | Yes (Compliance mode) | Yes (Compliance mode) | | 100GB Restore Speed | ~35 seconds | ~14 minutes | ~8 minutes | ~3 minutes |

Using local storage alongside Backblaze B2 gives us low costs and true immutable object locking, protecting our offsite backups against compromised credentials.

Operational Realities: Lessons learned

If you implement this setup, keep these edge cases in mind:

  • Watch out for memory limits on large repos: Restic builds an in-memory index during operations. For repositories with tens of millions of small files, restic can consume 4GB+ of RAM. Set systemd memory limits accordingly so the kernel OOM killer doesn't terminate the job halfway through.
  • Avoid continuous restic check --read-data operations: Scanning every encrypted block in cloud storage will result in massive bandwidth charges. Run a basic index check (restic check) weekly, and save full data reads (restic check --read-data-subset=2%) for monthly automated maintenance windows.
  • Keep passwords off the host machine: Store your master encryption keys in an external secret manager (Vault, 1Password CLI, AWS Secrets Manager) and inject them at job runtime, or restrict local key files to chmod 0400 owned strictly by root.

The Verdict

Backups are an operational expense; fast restores are an organizational asset.

Using restic for deduplication and local snapshots alongside rclone for multi-cloud distribution provides a lightweight, vendor-agnostic 3-2-1 backup pipeline. But the system is incomplete without programmatic restore drills. If your pipeline isn't constantly extracting snapshots, launching test containers, and executing assertion tests in CI/CD, you don't have backups—you just have hope.

Paj Digital Labs

The engineering journal of Paj Digital Solutions — hosting, server infrastructure and web engineering, written by the people who run the servers.

Related reading