opbox

Backup & restore

Protect a client's data - what to back up, how, and how to restore.

A tenant box holds one client’s whole working life: their matters, their parties, their documents, their audit history. This runbook is the operator’s plain procedure for protecting it and getting it back. Read The stack first so the services and secrets here are familiar.

Honesty up front: backup is operator-run today, not automated. There is no scheduled-backup cron, off-box shipper, or managed snapshot wired into the release stack yet. The dr.* verbs (dr.policy.set, dr.status) record a DR policy (RPO/RTO, drill cadence) and report status - they do not run a backup for you. Until a scheduler lands, the procedures below are run by hand (or by your own cron/systemd timer wrapping them). Treat that as the current state, not the end state.

1. What to back up (four things, and why)

A complete, restorable backup of a tenant box is four artifacts. Miss any one and the restore is either impossible or useless:

  1. The Postgres database - the single system of record. Almost everything is here: matters, parties, bills, the org roster, the encrypted PII columns on party, the signing events, and the hash-chained audit log. This is the artifact whose loss is unrecoverable.

  2. The MinIO object store - the file bytes. By default (OPBOX_STORAGE_BACKEND=minio) uploaded file bytes live in the in-stack MinIO object store (the opbox_minio_data volume), not in Postgres. The kernel seals every blob before it reaches MinIO, so the store holds AAD-bound AES-256-GCM ciphertext only - but you must still back the bytes up, or a restored database points at files that no longer exist. (If the box runs OPBOX_STORAGE_BACKEND=bytea instead, the bytes live in an encrypted bytea column and the Postgres dump captures them - then there is no separate store.)

  3. The encryption key - OPBOX_PII_KEY. The file bytes and the personal data are AES-256-GCM ciphertext. Without the key that encrypted them, the backup is gibberish - by design (see Security & permissions, Lock 6: “a stolen backup is not a breach”). The flip side is operationally brutal: lose the key and you lose the data, even though you still hold the dump. The key lives in .env, not in the database, so it must be backed up separately and, where possible, to a different place than the dump and the object store - so one stolen copy is never both halves.

  4. Config and the rest of .env. The other secrets (POSTGRES_PASSWORD, DOC_RENDER_HMAC_SECRET, OPBOX_WEBHOOK_KEY, OPBOX_AGENT_KEY, OPBOX_SITE_ADDRESS, the LLM key) and the Caddyfile. Losing these does not lose client data, but it costs you a re-genesis or a manual re-wire to bring the box back. OPBOX_WEBHOOK_KEY in particular derives the per-tenant inbound-webhook HMAC - change it and existing webhook registrations stop verifying.

The Caddy volumes (opbox_caddy_data, opbox_caddy_config) hold auto-provisioned TLS certs and are not worth backing up - Caddy re-issues certificates from Let’s Encrypt on the next bring-up. The database volume is opbox_pgdata (mounted at /var/lib/postgresql); we back it up logically with pg_dump, not by copying the volume. The opbox_minio_data volume holds the file ciphertext and is worth backing up (section 2b).

2. How to back up

2a. Dump the database

pg_dump produces a single consistent logical snapshot. Run it inside the Opbox-Postgres container (the release stack never publishes a Postgres host port, so this is the reachable path):

# from the repo root, alongside .env
TS=$(date -u +%Y%m%dT%H%M%SZ)
docker exec Opbox-Postgres pg_dump -U opbox -d opbox -Fc -f /tmp/opbox-$TS.dump
docker cp Opbox-Postgres:/tmp/opbox-$TS.dump ./opbox-$TS.dump
docker exec Opbox-Postgres rm -f /tmp/opbox-$TS.dump
  • -Fc is the custom (compressed) format - it restores with pg_restore and lets you restore selectively. A plain .sql dump (-Fp) is fine too and is human-readable, just larger.
  • The dump contains the encrypted file and PII bytes as-is. It is not plaintext client data, but it is still sensitive (it is one half of the pair) - store it somewhere access-controlled.
  • Move the dump off the box. A backup that only exists on the machine it is protecting is not a backup.

To wrap this in your own schedule until automation lands, drop the three lines into a script and run it from cron or a systemd timer; ship the resulting file to off-box storage and prune old ones.

2b. Back up the object store (the file bytes)

On the default minio backend the file ciphertext lives in the opbox_minio_data volume. Snapshot it straight from the volume (the MinIO API is on the internal network only):

docker run --rm -v opbox_minio_data:/data -v "$PWD":/backup alpine \
  tar czf /backup/opbox-minio-$TS.tgz -C /data .

Take this snapshot at the same time as the database dump and keep the pair together off-box - a dump without its matching object store (or the reverse) restores to dangling file references. (On the bytea backend this step is unnecessary: the bytes are already in the dump.)

2c. Back up the encryption key and config - separately

The key is plain text in .env. Capture it on its own, to a different vault than the dump:

grep '^OPBOX_PII_KEY=' .env      # the one that makes the dump readable

Store that value (and ideally the whole .env) in your secrets manager / password vault - not next to the database dump. The whole point of Lock 6 is that the dump and the key are separable; keeping them together throws that protection away. Also keep a copy of .env and the Caddyfile for a fast rebuild.

Key rotation note. If OPBOX_PII_KEY is ever rotated, every dump taken under the old key still needs the old key to restore. Keep retired keys for as long as you keep the dumps they decrypt.

3. How to restore

Restoring a tenant is: stand up a clean stack, load the dump, make sure the same key is in place, then verify. Order matters - the key must be the one the data was encrypted under, or every decrypt fails.

  1. Bring up a clean stack with the original secrets. On the target host, put back the backed-up .env (so OPBOX_PII_KEY, POSTGRES_PASSWORD, etc. are exactly what the data expects), then:

    docker compose -f docker-compose.yml -f docker-compose.beelink.yml --env-file .env up -d postgres

    Bring up only postgres first, so you load the dump into an empty database before the kernel runs migrations or touches anything.

  2. Restore the dump. Copy it into the container and load it:

    docker cp ./opbox-<TS>.dump Opbox-Postgres:/tmp/restore.dump
docker exec Opbox-Postgres pg_restore -U opbox -d opbox --clean --if-exists /tmp/restore.dump
docker exec Opbox-Postgres rm -f /tmp/restore.dump

    --clean --if-exists drops existing objects first so the restore is repeatable into a non-empty DB. For a truly empty target you can omit them. (If you took a plain -Fp dump, load it with docker exec -i Opbox-Postgres psql -U opbox -d opbox < opbox-<TS>.dump instead.)

  3. Confirm the key matches. This is the step that silently bites people: a restore into a stack whose OPBOX_PII_KEY differs from the one the dump was encrypted under will load fine but every file and PII read will fail to decrypt. The key in .env must be the original. Do not run genesis on a restored box - genesis founds a new owner; you already have one in the restored data.

  4. Restore the object store, then bring up the rest of the stack. If the box used the minio backend, load the file ciphertext back into its volume first (the volume is created if absent, and MinIO is not yet running):

    docker run --rm -v opbox_minio_data:/data -v "$PWD":/backup alpine \
  sh -c 'cd /data && tar xzf /backup/opbox-minio-<TS>.tgz'
docker compose -f docker-compose.yml -f docker-compose.beelink.yml --env-file .env up -d

    The kernel will apply any pending migrations on top of the restored schema and come up healthy.

4. Verify the restore

A restore is not done until you have proven the data is both present and intact. Two checks:

  1. Health check - the stack is up. The kernel exposes /health (the only unauthenticated door):

    docker exec Opbox-Kernel curl -fsS http://127.0.0.1:8088/health

    A 200 means the kernel booted, reached Postgres, and ran migrations cleanly.

  2. Audit chain verify - the record is intact and the key is right. The audit log is hash-chained: each record carries a fingerprint of the one before it, so any gap or tamper breaks the chain. Verifying it end to end proves the restored database is internally consistent. Call audit.chain.verify with an owner / admin bearer (it is an Admin-permission read verb):

    docker exec Opbox-Kernel curl -fsS -X POST http://127.0.0.1:8088/v/audit.chain.verify \
  -H "authorization: Bearer <OWNER_BEARER>" -H "content-type: application/json" -d '{}'

    A clean report means the chain is unbroken across the restored events. A break points to a truncated or inconsistent dump - investigate before declaring the box recovered. The signing chain has its own equivalent (signing.chain.verify) for the e-signature record.

  3. Spot-check a real file. Open one matter and one document in the app (or via doc.get / file.get). A document that opens proves the key matches the data - the decisive confirmation that step 3 of the restore was right. If files come back as errors or garbage while the rest of the app works, the OPBOX_PII_KEY is wrong: stop and fix the key before anyone uses the box.

5. The drill (do not skip)

A backup you have never restored is a hope, not a backup. Periodically restore the latest dump onto a throway host or a scratch stack and run the section-4 checks. dr.policy.set lets the owner record the intended drill cadence so it is on the record; honouring it is still a manual operator act today. The first time you exercise a restore should never be the day the box is gone.