Key & secret rotation

This runbook is for the operator who has to rotate a secret on a live Opbox box: because a key may have leaked, because a staff member left, because a provider key is being cycled, or just because policy says “rotate every N days.” It is honest about what the kernel does for you automatically and what is still a careful, manual procedure today.

Read Security for the model these secrets sit inside, and Files & storage for how the encryption binds to each file. The one sentence to carry in: encrypted data is sealed under the key that was live when it was written, so swapping a key is never as simple as editing one environment variable.

The secrets, at a glance

Each is covered below.

1. The encryption key (OPBOX_PII_KEY)

What it protects, and why a swap is not safe

OPBOX_PII_KEY is the 32-byte (AES-256) key for the regulated-PII domain: encrypted PII facets and file bytes. Every sealed blob is AAD-bound to its own row identity (workspace + natural key), so a blob cannot be lifted into another row or workspace and still decrypt.

The key is resolved at crypto::key_from_env from the env var (hex, 64 chars, or standard base64, decoding to exactly 32 bytes). There is no in-binary production key: a release binary with neither OPBOX_PII_KEY nor the dev opt-in set fails closed rather than guarding real PII with a known constant.

Here is the trap. Every ciphertext on disk was sealed under the key that was live at write time. If you change OPBOX_PII_KEY and restart, new writes seal under the new key, but every existing blob still needs the OLD key to open and now fails closed (aead decrypt failed (integrity)). A naive swap does not lose the old data, but it makes it unreadable until the old key returns. So rotation is not a swap; it is a re-encrypt (decrypt-under-old, re-seal-under-new) of every affected row.

Honest status: automated re-key is NOT built

There is no opbox rekey verb today. The code carries the seam for it - the encryption_scheme / encryptionScheme column is recorded on every blob, and key_from_env is the documented KMS handoff point - but the envelope/re-encrypt step that walks the store and re-seals under a fresh key has not been built. Treat key rotation as a planned maintenance operation, not a one-command action.

The careful manual posture (until automated re-key lands)

If you must rotate OPBOX_PII_KEY, do it as a controlled maintenance window, not in place:

1. Quiesce writes. Take the box out of service (or to read-only) so no new blob is sealed under the old key mid-procedure. A blob written after you start but before you re-encrypt it would be missed.
2. Snapshot first. Take a full database + file-store backup, and keep the old key with it. The backup is your only way back; a backup whose key you have discarded is unreadable.
3. Keep the old key available. You cannot decrypt the existing store without it. Do not delete or overwrite the old OPBOX_PII_KEY value until step 6 verifies the new one.
4. Re-encrypt every blob: decrypt-under-old, re-seal-under-new. This is the missing automated step. Until it ships, this is a one-off migration script, run against a quiesced copy, that for each PII facet and file row: opens it with the old key (and its existing AAD), then re-seals it with the new key (same AAD, fresh nonce, same aes-256-gcm scheme tag). The AAD must be reproduced exactly, or the re-seal binds to the wrong identity.
5. Cut over the env var to the new key (hex or base64, exactly 32 bytes) and restart.
6. Verify before discarding the old key. Read back a sample of re-encrypted facts and files end to end (open a matter document, fetch a file’s bytes). Only once reads succeed under the new key alone do you retire the old key (archive it with the pre-rotation backup; do not leave it in the live env).
7. Bring writes back. Restore service.

If a re-encrypt is not feasible in your window, the only safe alternative is do not rotate yet - never swap the key and leave the old ciphertext stranded.

The other at-rest keys follow the same rule

• OPBOX_CRED_KEY seals stored connection credentials (the same AEAD, AAD-bound to workspace + connection id). Rotating it has the identical re-encrypt problem: existing credential blobs are sealed under the old key. In practice the cleaner move for a single integration is to rotate the provider secret (section 3) and re-store it, which re-seals that one credential under the current key with no bulk migration.
• OPBOX_WEBHOOK_KEY derives per-rail webhook signing secrets. Rotating it changes the derived secrets, so any external party verifying those signatures must be updated in step with the env swap.

2. Bearer tokens / API keys

Tokens are the everyday access credential. The good news: rotation here is built, and the design makes it safe by construction.

A token is never stored in a form anyone can read back. Only a one-way hash (token_hash, sha256) is persisted; the plaintext is shown exactly once at mint and is unrecoverable thereafter. A stolen database copy yields no usable tokens. That means rotation is revoke + reissue, not “go read the old value.”

Two revocation axes exist (verbs/token.rs):

• token.revoke (SENSITIVE / L2 / ADMIN) - the terminal kill of one token. A later consume of a REVOKED token is refused (410-equivalent). Re-revoking a terminal token is an idempotent no-op.
• token.revoke-scope (SENSITIVE / L2 / ADMIN) - bumps a scope-version counter so every token bound to that scope goes stale at once (the fan kill).

To rotate a key:

1. Mint the replacement with token.mint (SENSITIVE / L2 / ADMIN). Capture the plaintext it returns once and hand it to the consumer. Cut the new token as narrowly as the job needs (capability scope + resource binding + autonomy ceiling) - see Security.
2. Cut over the caller to the new token.
3. Revoke the old token with token.revoke. For a compromised scope (e.g. a leaked portal-link family), use token.revoke-scope to kill the whole fan in one act.
4. Verify with token.list / token.get (READ / L0 / ADMIN) - these return inventory but never the hash or plaintext. Confirm the old token id is REVOKED and the new one ACTIVE.

If a token may be compromised, revoke first, then reissue - the order that closes the window fastest.

3. Integration secrets (Stripe, and other providers)

A provider secret (a Stripe secret key, an API key for some external service) is rotated at the provider, then reflected into Opbox. Opbox stores it sealed under OPBOX_CRED_KEY (section 1); it never holds the only copy of the rotation.

1. Rotate at the provider. In the provider’s own dashboard, roll the key (Stripe: create a new secret key / roll the existing one). Most providers let the old and new key both work briefly - use that overlap.
2. Update the stored credential in Opbox. Re-store the new secret through the connection verb that writes credentials, which re-seals it under the current OPBOX_CRED_KEY (fresh nonce, AAD-bound to workspace + connection id). Because you are re-storing, this one credential is naturally re-encrypted under the live key - no bulk migration needed for it.
3. Verify the integration with a low-risk call (a balance read, a test event) before the overlap window closes.
4. Revoke the old provider key at the provider once the new one is confirmed working.
5. Webhook signatures: if the provider also signs inbound webhooks and you rotated that signing secret, update it on both sides. If the rotation touched OPBOX_WEBHOOK_KEY (Opbox-derived rail secrets), update every external verifier in step with the env swap.

The provider’s dashboard is the system of record for whether the old key is dead. Opbox only needs the current value, sealed.

4. Sessions - forcing re-authentication

A logged-in human session is a bound SESSION token minted by login (12-hour TTL). You do not have to hunt individual session tokens to log someone out: each token is stamped at mint with the bound actor’s session_version, and the resolver fails closed when that stamp is older than the actor’s current session_version. Bumping the column is how you force re-auth.

• One user, all their sessions: session.revoke (SENSITIVE / L2 / ADMIN, { actorId }) bumps that actor’s session_version + 1. Every bearer minted before the bump is immediately stale; the user must sign in again. Re-revoking just bumps again (idempotent-shaped).
• A whole workspace at once: org.freeze (EPIC-OPS) bumps session_version across every actor in the workspace - the org-wide mass re-auth, for an incident.
• Password / credential change: a password reset (reset.rs) and a credential change both bump session_version automatically, so changing a password logs out every existing session for that user. You do not call session.revoke separately after a reset.

This is the right tool when a laptop is lost, a person leaves, or you suspect a session was hijacked: bump the version and every outstanding session for that actor (or workspace) dies at the next request, fail-closed, with no need to find the token itself.

Putting it together for an incident

If a credential may have leaked, work outward from the cheapest, most-built lever to the most manual:

1. session.revoke the affected actor (or org.freeze the workspace) - instant, built, no data migration. Logs everyone out.
2. token.revoke / token.revoke-scope any token that may be exposed, then token.mint replacements - built, fast.
3. Rotate the provider secret (section 3) and re-store it - re-seals that one credential cleanly.
4. Rotate OPBOX_PII_KEY / OPBOX_CRED_KEY only if the at-rest key itself is suspect - the careful manual, quiesce-snapshot-re-encrypt-verify procedure (section 1), because automated re-key is not yet built.

Every step above writes to the tamper-evident audit log, so the rotation is itself on the record. For the defences these secrets sit behind, see Security; for how the file seal binds to each row, see Files & storage.