Monitoring & health

A box runs unattended. This is the operator’s checklist for the three questions that matter: is it up (liveness), is it honest (the audit log has not been tampered with), and is the agent within bounds (autonomy and budget). It is deliberately honest about what is wired today versus what you assemble yourself - Opbox ships the primitives (a probe, a verifier, a spend lens); it does not ship a bundled metrics dashboard or an alerting daemon.

1. Liveness - is the process up?

The /health endpoint

The kernel exposes one unauthenticated liveness probe:

curl -fsS http://127.0.0.1:8088/health
# {"status":"ok"}

What it tells you, and what it deliberately does not:

• A 200 {"status":"ok"} means the kernel process is up and the database came up. The pool runs wait_until_ready() at boot before the server binds its port, so reaching this handler at all proves the DB connected. There is no separate “DB is healthy” flag to read.
• It returns a static body with no version or build string - an unauthenticated edge must not disclose it. Do not parse it for a version; there is nothing to parse.
• It is a shallow probe, not a deep one. It does not run a query per call. If you ever want a deep DB probe, it belongs on a separate authenticated route - do not add it here.

Container healthchecks

The compose stack wires a healthcheck into the data-plane and frontend services, so Docker restarts a dead container and caddy waits for a healthy kernel before it starts:

Note agent-chat and docs have no compose healthcheck (they are condition: service_started, not service_healthy). Liveness for the agent is a separate question - see section 3.

# the one-line "is everything healthy" check
docker compose -f docker-compose.yml -f docker-compose.beelink.yml ps
# look for (healthy) on kernel/postgres/docrender/frontend; agent-chat/docs/caddy show running, not health-gated

If kernel is stuck unhealthy, it failed its own /health curl - check its logs (section 4) for a law-pin drift, a bad DATABASE_URL, or a missing required secret (${VAR:?} aborts the bring-up loudly).

2. Integrity - is the audit log honest?

Every governed verb is permission-checked and written to an append-only event log before it runs (INV-8). That log is hash-chained: each entry commits to the previous one, so any retroactive edit, deletion, or insertion breaks the chain. You do not have to trust that nothing was tampered with - you can prove it.

# verify the chain (Admin permission required)
curl -fsS http://127.0.0.1:8088/v/audit.chain.verify \
  -H "Authorization: Bearer $ADMIN_BEARER" -d '{}'

• audit.chain.verify (audit.read, Admin, idempotent) recomputes the chain and reports whether it is intact. A clean result means no entry has been altered since it was written. A break tells you exactly where the chain diverges.
• This is the integrity backstop for the whole box: the agent is untrusted, but its every move is recorded in a log it cannot rewrite. Verifying the chain is how you turn “we record everything” into “we can prove the record is whole”.
• To read the log itself, use audit.query (the immutable event log) or audit.feed. Both are Admin, audit.read.

Run audit.chain.verify on a schedule (a daily cron is reasonable) and on demand after any suspected incident. See the audit verbs for the full surface.

3. The agent - is it within autonomy and budget?

The public chat runs as a bounded agent, not the owner key: a dedicated WORKER provisioned at autonomy 1, MEMBER tier, never the OWNER Chief-of-Staff key. Autonomy is the real bound - it permits reads and writes but denies every sensitive or owner operation, regardless of verb scope. That bound is enforced by the kernel gate on every call, so there is no “agent went rogue” failure mode where it quietly gains powers. See Agents & MCP and Security (Lock 3, autonomy).

What you monitor is therefore spend, not capability:

# operator spend lens - how much AI cost has accrued (Admin)
curl -fsS http://127.0.0.1:8088/v/cost.query \
  -H "Authorization: Bearer $ADMIN_BEARER" -d '{}'

# budget-vs-spend status: is this workspace over its monthly cap?
curl -fsS http://127.0.0.1:8088/v/cost.budget.check \
  -H "Authorization: Bearer $ADMIN_BEARER" -d '{}'

• cost.query (cost.read, Admin) is the operator’s spend read - the SUM over the append-only ai_cost_ledger. Each LLM call the agent loop makes records its own spend through cost.record (a governed write, INV-1 - not a privileged shadow insert), so the ledger is the honest tally.
• cost.budget.check (cost.read, Admin, idempotent) compares that ledger spend against the workspace budget (box_config key ai.budget.monthly_minor, set via the normal config.set - there is no separate budget store) and returns overBudget.

Be honest about enforcement. cost.budget.check is a check, not a gate. Today it is the M3 agent loop that calls it before an LLM call and decides to refuse - the refusal is the loop’s, this verb only reports. A dispatch-level enforcement gate (the kernel auto-refusing a verb when over budget) is a deferred trust-boundary change and is not wired into dispatch yet. So: if you rely on the budget as a hard stop, verify your agent loop actually calls cost.budget.check, and treat overBudget: true as an alert condition you act on (section 5), not something the kernel will block for you.

The budget value lives in workspace config:

# set the monthly AI budget (minor units, e.g. 50000 = 500.00)
curl -fsS http://127.0.0.1:8088/v/config.set \
  -H "Authorization: Bearer $ADMIN_BEARER" \
  -d '{"key":"ai.budget.monthly_minor","value":"50000"}'

See the cost verbs for the full surface.

Seat liveness

The agent is the agent-chat container, and as noted it has no compose healthcheck. To confirm the seat is actually alive (not just running), exercise its front door:

docker logs --tail 50 Opbox-Agent-Chat        # is the loop serving / erroring?

An empty OPBOX_AGENT_KEY is the common “seat is up but inert” case: agent-chat starts, but every MCP call 401s because genesis has not minted (or has not re-wired) the agent bearer. The fix is to re-run the genesis agent-mint step; the symptom is a chat that connects but can do nothing.

4. Logs - where to look

There is no log aggregation in the box; logs are container stdout. RUST_LOG=info is the kernel default.

docker logs -f Opbox-Kernel        # kernel: dispatch, trigger scheduler ticks, the law-pin verdict
docker logs -f Opbox-Agent-Chat    # the agent loop / cage gateway
docker logs -f Opbox-Postgres            # postgres
docker logs -f Opbox-Caddy         # TLS / reverse-proxy / cert renewal

Lines worth knowing on the kernel at boot and in steady state:

• VPS law pin verified (C06) on a clean boot, or VPS LAW PIN DRIFT - kern write verbs are refusing (C13) if the vendored policypack no longer hashes to its manifest pin. Drift is verb-scoped fail-closed: the kern.* write verbs refuse and reads serve flagged, but the rest of the box keeps serving - a law-file touch is deliberately not a box kill switch. Treat drift as a high-priority alert.
• opbox HTTP front door listening once the server binds.
• opbox trigger scheduler started / tick lines (the in-process scheduler; quiet unless something is dispatched, failed, or filtered). A repeating trigger scheduler tick failed warning is worth a look.

The durable record of what the agent did is not in these logs - it is in the audit log (section 2). Logs are for the process; the audit chain is for the actions.

5. What to alert on

Wire these into whatever you run (a cron + a notifier; there is no built-in alerter). In rough priority order:

1. /health not 200 for more than a couple of intervals - the kernel is down or the DB never came up. This is the page-someone signal.
2. Any container unhealthy / restart-looping in docker compose ps - especially kernel or postgres. A restart loop usually means a missing secret or a bad DATABASE_URL.
3. audit.chain.verify reports a break - integrity is compromised; capture state and investigate before doing anything that writes more entries.
4. Law-pin drift in the kernel log (LAW PIN DRIFT) - write verbs are refusing; reconcile the vendored policypack against its manifest.
5. cost.budget.check returns overBudget: true - the workspace has exceeded its AI cap. Remember the gate is advisory today, so this is an operator action: raise the cap, throttle, or stop the agent.
6. agent-chat 401-looping / empty OPBOX_AGENT_KEY - the seat is inert; re-mint the agent bearer.
7. Repeated trigger scheduler tick failed - the background scheduler is erroring; check the kernel log.

Everything above is a single curl or docker command, so a minimal monitor is a small script on a cron that runs sections 1-3, diffs the result, and notifies on change - no extra service required.