Observability and runbooks

Raidz currently has service health endpoints, structured Fastify logs with sensitive-field redaction, container health checks, deployment receipts, and an incident runbook. OpenTelemetry, Grafana, Sentry, and comprehensive business/chain monitoring are selected but not yet implemented.

Health semantics

• Liveness /health proves the API process can answer. It does not check dependencies.
• Readiness /ready runs configured dependency checks and returns 503 when any fails. PostgreSQL is the default check.
• Web /api/health proves the deployed Next.js surface responds; it is not equivalent to API readiness.

Use each signal for its intended purpose. A readiness failure should remove a service from traffic without automatically restarting an external mutation.

Minimum production signals

Application

Request rate, latency, status/error code, route, deployment version, readiness, database pool pressure, query timeouts, session creation, challenge failures, OAuth callback failures, and bot update lag.

Marketplace

Campaign count by state, reserved creator obligations, funding/creator-pool/fee reconciliation, open claims, submission age, review backlog, verifier disagreement, release failures, retention due, refunds, and duplicate/replay attempts.

Chain

Finalized campaign events, token transfers, escrow balances versus obligations, verifier-set changes, fee-recipient changes, pauses, Pons phase transitions, reorg handling, and indexing lag. Alerts must identify chain ID and contract address.

Product evidence

Consented creator/project activation, funded campaigns, verified campaign value settled, repeat projects, repeat creators, disputes, qualified outcomes, and source category. Fixture, operator, bot, internal, and paid traffic must be separable.

Logging rules

Fastify redacts authorization, cookie, set-cookie, token, secret, and signature fields. Extend redaction when adding provider payloads or wallet routes. Never log OAuth tokens, bot tokens, private keys, seed phrases, raw private content, full Telegram auth payloads, or unneeded personal data.

Use stable request/event IDs and pseudonymous actor IDs. Economic logs should include campaign/claim ID, chain, contract, expected amount, observed receipt, and state transition—not signer secrets.

Alerts

Page an operator for possible fund loss, signature/key compromise, privacy disclosure, contract pause, accounting mismatch, unknown external state, critical dependency compromise, or incorrect public contract information. Lower-severity alerts include provider degradation, source outage, indexing lag, and growing review backlog.

Runbooks

The incident response guide defines containment and recovery. Before value-bearing launch, add scenario runbooks for verifier compromise, escrow pause, provider outage, chain reorg, Pons graduation boundary, wrong public address, database recovery, bot takeover, and OAuth revocation.