# Attestor observability runbook (stub · 2025-11-29 demo)

## Dashboards (offline import)
- Grafana JSON: `docs/modules/attestor/operations/dashboards/attestor-observability.json` (import locally; no external data sources assumed).
- Planned panels: signing latency p95/p99, verification failure rate, transparency log submission lag, key rotation age, DSSE verification errors, queue depth/backlog, and attestation bundle size histogram.

## Key metrics
- `attestor_sign_latency_seconds_bucket` — signing latency per request.
- `attestor_verify_failures_total{reason}` — verification failures.
- `attestor_tlog_submission_latency_seconds_bucket` — transparency log latency.
- `attestor_key_rotation_age_seconds` — time since last rotation.
- `attestor_queue_backlog` — pending attestation jobs.
- `attestor_bundle_size_bytes_bucket` — bundle size distribution.

## Logs & traces
- Correlate by `correlationId`, `attestationId`, `tenant`, `keyId`. Include `policyVersion`, `tlogIndex`, and `rekorUrl` where applicable.
- Traces disabled by default for air-gap; enable by pointing OTLP exporter to on-prem collector.

## Health/diagnostics
- `/health/liveness` and `/health/readiness` check signer availability, tlog reachability, and storage.
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
- Verification probe: `POST /api/attestations/verify` with sample bundle once demo outputs land.

## Alert hints
- Signing latency p99 > 1s.
- Verification failure spikes.
- Transparency log submission lag > 10s.
- Key rotation age exceeding policy threshold.
- Queue backlog above configured threshold.

## Offline verification steps
1) Import Grafana JSON locally; point to Prometheus scrape labeled `attestor`.
2) Run sample verification once demo bundle available and confirm metrics/logs emit locally.
3) Fetch `/status` and compare commit/version to offline bundle manifest.

## Evidence locations
- Sprint tracker: `docs/implplan/SPRINT_0313_0001_0001_docs_modules_attestor.md`.
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
- Dashboard stub: `operations/dashboards/attestor-observability.json`.