# AOC Observability Guide

> **Audience:** Observability Guild, Concelier/Excititor SREs, platform operators.  
> **Scope:** Metrics, traces, logs, dashboards, and runbooks introduced as part of the Aggregation-Only Contract (AOC) rollout (Sprint 19).

This guide captures the canonical signals emitted by Concelier and Excititor once AOC guards are active. It explains how to consume the metrics in dashboards, correlate traces/logs for incident triage, and operate in offline environments. Pair this guide with the [AOC reference](../ingestion/aggregation-only-contract.md) and [architecture overview](../modules/platform/architecture-overview.md).

---

## 1 · Metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `ingestion_write_total` | Counter | `source`, `tenant`, `result` (`ok`, `reject`, `noop`) | Counts write attempts to `advisory_raw`/`vex_raw`. Rejects correspond to guard failures. |
| `ingestion_latency_seconds` | Histogram | `source`, `tenant`, `phase` (`fetch`, `transform`, `write`) | Measures end-to-end runtime for ingestion stages. Use `quantile=0.95` for alerting. |
| `aoc_violation_total` | Counter | `source`, `tenant`, `code` (`ERR_AOC_00x`) | Total guard violations bucketed by error code. Drives dashboard pills and alert thresholds. |
| `ingestion_signature_verified_total` | Counter | `source`, `tenant`, `result` (`ok`, `fail`, `skipped`) | Tracks signature/checksum verification outcomes. |
| `advisory_revision_count` | Gauge | `source`, `tenant` | Supersedes depth for raw documents; spikes indicate noisy upstream feeds. |
| `verify_runs_total` | Counter | `tenant`, `initiator` (`ui`, `cli`, `api`, `scheduled`) | How many `stella aoc verify` or `/aoc/verify` runs executed. |
| `verify_duration_seconds` | Histogram | `tenant`, `initiator` | Runtime of verification jobs; use P95 to detect regressions. |

### 1.1 Alerts

- **Violation spike:** Alert when `increase(aoc_violation_total[15m]) > 0` for critical sources. Page SRE if `code="ERR_AOC_005"` (signature failure) or `ERR_AOC_001` persists > 30 min.
- **Stale ingestion:** Alert when `max_over_time(ingestion_latency_seconds_sum / ingestion_latency_seconds_count)[30m]` exceeds 30 s or if `ingestion_write_total` has no growth for > 60 min.
- **Signature drop:** Warn when `rate(ingestion_signature_verified_total{result="fail"}[1h]) > 0`.

---

## 2 · Traces

### 2.1 Span taxonomy

| Span name | Parent | Key attributes |
|-----------|--------|----------------|
| `ingest.fetch` | job root span | `source`, `tenant`, `uri`, `contentHash` |
| `ingest.transform` | `ingest.fetch` | `documentType` (`csaf`, `osv`, `vex`), `payloadBytes` |
| `ingest.write` | `ingest.transform` | `collection` (`advisory_raw`, `vex_raw`), `result` (`ok`, `reject`) |
| `aoc.guard` | `ingest.write` | `code` (on violation), `violationCount`, `supersedes` |
| `verify.run` | verification job root | `tenant`, `window.from`, `window.to`, `sources`, `violations` |

### 2.2 Trace usage

- Correlate UI dashboard entries with traces via `traceId` surfaced in violation drawers (`docs/ui/console.md`).
- Use `aoc.guard` spans to inspect guard payload snapshots. Sensitive fields are redacted automatically; raw JSON lives in secure logs only.
- For scheduled verification, filter traces by `initiator="scheduled"` to compare runtimes pre/post change.

---

## 3 · Logs

Structured logs include the following keys (JSON):

| Key | Description |
|-----|-------------|
| `traceId` | Matches OpenTelemetry trace/span IDs for cross-system correlation. |
| `tenant` | Tenant identifier enforced by Authority middleware. |
| `source.vendor` | Logical source (e.g., `redhat`, `ubuntu`, `osv`, `ghsa`). |
| `upstream.upstreamId` | Vendor-provided ID (CVE, GHSA, etc.). |
| `contentHash` | `sha256:` digest of the raw document. |
| `violation.code` | Present when guard rejects `ERR_AOC_00x`. |
| `verification.window` | Present on `/aoc/verify` job logs. |

Logs are shipped to the central Loki/Elasticsearch cluster. Use the template query:

```logql
{app="concelier-web"} | json | violation_code != ""
```

to spot active AOC violations.

---

## 4 · Dashboards

Primary Grafana dashboard: **“AOC Ingestion Health”** (`dashboards/aoc-ingestion.json`). Panels include:

1. **Sources overview:** table fed by `ingestion_write_total` and `ingestion_latency_seconds` (mirrors Console tiles).
2. **Violation trend:** stacked bar chart of `aoc_violation_total` per code.
3. **Signature success rate:** timeseries derived from `ingestion_signature_verified_total`.
4. **Supersedes depth:** gauge showing `advisory_revision_count` P95.
5. **Verification runs:** histogram and latency boxplot using `verify_runs_total` / `verify_duration_seconds`.

Secondary dashboards:

- **AOC Alerts (Ops view):** summarises active alerts, last verify run, and links to incident runbook.
- **Offline Mode Dashboard:** fed from Offline Kit imports; highlights snapshot age and queued verification jobs.

Update `docs/assets/dashboards/` with screenshots when Grafana capture pipeline produces the latest renders.

---

## 5 · Operational workflows

1. **During ingestion incident:**
   - Check Console dashboard for offending sources.
   - Pivot to logs using document `contentHash`.
   - Re-run `stella sources ingest --dry-run` with problematic payloads to validate fixes.
   - After remediation, run `stella aoc verify --since 24h` and confirm exit code `0`.
2. **Scheduled verification:**
   - Configure cron job to run `stella aoc verify --format json --export ...`.
   - Ship JSON to `aoc-verify` bucket and ingest into metrics using custom exporter.
   - Alert on missing exports (no file uploaded within 26 h).
3. **Offline kit validation:**
   - Use Offline Dashboard to ensure snapshots contain latest metrics.
   - Run verification reports locally and attach to bundle before distribution.

---

## 6 · Offline considerations

- Metrics exporters bundled with Offline Kit write to local Prometheus snapshots; sync them with central Grafana once connectivity is restored.
- CLI verification reports should be hashed (`sha256sum`) and archived for audit trails.
- Dashboards include offline data sources (`prometheus-offline`) switchable via dropdown.

---

## 7 · References

- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
- [Architecture overview](../modules/platform/architecture-overview.md)
- [Console AOC dashboard](../ui/console.md)
- [CLI AOC commands](../cli/cli-reference.md)
- [Concelier architecture](../modules/concelier/architecture.md)
- [Excititor architecture](../modules/excititor/architecture.md)
- [Scheduler Worker observability guide](../ops/scheduler-worker-operations.md)

---

## 8 · Compliance checklist

- [ ] Metrics documented with label sets and alert guidance.
- [ ] Tracing span taxonomy aligned with Concelier/Excititor implementation.
- [ ] Log schema matches structured logging contracts (traceId, tenant, source, contentHash).
- [ ] Grafana dashboard references verified and screenshots scheduled.
- [ ] Offline/air-gap workflow captured.
- [ ] Cross-links to AOC reference, console, and CLI docs included.
- [ ] Observability Guild sign-off scheduled (OWNER: @obs-guild, due 2025-10-28).

---

*Last updated: 2025-10-26 (Sprint 19).*