git.stella-ops.org/docs/observability/observability.md

AOC Observability Guide

Audience: Observability Guild, Concelier/Excititor SREs, platform operators.
Scope: Metrics, traces, logs, dashboards, and runbooks introduced as part of the Aggregation-Only Contract (AOC) rollout (Sprint 19).

This guide captures the canonical signals emitted by Concelier and Excititor once AOC guards are active. It explains how to consume the metrics in dashboards, correlate traces/logs for incident triage, and operate in offline environments. Pair this guide with the AOC reference and architecture overview.

---

1 · Metrics

Metric	Type	Labels	Description
ingestion_write_total	Counter	source, tenant, result (ok, reject, noop)	Counts write attempts to advisory_raw/vex_raw. Rejects correspond to guard failures.
ingestion_latency_seconds	Histogram	source, tenant, phase (fetch, transform, write)	Measures end-to-end runtime for ingestion stages. Use quantile=0.95 for alerting.
aoc_violation_total	Counter	source, tenant, code (ERR_AOC_00x)	Total guard violations bucketed by error code. Drives dashboard pills and alert thresholds.
ingestion_signature_verified_total	Counter	source, tenant, result (ok, fail, skipped)	Tracks signature/checksum verification outcomes.
advisory_revision_count	Gauge	source, tenant	Supersedes depth for raw documents; spikes indicate noisy upstream feeds.
verify_runs_total	Counter	tenant, initiator (ui, cli, api, scheduled)	How many stella aoc verify or /aoc/verify runs executed.
verify_duration_seconds	Histogram	tenant, initiator	Runtime of verification jobs; use P95 to detect regressions.

1.1 Alerts

• Violation spike: Alert when increase(aoc_violation_total[15m]) > 0 for critical sources. Page SRE if code="ERR_AOC_005" (signature failure) or ERR_AOC_001 persists > 30 min.
• Stale ingestion: Alert when max_over_time(ingestion_latency_seconds_sum / ingestion_latency_seconds_count)[30m] exceeds 30 s or if ingestion_write_total has no growth for > 60 min.
• Signature drop: Warn when rate(ingestion_signature_verified_total{result="fail"}[1h]) > 0.

---

2 · Traces

2.1 Span taxonomy

Span name	Parent	Key attributes
ingest.fetch	job root span	source, tenant, uri, contentHash
ingest.transform	ingest.fetch	documentType (csaf, osv, vex), payloadBytes
ingest.write	ingest.transform	collection (advisory_raw, vex_raw), result (ok, reject)
aoc.guard	ingest.write	code (on violation), violationCount, supersedes
verify.run	verification job root	tenant, window.from, window.to, sources, violations

2.2 Trace usage

• Correlate UI dashboard entries with traces via traceId surfaced in violation drawers (docs/ui/console.md).
• Use aoc.guard spans to inspect guard payload snapshots. Sensitive fields are redacted automatically; raw JSON lives in secure logs only.
• For scheduled verification, filter traces by initiator="scheduled" to compare runtimes pre/post change.

---

3 · Logs

Structured logs include the following keys (JSON):

Key	Description
traceId	Matches OpenTelemetry trace/span IDs for cross-system correlation.
tenant	Tenant identifier enforced by Authority middleware.
source.vendor	Logical source (e.g., redhat, ubuntu, osv, ghsa).
upstream.upstreamId	Vendor-provided ID (CVE, GHSA, etc.).
contentHash	sha256: digest of the raw document.
violation.code	Present when guard rejects ERR_AOC_00x.
verification.window	Present on /aoc/verify job logs.

Logs are shipped to the central Loki/Elasticsearch cluster. Use the template query:

{app="concelier-web"} | json | violation_code != ""

to spot active AOC violations.

---

4 · Dashboards

Primary Grafana dashboard: “AOC Ingestion Health” (dashboards/aoc-ingestion.json). Panels include:

1. Sources overview: table fed by ingestion_write_total and ingestion_latency_seconds (mirrors Console tiles).
2. Violation trend: stacked bar chart of aoc_violation_total per code.
3. Signature success rate: timeseries derived from ingestion_signature_verified_total.
4. Supersedes depth: gauge showing advisory_revision_count P95.
5. Verification runs: histogram and latency boxplot using verify_runs_total / verify_duration_seconds.

Secondary dashboards:

• AOC Alerts (Ops view): summarises active alerts, last verify run, and links to incident runbook.
• Offline Mode Dashboard: fed from Offline Kit imports; highlights snapshot age and queued verification jobs.

Update docs/assets/dashboards/ with screenshots when Grafana capture pipeline produces the latest renders.

---

5 · Operational workflows

1. During ingestion incident:
   • Check Console dashboard for offending sources.
   • Pivot to logs using document contentHash.
   • Re-run stella sources ingest --dry-run with problematic payloads to validate fixes.
   • After remediation, run stella aoc verify --since 24h and confirm exit code 0.
2. Scheduled verification:
   • Configure cron job to run stella aoc verify --format json --export ....
   • Ship JSON to aoc-verify bucket and ingest into metrics using custom exporter.
   • Alert on missing exports (no file uploaded within 26 h).
3. Offline kit validation:
   • Use Offline Dashboard to ensure snapshots contain latest metrics.
   • Run verification reports locally and attach to bundle before distribution.

---

6 · Offline considerations

• Metrics exporters bundled with Offline Kit write to local Prometheus snapshots; sync them with central Grafana once connectivity is restored.
• CLI verification reports should be hashed (sha256sum) and archived for audit trails.
• Dashboards include offline data sources (prometheus-offline) switchable via dropdown.

---

7 · References

• Aggregation-Only Contract reference
• Architecture overview
• Console AOC dashboard
• CLI AOC commands
• Concelier architecture
• Excititor architecture
• Scheduler Worker observability guide

---

8 · Compliance checklist

• Metrics documented with label sets and alert guidance.
• Tracing span taxonomy aligned with Concelier/Excititor implementation.
• Log schema matches structured logging contracts (traceId, tenant, source, contentHash).
• Grafana dashboard references verified and screenshots scheduled.
• Offline/air-gap workflow captured.
• Cross-links to AOC reference, console, and CLI docs included.
• Observability Guild sign-off scheduled (OWNER: @obs-guild, due 2025-10-28).

---

Last updated: 2025-10-26 (Sprint 19).