git.stella-ops.org/docs/observability/policy.md

Policy Engine Observability

Audience: Observability Guild, SRE/Platform operators, Policy Guild.
Scope: Metrics, logs, traces, dashboards, alerting, sampling, and incident workflows for the Policy Engine service (Sprint 20).
Prerequisites: Policy Engine v2 deployed with OpenTelemetry exporters enabled (observability:enabled=true in config).

---

1 · Instrumentation Overview

• Telemetry stack: OpenTelemetry SDK (metrics + traces), Serilog structured logging, OTLP exporters → Collector → Prometheus/Loki/Tempo.
• Namespace conventions: policy.* for metrics/traces/log categories; labels use tenant, policy, mode, runId.
• Sampling: Default 10 % trace sampling, 1 % rule-hit log sampling; incident mode overrides to 100 % (see §6).
• Correlation IDs: Every API request gets traceId + requestId. CLI/UI display IDs to streamline support.

---

2 · Metrics

2.1 Run Pipeline

Metric	Type	Labels	Notes
policy_run_seconds	Histogram	tenant, policy, mode (full, incremental, simulate)	P95 target ≤ 5 min incremental, ≤ 30 min full.
policy_run_queue_depth	Gauge	tenant	Number of pending jobs per tenant (updated each enqueue/dequeue).
policy_run_failures_total	Counter	tenant, policy, reason (err_pol_*, network, cancelled)	Aligns with error codes.
policy_run_retries_total	Counter	tenant, policy	Helps identify noisy sources.
policy_run_inputs_pending_bytes	Gauge	tenant	Size of buffered change batches awaiting run.

2.2 Evaluator Insights

Metric	Type	Labels	Notes
policy_rules_fired_total	Counter	tenant, policy, rule	Increment per rule match (sampled).
policy_vex_overrides_total	Counter	tenant, policy, vendor, justification	Tracks VEX precedence decisions.
policy_suppressions_total	Counter	tenant, policy, action (ignore, warn, quiet)	Audits suppression usage.
policy_selection_batch_duration_seconds	Histogram	tenant, policy	Measures joiner performance.
policy_materialization_conflicts_total	Counter	tenant, policy	Non-zero indicates optimistic concurrency retries.

2.3 API Surface

Metric	Type	Labels	Notes
policy_api_requests_total	Counter	endpoint, method, status	Exposed via Minimal API instrumentation.
policy_api_latency_seconds	Histogram	endpoint, method	Budget ≤ 250 ms for GETs, ≤ 1 s for POSTs.
policy_api_rate_limited_total	Counter	endpoint	Tied to throttles (429).

2.4 Queue & Change Streams

Metric	Type	Labels	Notes
policy_queue_leases_active	Gauge	tenant	Number of leased jobs.
policy_queue_lease_expirations_total	Counter	tenant	Alerts when workers fail to ack.
policy_delta_backlog_age_seconds	Gauge	tenant, source (concelier, excititor, sbom)	Age of oldest unprocessed change event.

---

3 · Logs

• Format: JSON (Serilog). Core fields: timestamp, level, message, policyId, policyVersion, tenant, runId, rule, traceId, env.sealed, error.code.
• Log categories:
  • policy.run (queue lifecycle, run begin/end, stats)
  • policy.evaluate (batch execution summaries; rule-hit sampling)
  • policy.materialize (Mongo operations, conflicts, retries)
  • policy.simulate (diff results, CLI invocation metadata)
  • policy.lifecycle (submit/review/approve events)
• Sampling: Rule-hit logs sample 1 % by default; toggled to 100 % in incident mode or when --trace flag used in CLI.
• PII: No user secrets recorded; user identities referenced as user:<id> or group:<id> only.

---

4 · Traces

• Spans emit via OpenTelemetry instrumentation.
• Primary spans:
  • policy.api – wraps HTTP request, records endpoint, status, scope.
  • policy.select – change stream ingestion and batch assembly (attributes: candidateCount, cursor).
  • policy.evaluate – evaluation batch (attributes: batchSize, ruleHits, severityChanges).
  • policy.materialize – Mongo writes (attributes: writes, historyWrites, retryCount).
  • policy.simulate – simulation diff generation (attributes: sbomCount, diffAdded, diffRemoved).
• Trace context propagated to CLI via response headers traceparent; UI surfaces in run detail view.
• Incident mode forces span sampling to 100 % and extends retention via Collector config override.

---

5 · Dashboards

5.1 Policy Runs Overview

Widgets:

• Run duration histogram (per mode/tenant).
• Queue depth + backlog age line charts.
• Failure rate stacked by error code.
• Incremental backlog heatmap (policy × age).
• Active vs scheduled runs table.

5.2 Rule Impact & VEX

• Top N rules by firings (bar chart).
• VEX overrides by vendor/justification (stacked chart).
• Suppression usage (pie + table with justifications).
• Quieted findings trend (line).

5.3 Simulation & Approval Health

• Simulation diff histogram (added vs removed).
• Pending approvals by age (table with SLA colour coding).
• Compliance checklist status (lint, determinism CI, simulation evidence).

Placeholders for Grafana panels should be replaced with actual screenshots once dashboards land (../assets/policy-observability/*.png).

---

6 · Alerting

Alert	Condition	Suggested Action
PolicyRunSlaBreach	policy_run_seconds{mode="incremental"} P95 > 300 s for 3 windows	Check queue depth, upstream services, scale worker pool.
PolicyQueueStuck	policy_delta_backlog_age_seconds > 600	Investigate change stream connectivity.
DeterminismMismatch	Run status failed with ERR_POL_004 OR CI replay diff	Switch to incident sampling, gather replay bundle, notify Policy Guild.
SimulationDrift	CLI/CI simulation exit 20 (blocking diff) over threshold	Review policy changes before approval.
VexOverrideSpike	policy_vex_overrides_total > configured baseline (per vendor)	Verify upstream VEX feed; ensure justification codes expected.
SuppressionSurge	policy_suppressions_total increase > 3σ vs baseline	Audit new suppress rules; check approvals.

Alerts integrate with Notifier channels (policy.alerts) and Ops on-call rotations.

---

7 · Incident Mode & Forensics

• Toggle via POST /api/policy/incidents/activate (requires policy:operate scope).
• Effects:
  • Trace sampling → 100 %.
  • Rule-hit log sampling → 100 %.
  • Retention window extended to 30 days for incident duration.
  • policy.incident.activated event emitted (Console + Notifier banners).
• Post-incident tasks:
  • stella policy run replay for affected runs; attach bundles to incident record.
  • Restore sampling defaults with .../deactivate.
  • Update incident checklist in /docs/policy/lifecycle.md (section 8) with findings.

---

8 · Integration Points

• Authority: Exposes metric policy_scope_denied_total for failed authorisation; correlate with policy_api_requests_total.
• Concelier/Excititor: Shared trace IDs propagate via gRPC metadata to help debug upstream latency.
• Scheduler: Future integration will push run queues into shared scheduler dashboards (planned in SCHED-MODELS-20-002).
• Offline Kit: CLI exports logs + metrics snapshots (stella offline bundle metrics) for air-gapped audits.

---

9 · Compliance Checklist

• Metrics registered: All metrics listed above exported and documented in Grafana dashboards.
• Alert policies configured: Ops or Observability Guild created alerts matching table in §6.
• Sampling overrides tested: Incident mode toggles verified in staging; retention roll-back rehearsed.
• Trace propagation validated: CLI/UI display trace IDs and allow copy for support.
• Log scrubbing enforced: Unit tests guarantee no secrets/PII in logs; sampling respects configuration.
• Offline capture rehearsed: Metrics/log snapshot commands executed in sealed environment.
• Docs cross-links: Links to architecture, runs, lifecycle, CLI, API docs verified.

---

Last updated: 2025-10-26 (Sprint 20).