stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
b565e559423cd02f65329e77f65e38ed9a02c978
git.stella-ops.org/docs/modules/telemetry/architecture.md

8.7 KiB
Raw Blame History

Telemetry architecture

Derived from Epic 15 – Observability & Forensics; details collector topology, storage profiles, forensic pipelines, and offline packaging.

1) Topology

  • Collector tier. OpenTelemetry Collector instances deployed per environment (ingest TLS, GRPC/OTLP receivers, tail-based sampling). Config packages delivered via Offline Kit.
  • Processing pipelines. Pipelines for traces, metrics, logs with processors (batch, tail sampling, attributes redaction, resource detection). Profiles: default, forensic (high-retention), airgap (file-based exporters).
  • Exporters. OTLP to Prometheus/Tempo/Loki (online) or file/OTLP-HTTP to Offline Kit staging (air-gapped). Exporters are allow-listed to satisfy Sovereign readiness.

2) Storage

  • Prometheus for metrics with remote-write support and retention windows (default 30 days, forensic 180 days).
  • Tempo (or Jaeger all-in-one) for traces with block storage backend (S3-compatible or filesystem) and deterministic chunk manifests.
  • Loki for logs stored in immutable chunks; index shards hashed for reproducibility.
  • Forensic archive — periodic export of raw OTLP records into signed bundles (otlp/metrics.pb, otlp/traces.pb, otlp/logs.pb, manifest.json).

3) Pipelines & Guardrails

  • Redaction. Attribute processors strip PII/secrets based on policy-managed allowed keys. Redaction profiles mirrored in Offline Kit.
  • Sampling. Tail sampling by service/error; incident mode (triggered by Orchestrator) promotes services to 100 % sampling, extends retention, and toggles Notify alerts.
  • Alerting. Prometheus rules/Dashboards packaged with Export Center: service SLOs, queue depth, policy run latency, ingestion AOC violations.
  • Sealed-mode guard. StellaOps.Telemetry.Core enforces IEgressPolicy on OTLP exporters; when air-gap mode is sealed any non-loopback collector endpoints are automatically disabled and a structured warning with remediation is emitted.

4) APIs & integration

  • GET /telemetry/config/profile/{name} — download collector config bundle (YAML + signature).
  • POST /telemetry/incidents/mode — toggle incident sampling + forensic bundle generation.
  • GET /telemetry/exports/forensic/{window} — stream signed OTLP bundles for compliance.
  • CLI commands: stella telemetry deploy --profile default, stella telemetry capture --window 24h --out bundle.tar.gz.

5) Offline support

  • Offline Kit ships collector binaries/config, bootstrap scripts, dashboards, alert rules, and OTLP replay tooling. Bundles include manifest.json with digests, DSSE signatures, and instructions.
  • For offline environments, exporters write to local filesystem; operators transfer bundles to analysis workstation using signed manifests.

6) Observability of telemetry stack

  • Meta-metrics: collector_export_failures_total, telemetry_bundle_generation_seconds, telemetry_incident_mode{state}.
  • Health endpoints for collectors and storage clusters, plus dashboards for ingestion rate, retention, rule evaluations.

7) DORA Metrics

Stella Ops tracks the four key DORA (DevOps Research and Assessment) metrics for software delivery performance:

7.1) Metrics Tracked

  • Deployment Frequency (dora_deployments_total, dora_deployment_frequency_per_day) — How often deployments occur per day/week.
  • Lead Time for Changes (dora_lead_time_hours) — Time from commit to deployment in production.
  • Change Failure Rate (dora_deployment_failure_total, dora_change_failure_rate_percent) — Percentage of deployments requiring rollback, hotfix, or failing.
  • Mean Time to Recovery (MTTR) (dora_time_to_recovery_hours) — Average time to recover from incidents.

7.2) Performance Classification

The system classifies teams into DORA performance levels:

  • Elite: On-demand deployments, <24h lead time, <15% CFR, <1h MTTR
  • High: Weekly deployments, <1 week lead time, <30% CFR, <1 day MTTR
  • Medium: Monthly deployments, <6 months lead time, <45% CFR, <1 week MTTR
  • Low: Quarterly or less frequent deployments with higher failure rates

7.3) Integration Points

  • IDoraMetricsService — Service interface for recording deployments and incidents
  • DoraMetrics — OpenTelemetry-style metrics class with SLO breach tracking
  • DI registration: services.AddDoraMetrics(options => { ... })
  • Events are recorded when Release Orchestrator completes promotions or rollbacks

7.4) SLO Tracking

Configurable SLO targets via DoraMetricsOptions:

  • LeadTimeSloHours (default: 24)
  • DeploymentFrequencySloPerDay (default: 1)
  • ChangeFailureRateSloPercent (default: 15)
  • MttrSloHours (default: 1)

SLO breaches are recorded as dora_slo_breach_total with metric label.

7.5) Outcome Analytics and Attribution (Sprint 20260208_065)

Telemetry now includes deterministic executive outcome attribution built on top of the existing DORA event stream:

  • IOutcomeAnalyticsService (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/IOutcomeAnalyticsService.cs)
  • DoraOutcomeAnalyticsService (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/DoraOutcomeAnalyticsService.cs)
  • Outcome report models (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/OutcomeAnalyticsModels.cs)

Outcome attribution behavior:

  • Produces OutcomeExecutiveReport for a fixed tenant/environment/time window with deterministic ordering.
  • Adds MTTA support via DoraIncidentEvent.AcknowledgedAt and TimeToAcknowledge.
  • Groups deployment outcomes by normalized pipeline (pipeline-a, pipeline-b, unknown) with per-pipeline change failure rate and median lead time.
  • Groups incidents by severity with resolved/acknowledged counts plus MTTA/MTTR aggregates.
  • Produces daily cohort slices across the requested date range for executive trend views.

Dependency injection integration:

  • TelemetryServiceCollectionExtensions.AddDoraMetrics(...) now also registers IOutcomeAnalyticsService, so existing telemetry entry points automatically expose attribution reporting without additional module wiring.

Verification coverage:

  • src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/OutcomeAnalyticsServiceTests.cs
  • src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/DoraMetricsServiceTests.cs
  • Full telemetry core test suite pass (262 tests) remains green after integration.

Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.

8) Federation DSSE Security Posture (Updated 2026-03-04)

Status:

  • Advisory gap TEL-001 is closed. Federation consent and bundle paths now emit signed DSSE envelopes instead of payload passthrough placeholders.

Implemented contract:

  • Consent and bundle envelopes now use explicit DSSE JSON structure: payloadType, base64 payload, and signatures[] (keyid, sig).
  • Consent proofs and bundle summaries carry signer identity metadata (SignerKeyId) for auditability.
  • Bundle payload canonicalization is deterministic for identical logical inputs:
    • bucket ordering: cveId (ordinal), then noisyCount (descending), artifactCount, observationCount
    • deterministic bundle ID derivation from canonical payload seed + fixed clock input
  • Bundle verification enforces:
    • envelope digest integrity (sha256: over envelope bytes)
    • payload type match
    • trusted-key signature verification
    • consent digest linkage (consentDigest in payload must match ConsentDsseDigest)

Signer/verifier integration and fallback:

  • Federation now uses explicit abstractions:
    • IFederationDsseEnvelopeSigner
    • IFederationDsseEnvelopeVerifier
  • Default adapter: HmacFederationDsseEnvelopeService (offline-safe HMAC-SHA256 DSSE sign/verify using local trusted key map in FederatedTelemetryOptions).
  • Failure mode is deterministic and auditable:
    • signing failures throw FederationSignatureException with stable error codes (for example federation.dsse.sign_failed, federation.dsse.signer_unavailable)
    • optional unsigned fallback (AllowUnsignedDsseFallback) emits envelopes tagged with offline-unsigned-fallback for explicit operator visibility.

Verification evidence:

  • dotnet test src/Telemetry/StellaOps.Telemetry.Federation.Tests/StellaOps.Telemetry.Federation.Tests.csproj -m:1 -v minimal
  • Result: 47 passed, 0 failed.
  • Coverage includes payload tamper, signature tamper, wrong-key verification failure, consent expiry + signature validity combination, and deterministic replay digest checks.

Tracking sprint:

  • docs/implplan/SPRINT_20260304_307_Telemetry_federation_dsse_bundle_hardening.md