144 lines
8.7 KiB
Markdown
144 lines
8.7 KiB
Markdown
# Telemetry architecture
|
||
|
||
> Derived from Epic 15 – Observability & Forensics; details collector topology, storage profiles, forensic pipelines, and offline packaging.
|
||
|
||
## 1) Topology
|
||
|
||
- **Collector tier.** OpenTelemetry Collector instances deployed per environment (ingest TLS, GRPC/OTLP receivers, tail-based sampling). Config packages delivered via Offline Kit.
|
||
- **Processing pipelines.** Pipelines for traces, metrics, logs with processors (batch, tail sampling, attributes redaction, resource detection). Profiles: `default`, `forensic` (high-retention), `airgap` (file-based exporters).
|
||
- **Exporters.** OTLP to Prometheus/Tempo/Loki (online) or file/OTLP-HTTP to Offline Kit staging (air-gapped). Exporters are allow-listed to satisfy Sovereign readiness.
|
||
|
||
## 2) Storage
|
||
|
||
- **Prometheus** for metrics with remote-write support and retention windows (default 30 days, forensic 180 days).
|
||
- **Tempo** (or Jaeger all-in-one) for traces with block storage backend (S3-compatible or filesystem) and deterministic chunk manifests.
|
||
- **Loki** for logs stored in immutable chunks; index shards hashed for reproducibility.
|
||
- **Forensic archive** — periodic export of raw OTLP records into signed bundles (`otlp/metrics.pb`, `otlp/traces.pb`, `otlp/logs.pb`, `manifest.json`).
|
||
|
||
## 3) Pipelines & Guardrails
|
||
|
||
- **Redaction.** Attribute processors strip PII/secrets based on policy-managed allowed keys. Redaction profiles mirrored in Offline Kit.
|
||
- **Sampling.** Tail sampling by service/error; incident mode (triggered by Orchestrator) promotes services to 100 % sampling, extends retention, and toggles Notify alerts.
|
||
- **Alerting.** Prometheus rules/Dashboards packaged with Export Center: service SLOs, queue depth, policy run latency, ingestion AOC violations.
|
||
- **Sealed-mode guard.** `StellaOps.Telemetry.Core` enforces `IEgressPolicy` on OTLP exporters; when air-gap mode is sealed any non-loopback collector endpoints are automatically disabled and a structured warning with remediation is emitted.
|
||
|
||
## 4) APIs & integration
|
||
|
||
- `GET /telemetry/config/profile/{name}` — download collector config bundle (YAML + signature).
|
||
- `POST /telemetry/incidents/mode` — toggle incident sampling + forensic bundle generation.
|
||
- `GET /telemetry/exports/forensic/{window}` — stream signed OTLP bundles for compliance.
|
||
- CLI commands: `stella telemetry deploy --profile default`, `stella telemetry capture --window 24h --out bundle.tar.gz`.
|
||
|
||
## 5) Offline support
|
||
|
||
- Offline Kit ships collector binaries/config, bootstrap scripts, dashboards, alert rules, and OTLP replay tooling. Bundles include `manifest.json` with digests, DSSE signatures, and instructions.
|
||
- For offline environments, exporters write to local filesystem; operators transfer bundles to analysis workstation using signed manifests.
|
||
|
||
## 6) Observability of telemetry stack
|
||
|
||
- Meta-metrics: `collector_export_failures_total`, `telemetry_bundle_generation_seconds`, `telemetry_incident_mode{state}`.
|
||
- Health endpoints for collectors and storage clusters, plus dashboards for ingestion rate, retention, rule evaluations.
|
||
|
||
## 7) DORA Metrics
|
||
|
||
Stella Ops tracks the four key DORA (DevOps Research and Assessment) metrics for software delivery performance:
|
||
|
||
### 7.1) Metrics Tracked
|
||
|
||
- **Deployment Frequency** (`dora_deployments_total`, `dora_deployment_frequency_per_day`) — How often deployments occur per day/week.
|
||
- **Lead Time for Changes** (`dora_lead_time_hours`) — Time from commit to deployment in production.
|
||
- **Change Failure Rate** (`dora_deployment_failure_total`, `dora_change_failure_rate_percent`) — Percentage of deployments requiring rollback, hotfix, or failing.
|
||
- **Mean Time to Recovery (MTTR)** (`dora_time_to_recovery_hours`) — Average time to recover from incidents.
|
||
|
||
### 7.2) Performance Classification
|
||
|
||
The system classifies teams into DORA performance levels:
|
||
- **Elite**: On-demand deployments, <24h lead time, <15% CFR, <1h MTTR
|
||
- **High**: Weekly deployments, <1 week lead time, <30% CFR, <1 day MTTR
|
||
- **Medium**: Monthly deployments, <6 months lead time, <45% CFR, <1 week MTTR
|
||
- **Low**: Quarterly or less frequent deployments with higher failure rates
|
||
|
||
### 7.3) Integration Points
|
||
|
||
- `IDoraMetricsService` — Service interface for recording deployments and incidents
|
||
- `DoraMetrics` — OpenTelemetry-style metrics class with SLO breach tracking
|
||
- DI registration: `services.AddDoraMetrics(options => { ... })`
|
||
- Events are recorded when Release Orchestrator completes promotions or rollbacks
|
||
|
||
### 7.4) SLO Tracking
|
||
|
||
Configurable SLO targets via `DoraMetricsOptions`:
|
||
- `LeadTimeSloHours` (default: 24)
|
||
- `DeploymentFrequencySloPerDay` (default: 1)
|
||
- `ChangeFailureRateSloPercent` (default: 15)
|
||
- `MttrSloHours` (default: 1)
|
||
|
||
SLO breaches are recorded as `dora_slo_breach_total` with `metric` label.
|
||
|
||
### 7.5) Outcome Analytics and Attribution (Sprint 20260208_065)
|
||
|
||
Telemetry now includes deterministic executive outcome attribution built on top of the existing DORA event stream:
|
||
|
||
- `IOutcomeAnalyticsService` (`src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/IOutcomeAnalyticsService.cs`)
|
||
- `DoraOutcomeAnalyticsService` (`src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/DoraOutcomeAnalyticsService.cs`)
|
||
- Outcome report models (`src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/OutcomeAnalyticsModels.cs`)
|
||
|
||
Outcome attribution behavior:
|
||
|
||
- Produces `OutcomeExecutiveReport` for a fixed tenant/environment/time window with deterministic ordering.
|
||
- Adds MTTA support via `DoraIncidentEvent.AcknowledgedAt` and `TimeToAcknowledge`.
|
||
- Groups deployment outcomes by normalized pipeline (`pipeline-a`, `pipeline-b`, `unknown`) with per-pipeline change failure rate and median lead time.
|
||
- Groups incidents by severity with resolved/acknowledged counts plus MTTA/MTTR aggregates.
|
||
- Produces daily cohort slices across the requested date range for executive trend views.
|
||
|
||
Dependency injection integration:
|
||
|
||
- `TelemetryServiceCollectionExtensions.AddDoraMetrics(...)` now also registers `IOutcomeAnalyticsService`, so existing telemetry entry points automatically expose attribution reporting without additional module wiring.
|
||
|
||
Verification coverage:
|
||
|
||
- `src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/OutcomeAnalyticsServiceTests.cs`
|
||
- `src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/DoraMetricsServiceTests.cs`
|
||
- Full telemetry core test suite pass (`262` tests) remains green after integration.
|
||
|
||
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.
|
||
|
||
## 8) Federation DSSE Security Posture (Updated 2026-03-04)
|
||
|
||
Status:
|
||
|
||
- Advisory gap `TEL-001` is closed. Federation consent and bundle paths now emit signed DSSE envelopes instead of payload passthrough placeholders.
|
||
|
||
Implemented contract:
|
||
|
||
- Consent and bundle envelopes now use explicit DSSE JSON structure: `payloadType`, base64 `payload`, and `signatures[]` (`keyid`, `sig`).
|
||
- Consent proofs and bundle summaries carry signer identity metadata (`SignerKeyId`) for auditability.
|
||
- Bundle payload canonicalization is deterministic for identical logical inputs:
|
||
- bucket ordering: `cveId` (ordinal), then `noisyCount` (descending), `artifactCount`, `observationCount`
|
||
- deterministic bundle ID derivation from canonical payload seed + fixed clock input
|
||
- Bundle verification enforces:
|
||
- envelope digest integrity (`sha256:` over envelope bytes)
|
||
- payload type match
|
||
- trusted-key signature verification
|
||
- consent digest linkage (`consentDigest` in payload must match `ConsentDsseDigest`)
|
||
|
||
Signer/verifier integration and fallback:
|
||
|
||
- Federation now uses explicit abstractions:
|
||
- `IFederationDsseEnvelopeSigner`
|
||
- `IFederationDsseEnvelopeVerifier`
|
||
- Default adapter: `HmacFederationDsseEnvelopeService` (offline-safe HMAC-SHA256 DSSE sign/verify using local trusted key map in `FederatedTelemetryOptions`).
|
||
- Failure mode is deterministic and auditable:
|
||
- signing failures throw `FederationSignatureException` with stable error codes (for example `federation.dsse.sign_failed`, `federation.dsse.signer_unavailable`)
|
||
- optional unsigned fallback (`AllowUnsignedDsseFallback`) emits envelopes tagged with `offline-unsigned-fallback` for explicit operator visibility.
|
||
|
||
Verification evidence:
|
||
|
||
- `dotnet test src/Telemetry/StellaOps.Telemetry.Federation.Tests/StellaOps.Telemetry.Federation.Tests.csproj -m:1 -v minimal`
|
||
- Result: `47` passed, `0` failed.
|
||
- Coverage includes payload tamper, signature tamper, wrong-key verification failure, consent expiry + signature validity combination, and deterministic replay digest checks.
|
||
|
||
Tracking sprint:
|
||
|
||
- `docs/implplan/SPRINT_20260304_307_Telemetry_federation_dsse_bundle_hardening.md`
|