stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity
Files
95daa159c48e9b50f70dee1024eccc935da52145
git.stella-ops.org/docs/observability/ui-telemetry.md
master 14617e9c3b feat: Implement Scheduler Worker Options and Planner Loop 
- Added `SchedulerWorkerOptions` class to encapsulate configuration for the scheduler worker.
- Introduced `PlannerBackgroundService` to manage the planner loop, fetching and processing planning runs.
- Created `PlannerExecutionService` to handle the execution logic for planning runs, including impact targeting and run persistence.
- Developed `PlannerExecutionResult` and `PlannerExecutionStatus` to standardize execution outcomes.
- Implemented validation logic within `SchedulerWorkerOptions` to ensure proper configuration.
- Added documentation for the planner loop and impact targeting features.
- Established health check endpoints and authentication mechanisms for the Signals service.
- Created unit tests for the Signals API to ensure proper functionality and response handling.
- Configured options for authority integration and fallback authentication methods.
2025-10-27 09:46:31 +02:00

13 KiB
Raw Blame History

Console Observability

Audience: Observability Guild, Console Guild, SRE/operators.
Scope: Metrics, logs, traces, dashboards, alerting, feature flags, and offline workflows for the StellaOps Console (Sprint 23).
Prerequisites: Console deployed with metrics enabled (CONSOLE_METRICS_ENABLED=true) and OTLP exporters configured (OTEL_EXPORTER_OTLP_*).

1 · Instrumentation Overview

  • Telemetry stack: OpenTelemetry Web SDK (browser) + Console telemetry bridge → OTLP collector (Tempo/Prometheus/Loki). Server-side endpoints expose /metrics (Prometheus) and /health/*.
  • Sampling: Front-end spans sample at 5 % by default (OTEL_TRACES_SAMPLER=parentbased_traceidratio). Metrics are un-sampled; log sampling is handled per category (§3).
  • Correlation IDs: Every API call carries x-stellaops-correlation-id; structured UI events mirror that value so operators can follow a request across gateway, backend, and UI.
  • Scope gating: Operators need the ui.telemetry scope to view live charts in the Admin workspace; the scope also controls access to /console/telemetry SSE streams.

2 · Metrics

2.1 Experience & Navigation

Metric Type Labels Notes
ui_route_render_seconds Histogram route, tenant, device (desktop,tablet) Time between route activation and first interactive paint. Target P95 ≤ 1.5 s (cached).
ui_request_duration_seconds Histogram service, method, status, tenant Gateway proxy timing for backend calls performed by the console. Alerts when backend latency degrades.
ui_filter_apply_total Counter route, filter, tenant Increments when a global filter or context chip is applied. Used to track adoption of saved views.
ui_tenant_switch_total Counter fromTenant, toTenant, trigger (picker, shortcut, link) Emitted after a successful tenant switch; correlates with Authority ui.tenant.switch logs.
ui_offline_banner_seconds Histogram reason (authority, manifest, gateway), tenant Duration of offline banner visibility; integrate with air-gap SLAs.

2.2 Security & Session

Metric Type Labels Notes
ui_dpop_failure_total Counter endpoint, reason (nonce, jkt, clockSkew) Raised when DPoP validation fails; pair with Authority audit trail.
ui_fresh_auth_prompt_total Counter action (token.revoke, policy.activate, client.create), tenant Counts fresh-auth modals; backlog above baseline indicates workflow friction.
ui_fresh_auth_failure_total Counter action, reason (timeout,cancelled,auth_error) Optional metric (set CONSOLE_FRESH_AUTH_METRICS=true when feature flag lands).

2.3 Downloads & Offline Kit

Metric Type Labels Notes
ui_download_manifest_refresh_seconds Histogram tenant, channel (edge,stable,airgap) Time to fetch and verify downloads manifest. Target < 3 s.
ui_download_export_queue_depth Gauge tenant, artifactType (sbom,policy,attestation,console) Mirrors /console/downloads queue depth; triggers when offline bundles lag.
ui_download_command_copied_total Counter tenant, artifactType Increments when users copy CLI commands from the UI. Useful to observe CLI parity adoption.

2.4 Telemetry Emission & Errors

Metric Type Labels Notes
ui_telemetry_batch_failures_total Counter transport (otlp-http,otlp-grpc), reason Emitted by OTLP bridge when batches fail. Enable via CONSOLE_METRICS_VERBOSE=true.
ui_telemetry_queue_depth Gauge priority (normal,high), tenant Browser-side buffer depth; monitor for spikes under degraded collectors.

Scraping tips:

  • Enable /metrics via CONSOLE_METRICS_ENABLED=true.
  • Set OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.collector:4318 and relevant headers (OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer <token>).
  • For air-gapped sites, point the exporter to the Offline Kit collector (localhost:4318) and forward the metrics snapshot using stella offline bundle metrics.

3 · Logs

  • Format: JSON via Console log bridge; emitted to stdout and optional OTLP log exporter. Core fields: timestamp, level, action, route, tenant, subject, correlationId, dpop.jkt, device, offlineMode.
  • Categories:
    • ui.action – general user interactions (route changes, command palette, filter updates). Sampled 50 % by default; override with feature flag telemetry.logVerbose.
    • ui.tenant.switch – always logged; includes fromTenant, toTenant, tokenId, and Authority audit correlation.
    • ui.download.commandCopied – download commands copied; includes artifactId, digest, manifestVersion.
    • ui.security.anomaly – DPoP mismatches, tenant header errors, CSP violations (level = Warning).
    • ui.telemetry.failure – OTLP export errors; include httpStatus, batchSize, retryCount.
  • PII handling: Full emails are scrubbed; only hashed values (user:<sha256>) appear unless ui.admin + fresh-auth were granted for the action (still redacted in logs).
  • Retention: Recommended 14 days for connected sites, 30 days for sealed/air-gap audits. Ship logs to Loki/Elastic with ingest label service="stellaops-web-ui".

4 · Traces

  • Span names & attributes:
    • ui.route.transition – wraps route navigation; attributes: route, tenant, renderMillis, prefetchHit.
    • ui.api.fetch – HTTP fetch to backend; attributes: service, endpoint, status, networkTime.
    • ui.sse.stream – Server-sent event subscriptions (status ticker, runs); attributes: channel, connectedMillis, reconnects.
    • ui.telemetry.batch – Browser OTLP flush; attributes: batchSize, success, retryCount.
    • ui.policy.action – Policy workspace actions (simulate, approve, activate) per docs/ui/policy-editor.md.
  • Propagation: Spans use W3C traceparent; gateway echoes header to backend APIs so traces stitch across UI → gateway → service.
  • Sampling controls: OTEL_TRACES_SAMPLER_ARG (ratio) and feature flag telemetry.forceSampling (sets to 100 % for incident debugging).
  • Viewing traces: Grafana Tempo or Jaeger via collector. Filter by service.name = stellaops-console. For cross-service debugging, filter on correlationId and tenant.

5 · Dashboards

5.1 Experience Overview

Panels:

  • Route render histogram (P50/P90/P99) by route.
  • Backend call latency stacked by service (ui_request_duration_seconds).
  • Offline banner duration trend (ui_offline_banner_seconds).
  • Tenant switch volume vs failure rate (overlay ui_dpop_failure_total).
  • Command palette usage (ui_filter_apply_total + ui.action log counts).

5.2 Downloads & Offline Kit

  • Manifest refresh time chart (per channel).
  • Export queue depth gauge with alert thresholds.
  • CLI command adoption (bar chart per artifact type, using ui_download_command_copied_total).
  • Offline parity banner occurrences (downloads.offlineParity flag from API → derived metric).
  • Last Offline Kit import timestamp (join with Downloads API metadata).

5.3 Security & Session

  • Fresh-auth prompt counts vs success/fail ratios.
  • DPoP failure stacked by reason.
  • Tenant mismatch warnings (from ui.security.anomaly logs).
  • Scope usage heatmap (derived from Authority audit events + UI logs).
  • CSP violation counts (browser securitypolicyviolation listener forwarded to logs).

Capture screenshots for Grafana once dashboards stabilise (docs/assets/ui/observability/*.png). Replace placeholders before releasing the doc.

6 · Alerting

Alert Condition Suggested Action
ConsoleLatencyHigh ui_route_render_seconds_bucket{le="1.5"} drops below 0.95 for 3 intervals Inspect route splits, check backend latencies, review CDN cache.
BackendLatencyHigh ui_request_duration_seconds_sum / ui_request_duration_seconds_count > 1 s for any service Correlate with gateway/service dashboards; escalate to owning guild.
TenantSwitchFailures Increase in ui_dpop_failure_total or ui.security.anomaly (tenant mismatch) > 3/min Validate Authority issuer, check clock skew, confirm tenant config.
FreshAuthLoop ui_fresh_auth_prompt_total spikes with matching ui_fresh_auth_failure_total Review Authority /fresh-auth endpoint, session timeout config, UX regressions.
OfflineBannerLong ui_offline_banner_seconds P95 > 120 s Investigate Authority/gateway availability; verify Offline Kit freshness.
DownloadsBacklog ui_download_export_queue_depth > 5 for 10 min OR queue age > alert threshold Ping Downloads service, ensure manifest pipeline (DOWNLOADS-CONSOLE-23-001) is healthy.
TelemetryExportErrors ui_telemetry_batch_failures_total > 0 for ≥5 min Check collector health, credentials, or TLS trust.

Integrate alerts with Notifier (ui.alerts) or existing Ops channels. Tag incidents with component=console for correlation.

7 · Feature Flags & Configuration

Flag / Env Var Purpose Default
CONSOLE_FEATURE_FLAGS Enables UI modules (runs, downloads, policies, telemetry). Telemetry panel requires telemetry. runs,downloads,policies
CONSOLE_METRICS_ENABLED Exposes /metrics for Prometheus scrape. true
CONSOLE_METRICS_VERBOSE Emits additional batching metrics (ui_telemetry_*). false
CONSOLE_LOG_LEVEL Minimum log level (Information, Debug). Use Debug for incident sampling. Information
CONSOLE_METRICS_SAMPLING (planned) Controls front-end span sampling ratio. Document once released. 0.05
OTEL_EXPORTER_OTLP_ENDPOINT Collector URL; supports HTTPS. unset
OTEL_EXPORTER_OTLP_HEADERS Comma-separated headers (auth). unset
OTEL_EXPORTER_OTLP_INSECURE Allow HTTP (dev only). false
OTEL_SERVICE_NAME Service tag for traces/logs. Set to stellaops-console. auto
CONSOLE_TELEMETRY_SSE_ENABLED Enables /console/telemetry SSE feed for dashboards. true

Feature flag changes should be tracked in release notes and mirrored in /docs/ui/navigation.md (shortcuts may change when modules toggle).

8 · Offline / Air-Gapped Workflow

  • Mirror the console image and telemetry collector as part of the Offline Kit (see /docs/install/docker.md §4).
  • Scrape metrics locally via curl -k https://console.local/metrics > metrics.prom; archive alongside logs for audits.
  • Use stella offline kit import to keep the downloads manifest in sync; dashboards display staleness using ui_download_manifest_refresh_seconds.
  • When collectors are unavailable, console queues OTLP batches (up to 5 min) and exposes backlog through ui_telemetry_queue_depth; export queue metrics to prove no data loss.
  • After reconnecting, run stella console status --telemetry (CLI parity pending; see DOCS-CONSOLE-23-014) or verify ui_telemetry_batch_failures_total resets to zero.
  • Retain telemetry bundles for 30 days per compliance guidelines; include Grafana JSON exports in audit packages.

9 · Compliance Checklist

  • /metrics scraped in staging & production; dashboards display ui_route_render_seconds, ui_request_duration_seconds, and downloads metrics.
  • OTLP traces/logs confirmed end-to-end (collector, Tempo/Loki).
  • Alert rules from §6 implemented in monitoring stack with runbooks linked.
  • Feature flags documented and change-controlled; telemetry disabled only with approval.
  • DPoP/fresh-auth anomalies correlated with Authority audit logs during drill.
  • Offline capture workflow exercised; evidence stored in audit vault.
  • Screenshots of Grafana dashboards committed once they stabilise (update references).
  • Cross-links verified (docs/deploy/console.md, docs/security/console-security.md, docs/ui/downloads.md, docs/ui/console-overview.md).

10 · References

  • /docs/deploy/console.md – Metrics endpoint, OTLP config, health checks.
  • /docs/security/console-security.md – Security metrics & alert hints.
  • /docs/ui/console-overview.md – Telemetry primitives and performance budgets.
  • /docs/ui/downloads.md – Downloads metrics and parity workflow.
  • /docs/observability/observability.md – Platform-wide practices.
  • /ops/telemetry-collector.md & /ops/telemetry-storage.md – Collector deployment.
  • /docs/install/docker.md – Compose/Helm environment variables.

Last updated: 2025-10-28 (Sprint 23).