git.stella-ops.org/docs/observability/ui-telemetry.md

Console Observability

Audience: Observability Guild, Console Guild, SRE/operators.
Scope: Metrics, logs, traces, dashboards, alerting, feature flags, and offline workflows for the StellaOps Console (Sprint 23).
Prerequisites: Console deployed with metrics enabled (CONSOLE_METRICS_ENABLED=true) and OTLP exporters configured (OTEL_EXPORTER_OTLP_*).

---

1 · Instrumentation Overview

• Telemetry stack: OpenTelemetry Web SDK (browser) + Console telemetry bridge → OTLP collector (Tempo/Prometheus/Loki). Server-side endpoints expose /metrics (Prometheus) and /health/*.
• Sampling: Front-end spans sample at 5 % by default (OTEL_TRACES_SAMPLER=parentbased_traceidratio). Metrics are un-sampled; log sampling is handled per category (§3).
• Correlation IDs: Every API call carries x-stellaops-correlation-id; structured UI events mirror that value so operators can follow a request across gateway, backend, and UI.
• Scope gating: Operators need the ui.telemetry scope to view live charts in the Admin workspace; the scope also controls access to /console/telemetry SSE streams.

---

2 · Metrics

2.1 Experience & Navigation

Metric	Type	Labels	Notes
ui_route_render_seconds	Histogram	route, tenant, device (desktop,tablet)	Time between route activation and first interactive paint. Target P95 ≤ 1.5 s (cached).
ui_request_duration_seconds	Histogram	service, method, status, tenant	Gateway proxy timing for backend calls performed by the console. Alerts when backend latency degrades.
ui_filter_apply_total	Counter	route, filter, tenant	Increments when a global filter or context chip is applied. Used to track adoption of saved views.
ui_tenant_switch_total	Counter	fromTenant, toTenant, trigger (picker, shortcut, link)	Emitted after a successful tenant switch; correlates with Authority ui.tenant.switch logs.
ui_offline_banner_seconds	Histogram	reason (authority, manifest, gateway), tenant	Duration of offline banner visibility; integrate with air-gap SLAs.

2.2 Security & Session

Metric	Type	Labels	Notes
ui_dpop_failure_total	Counter	endpoint, reason (nonce, jkt, clockSkew)	Raised when DPoP validation fails; pair with Authority audit trail.
ui_fresh_auth_prompt_total	Counter	action (token.revoke, policy.activate, client.create), tenant	Counts fresh-auth modals; backlog above baseline indicates workflow friction.
ui_fresh_auth_failure_total	Counter	action, reason (timeout,cancelled,auth_error)	Optional metric (set CONSOLE_FRESH_AUTH_METRICS=true when feature flag lands).

2.3 Downloads & Offline Kit

Metric	Type	Labels	Notes
ui_download_manifest_refresh_seconds	Histogram	tenant, channel (edge,stable,airgap)	Time to fetch and verify downloads manifest. Target < 3 s.
ui_download_export_queue_depth	Gauge	tenant, artifactType (sbom,policy,attestation,console)	Mirrors /console/downloads queue depth; triggers when offline bundles lag.
ui_download_command_copied_total	Counter	tenant, artifactType	Increments when users copy CLI commands from the UI. Useful to observe CLI parity adoption.

2.4 Telemetry Emission & Errors

Metric	Type	Labels	Notes
ui_telemetry_batch_failures_total	Counter	transport (otlp-http,otlp-grpc), reason	Emitted by OTLP bridge when batches fail. Enable via CONSOLE_METRICS_VERBOSE=true.
ui_telemetry_queue_depth	Gauge	priority (normal,high), tenant	Browser-side buffer depth; monitor for spikes under degraded collectors.

Scraping tips:

• Enable /metrics via CONSOLE_METRICS_ENABLED=true.
• Set OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.collector:4318 and relevant headers (OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer <token>).
• For air-gapped sites, point the exporter to the Offline Kit collector (localhost:4318) and forward the metrics snapshot using stella offline bundle metrics.

---

3 · Logs

• Format: JSON via Console log bridge; emitted to stdout and optional OTLP log exporter. Core fields: timestamp, level, action, route, tenant, subject, correlationId, dpop.jkt, device, offlineMode.
• Categories:
  • ui.action – general user interactions (route changes, command palette, filter updates). Sampled 50 % by default; override with feature flag telemetry.logVerbose.
  • ui.tenant.switch – always logged; includes fromTenant, toTenant, tokenId, and Authority audit correlation.
  • ui.download.commandCopied – download commands copied; includes artifactId, digest, manifestVersion.
  • ui.security.anomaly – DPoP mismatches, tenant header errors, CSP violations (level = Warning).
  • ui.telemetry.failure – OTLP export errors; include httpStatus, batchSize, retryCount.
• PII handling: Full emails are scrubbed; only hashed values (user:<sha256>) appear unless ui.admin + fresh-auth were granted for the action (still redacted in logs).
• Retention: Recommended 14 days for connected sites, 30 days for sealed/air-gap audits. Ship logs to Loki/Elastic with ingest label service="stellaops-web-ui".

---

4 · Traces

• Span names & attributes:
  • ui.route.transition – wraps route navigation; attributes: route, tenant, renderMillis, prefetchHit.
  • ui.api.fetch – HTTP fetch to backend; attributes: service, endpoint, status, networkTime.
  • ui.sse.stream – Server-sent event subscriptions (status ticker, runs); attributes: channel, connectedMillis, reconnects.
  • ui.telemetry.batch – Browser OTLP flush; attributes: batchSize, success, retryCount.
  • ui.policy.action – Policy workspace actions (simulate, approve, activate) per docs/ui/policy-editor.md.
• Propagation: Spans use W3C traceparent; gateway echoes header to backend APIs so traces stitch across UI → gateway → service.
• Sampling controls: OTEL_TRACES_SAMPLER_ARG (ratio) and feature flag telemetry.forceSampling (sets to 100 % for incident debugging).
• Viewing traces: Grafana Tempo or Jaeger via collector. Filter by service.name = stellaops-console. For cross-service debugging, filter on correlationId and tenant.

---

5 · Dashboards

5.1 Experience Overview

Panels:

• Route render histogram (P50/P90/P99) by route.
• Backend call latency stacked by service (ui_request_duration_seconds).
• Offline banner duration trend (ui_offline_banner_seconds).
• Tenant switch volume vs failure rate (overlay ui_dpop_failure_total).
• Command palette usage (ui_filter_apply_total + ui.action log counts).

5.2 Downloads & Offline Kit

• Manifest refresh time chart (per channel).
• Export queue depth gauge with alert thresholds.
• CLI command adoption (bar chart per artifact type, using ui_download_command_copied_total).
• Offline parity banner occurrences (downloads.offlineParity flag from API → derived metric).
• Last Offline Kit import timestamp (join with Downloads API metadata).

5.3 Security & Session

• Fresh-auth prompt counts vs success/fail ratios.
• DPoP failure stacked by reason.
• Tenant mismatch warnings (from ui.security.anomaly logs).
• Scope usage heatmap (derived from Authority audit events + UI logs).
• CSP violation counts (browser securitypolicyviolation listener forwarded to logs).

Capture screenshots for Grafana once dashboards stabilise (docs/assets/ui/observability/*.png). Replace placeholders before releasing the doc.

---

6 · Alerting

Alert	Condition	Suggested Action
ConsoleLatencyHigh	ui_route_render_seconds_bucket{le="1.5"} drops below 0.95 for 3 intervals	Inspect route splits, check backend latencies, review CDN cache.
BackendLatencyHigh	ui_request_duration_seconds_sum / ui_request_duration_seconds_count > 1 s for any service	Correlate with gateway/service dashboards; escalate to owning guild.
TenantSwitchFailures	Increase in ui_dpop_failure_total or ui.security.anomaly (tenant mismatch) > 3/min	Validate Authority issuer, check clock skew, confirm tenant config.
FreshAuthLoop	ui_fresh_auth_prompt_total spikes with matching ui_fresh_auth_failure_total	Review Authority /fresh-auth endpoint, session timeout config, UX regressions.
OfflineBannerLong	ui_offline_banner_seconds P95 > 120 s	Investigate Authority/gateway availability; verify Offline Kit freshness.
DownloadsBacklog	ui_download_export_queue_depth > 5 for 10 min OR queue age > alert threshold	Ping Downloads service, ensure manifest pipeline (DOWNLOADS-CONSOLE-23-001) is healthy.
TelemetryExportErrors	ui_telemetry_batch_failures_total > 0 for ≥5 min	Check collector health, credentials, or TLS trust.

Integrate alerts with Notifier (ui.alerts) or existing Ops channels. Tag incidents with component=console for correlation.

---

7 · Feature Flags & Configuration

Flag / Env Var	Purpose	Default
CONSOLE_FEATURE_FLAGS	Enables UI modules (runs, downloads, policies, telemetry). Telemetry panel requires telemetry.	runs,downloads,policies
CONSOLE_METRICS_ENABLED	Exposes /metrics for Prometheus scrape.	true
CONSOLE_METRICS_VERBOSE	Emits additional batching metrics (ui_telemetry_*).	false
CONSOLE_LOG_LEVEL	Minimum log level (Information, Debug). Use Debug for incident sampling.	Information
CONSOLE_METRICS_SAMPLING (planned)	Controls front-end span sampling ratio. Document once released.	0.05
OTEL_EXPORTER_OTLP_ENDPOINT	Collector URL; supports HTTPS.	unset
OTEL_EXPORTER_OTLP_HEADERS	Comma-separated headers (auth).	unset
OTEL_EXPORTER_OTLP_INSECURE	Allow HTTP (dev only).	false
OTEL_SERVICE_NAME	Service tag for traces/logs. Set to stellaops-console.	auto
CONSOLE_TELEMETRY_SSE_ENABLED	Enables /console/telemetry SSE feed for dashboards.	true

Feature flag changes should be tracked in release notes and mirrored in /docs/ui/navigation.md (shortcuts may change when modules toggle).

---

8 · Offline / Air-Gapped Workflow

• Mirror the console image and telemetry collector as part of the Offline Kit (see /docs/install/docker.md §4).
• Scrape metrics locally via curl -k https://console.local/metrics > metrics.prom; archive alongside logs for audits.
• Use stella offline kit import to keep the downloads manifest in sync; dashboards display staleness using ui_download_manifest_refresh_seconds.
• When collectors are unavailable, console queues OTLP batches (up to 5 min) and exposes backlog through ui_telemetry_queue_depth; export queue metrics to prove no data loss.
• After reconnecting, run stella console status --telemetry (CLI parity pending; see DOCS-CONSOLE-23-014) or verify ui_telemetry_batch_failures_total resets to zero.
• Retain telemetry bundles for 30 days per compliance guidelines; include Grafana JSON exports in audit packages.

---

9 · Compliance Checklist

• /metrics scraped in staging & production; dashboards display ui_route_render_seconds, ui_request_duration_seconds, and downloads metrics.
• OTLP traces/logs confirmed end-to-end (collector, Tempo/Loki).
• Alert rules from §6 implemented in monitoring stack with runbooks linked.
• Feature flags documented and change-controlled; telemetry disabled only with approval.
• DPoP/fresh-auth anomalies correlated with Authority audit logs during drill.
• Offline capture workflow exercised; evidence stored in audit vault.
• Screenshots of Grafana dashboards committed once they stabilise (update references).
• Cross-links verified (docs/deploy/console.md, docs/security/console-security.md, docs/ui/downloads.md, docs/ui/console-overview.md).

---

10 · References

• /docs/deploy/console.md – Metrics endpoint, OTLP config, health checks.
• /docs/security/console-security.md – Security metrics & alert hints.
• /docs/ui/console-overview.md – Telemetry primitives and performance budgets.
• /docs/ui/downloads.md – Downloads metrics and parity workflow.
• /docs/observability/observability.md – Platform-wide practices.
• /ops/telemetry-collector.md & /ops/telemetry-storage.md – Collector deployment.
• /docs/install/docker.md – Compose/Helm environment variables.

---

Last updated: 2025-10-28 (Sprint 23).