feat: Implement Filesystem and MongoDB provenance writers for PackRun execution context
- Added `FilesystemPackRunProvenanceWriter` to write provenance manifests to the filesystem. - Introduced `MongoPackRunArtifactReader` to read artifacts from MongoDB. - Created `MongoPackRunProvenanceWriter` to store provenance manifests in MongoDB. - Developed unit tests for filesystem and MongoDB provenance writers. - Established `ITimelineEventStore` and `ITimelineIngestionService` interfaces for timeline event handling. - Implemented `TimelineIngestionService` to validate and persist timeline events with hashing. - Created PostgreSQL schema and migration scripts for timeline indexing. - Added dependency injection support for timeline indexer services. - Developed tests for timeline ingestion and schema validation.
This commit is contained in:
StellaOps Bot
@@ -2,8 +2,10 @@
|
||||
|
||||
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
|
||||
|
||||
## Latest updates (2025-10-19)
|
||||
- Platform Events refresh published canonical `attestor.logged@1` samples under `docs/events/samples/` and validated schemas (`docs/updates/2025-10-18-docs-guild.md`, `docs/updates/2025-10-19-docs-guild.md`). Consumers should align verification workflows and tests with those sample envelopes.
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0313_0001_0001_docs_modules_attestor.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability runbook stub + dashboard placeholder added under `operations/` (offline import) pending next demo outputs.
|
||||
- Platform Events samples (2025-10-18/19) remain the current canonical `attestor.logged@1`; keep verification workflows aligned.
|
||||
|
||||
## Why it exists
|
||||
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
|
||||
@@ -44,10 +46,11 @@ All predicates capture subjects, issuer metadata, policy context, materials, opt
|
||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||
|
||||
## Observability & performance
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_cache_hit_ratio`, `attestor_rekor_latency_seconds`.
|
||||
- Logs capture tenant, issuer, subject digests, Rekor UUID, proof status, and policy verdict.
|
||||
- Performance target: ≥1 000 envelopes/minute per worker with cached verification, batched operations, and concurrency controls.
|
||||
## Observability & performance
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_cache_hit_ratio`, `attestor_rekor_latency_seconds`.
|
||||
- Logs capture tenant, issuer, subject digests, Rekor UUID, proof status, and policy verdict.
|
||||
- Performance target: ≥1 000 envelopes/minute per worker with cached verification, batched operations, and concurrency controls.
|
||||
- Observability assets: `operations/observability.md` and `operations/dashboards/attestor-observability.json` (offline import).
|
||||
|
||||
## Key integrations
|
||||
- Signer (DSSE source), Authority (scopes & tenancy), Export Center (attestation bundles), Policy Engine (verification policies), Scanner/Excititor (subject evidence), Notify (key rotation & verification alerts), Observability stack (dashboards/alerts).
|
||||
|
||||
9
docs/modules/attestor/TASKS.md
Normal file
9
docs/modules/attestor/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Attestor · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| ATTESTOR-DOCS-0001 | DONE (2025-11-05) | Docs Guild | README aligned with latest release notes and attestation samples. |
|
||||
| ATTESTOR-OPS-0001 | BLOCKED (2025-11-30) | Ops Guild | Await next demo outputs; observability runbook stub added. |
|
||||
| ATTESTOR-ENG-0001 | DONE (2025-11-27) | Module Team | Implementation plan readiness tracker added. |
|
||||
|
||||
> Keep this table in lockstep with `docs/implplan/SPRINT_0313_0001_0001_docs_modules_attestor.md` (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Attestor. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Attestor Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
39
docs/modules/attestor/operations/observability.md
Normal file
39
docs/modules/attestor/operations/observability.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# Attestor observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/attestor/operations/dashboards/attestor-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: signing latency p95/p99, verification failure rate, transparency log submission lag, key rotation age, DSSE verification errors, queue depth/backlog, and attestation bundle size histogram.
|
||||
|
||||
## Key metrics
|
||||
- `attestor_sign_latency_seconds_bucket` — signing latency per request.
|
||||
- `attestor_verify_failures_total{reason}` — verification failures.
|
||||
- `attestor_tlog_submission_latency_seconds_bucket` — transparency log latency.
|
||||
- `attestor_key_rotation_age_seconds` — time since last rotation.
|
||||
- `attestor_queue_backlog` — pending attestation jobs.
|
||||
- `attestor_bundle_size_bytes_bucket` — bundle size distribution.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId`, `attestationId`, `tenant`, `keyId`. Include `policyVersion`, `tlogIndex`, and `rekorUrl` where applicable.
|
||||
- Traces disabled by default for air-gap; enable by pointing OTLP exporter to on-prem collector.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` check signer availability, tlog reachability, and storage.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Verification probe: `POST /api/attestations/verify` with sample bundle once demo outputs land.
|
||||
|
||||
## Alert hints
|
||||
- Signing latency p99 > 1s.
|
||||
- Verification failure spikes.
|
||||
- Transparency log submission lag > 10s.
|
||||
- Key rotation age exceeding policy threshold.
|
||||
- Queue backlog above configured threshold.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `attestor`.
|
||||
2) Run sample verification once demo bundle available and confirm metrics/logs emit locally.
|
||||
3) Fetch `/status` and compare commit/version to offline bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0313_0001_0001_docs_modules_attestor.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/attestor-observability.json`.
|
||||
@@ -8,6 +8,7 @@ Authority is the platform OIDC/OAuth2 control plane that mints short-lived, send
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./operations/monitoring.md) and Grafana JSON (offline import)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
|
||||
@@ -2,7 +2,12 @@
|
||||
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Responsibilities
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md` and module `TASKS.md` added to mirror status.
|
||||
- Monitoring/observability references consolidated; Grafana JSON remains offline import (`operations/grafana-dashboard.json`).
|
||||
- Prior content retained: OpTok/DPoP/mTLS responsibilities, backup/restore, key rotation.
|
||||
|
||||
## Responsibilities
|
||||
- Expose device-code, auth-code, and client-credential flows with DPoP or mTLS binding.
|
||||
- Manage signing keys, JWKS rotation, and PoE integration for plan enforcement.
|
||||
- Emit structured audit events and enforce tenant-aware scope policies.
|
||||
@@ -18,16 +23,17 @@ Authority is the platform OIDC/OAuth2 control plane that mints short-lived, send
|
||||
- CLI/UI for login flows and token management.
|
||||
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (see ./operations/key-rotation.md).
|
||||
- Grafana/Prometheus dashboards for auth latency/issuance.
|
||||
## Operational notes
|
||||
- MongoDB for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (`operations/key-rotation.md`).
|
||||
- Monitoring runbook (`operations/monitoring.md`) and offline-import Grafana JSON (`operations/grafana-dashboard.json`).
|
||||
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
- Sprint/status mirrors: `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md`, `docs/modules/authority/TASKS.md`
|
||||
|
||||
## Backlog references
|
||||
- DOCS-SEC-62-001 (scope hardening doc) in ../../TASKS.md.
|
||||
|
||||
9
docs/modules/authority/TASKS.md
Normal file
9
docs/modules/authority/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Authority · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| AUTHORITY-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refreshed; sprint + monitoring links added. |
|
||||
| AUTHORITY-ENG-0001 | DONE (2025-11-27) | Module Team | Readiness tracker in implementation_plan mapped to epics/sprints. |
|
||||
| AUTHORITY-OPS-0001 | DONE (2025-11-30) | Ops Guild | TASKS board created; monitoring/grafana references aligned; offline-friendly. |
|
||||
|
||||
> Keep this table in lockstep with `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md` (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -16,16 +16,21 @@
|
||||
- **Epic 14 – Identity & Tenancy:** implement tenant isolation, RBAC hierarchies, audit trails, and PoE integration.
|
||||
- Track additional work (DOCS-SEC-62-001, AUTH-POLICY-20-001/002) in ../../TASKS.md and src/Authority/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
---
|
||||
|
||||
## Sprint readiness tracker
|
||||
|
||||
> Last updated: 2025-11-27 (AUTHORITY-ENG-0001)
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md`; statuses mirrored in `docs/modules/authority/TASKS.md`.
|
||||
- Observability assets remain in `operations/monitoring.md` with Grafana JSON `operations/grafana-dashboard.json` (offline import).
|
||||
- Authority readiness tracker (AUTHORITY-ENG-0001) delivered 2025-11-27; future updates should adjust both sprint and TASKS.
|
||||
|
||||
---
|
||||
|
||||
## Sprint readiness tracker
|
||||
|
||||
> Last updated: 2025-11-27 (AUTHORITY-ENG-0001)
|
||||
|
||||
This section maps epic milestones to implementation sprints and tracks readiness checkpoints.
|
||||
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
|
||||
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
|
||||
|
||||
## Latest updates (2025-11-05)
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0333_0001_0001_docs_modules_excititor.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability/runbook assets remain in `operations/observability.md` and `observability/` (timeline, locker manifests); dashboards stay offline-import friendly.
|
||||
- Prior updates (2025-11-05): Link-Not-Merge readiness and consensus beta note (`../../updates/2025-11-05-excitor-consensus-beta.md`), observability guide additions, DSSE packaging guidance, and Policy/CLI follow-ups tracked in SPRINT_200.
|
||||
- Link-Not-Merge readiness: release note [Excitor consensus beta](../../updates/2025-11-05-excitor-consensus-beta.md) captures how Excititor feeds power the Excititor consensus beta (sample payload in [consensus JSON](../../vex/consensus-json.md)).
|
||||
- Added [observability guide](operations/observability.md) describing the evidence metrics emitted by `EXCITITOR-AIAI-31-003` (request counters, statement histogram, signature status, guard violations) so Ops/Lens can alert on misuse.
|
||||
- README now points policy/UI teams to the upcoming consensus integration work.
|
||||
|
||||
12
docs/modules/excititor/TASKS.md
Normal file
12
docs/modules/excititor/TASKS.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Excititor · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| EXCITOR-DOCS-0001 | DONE (2025-11-07) | Docs Guild | README aligned to consensus beta release notes. |
|
||||
| EXCITOR-OPS-0001 | DONE (2025-11-07) | Ops Guild | Runbooks/observability checklist added (`mirrors.md`). |
|
||||
| EXCITOR-ENG-0001 | DONE (2025-11-07) | Module Team | Implementation plan alignment with SPRINT_200 updates. |
|
||||
| EXCITITOR-DOCS-0001 | BLOCKED (2025-11-19) | Docs Guild | Await chunk API CI validation + OpenAPI freeze before finalizing docs. |
|
||||
| EXCITITOR-ENG-0001 | TODO | Module Team | Update engineering notes once chunk API/OpenAPI unblock. |
|
||||
| EXCITITOR-OPS-0001 | TODO | Ops Guild | Sync observability/runbook updates after OpenAPI freeze. |
|
||||
|
||||
> Keep this table in lockstep with `docs/implplan/SPRINT_0333_0001_0001_docs_modules_excititor.md` (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -20,12 +20,15 @@
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
## Sprint alignment (2025-11-07)
|
||||
## Sprint alignment (2025-11-30)
|
||||
|
||||
| Sprint task | State (SPRINT_200) | Notes |
|
||||
| Sprint task | State (SPRINT_0333_0001_0001_docs_modules_excititor) | Notes |
|
||||
| --- | --- | --- |
|
||||
| EXCITITOR-DOCS-0001 | DONE | README release alignment + consensus beta references refreshed (DSSE/export guidance). |
|
||||
| EXCITITOR-ENG-0001 | DONE | Implementation plan now mirrors `SPRINT_200_documentation_process.md` through this table. |
|
||||
| EXCITITOR-OPS-0001 | DONE | Runbook/observability checklist added to `docs/modules/excititor/mirrors.md`. |
|
||||
| EXCITOR-DOCS-0001 | DONE | README release alignment + consensus beta references refreshed (DSSE/export guidance). |
|
||||
| EXCITOR-ENG-0001 | DONE | Implementation plan mirrored to sprint status; TASKS board created. |
|
||||
| EXCITOR-OPS-0001 | DONE | Runbook/observability checklist added to `mirrors.md`. |
|
||||
| EXCITITOR-DOCS-0001 | BLOCKED | Waiting on chunk API CI validation + OpenAPI freeze. |
|
||||
| EXCITITOR-ENG-0001 | TODO | Will update engineering notes once DOCS unblock. |
|
||||
| EXCITITOR-OPS-0001 | TODO | Sync observability/runbook updates after OpenAPI freeze. |
|
||||
|
||||
See `/docs/implplan/SPRINT_200_documentation_process.md` for the canonical status table.
|
||||
See `/docs/implplan/SPRINT_0333_0001_0001_docs_modules_excititor.md` for the canonical status table.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Export Center KMS Envelope Pattern (age + AES-GCM)
|
||||
|
||||
Status: Adopted for Sprint 0164-0001-0001 (ExportCenter III)
|
||||
|
||||
Scope: Defines deterministic envelope handling for mirror bundle encryption (`EXPORT-SVC-37-002`) and general export signing. Applies to worker path and verification docs.
|
||||
|
||||
## Key hierarchy
|
||||
- **Content key (DEK):** 32-byte random generated per export run. Used for AES-256-GCM over encrypted payloads (`/data` subtree for mirror; optional for others).
|
||||
- **Nonce:** 12-byte random per file; stored alongside ciphertext; derive Additional Authenticated Data (AAD) as `{runId}:{relativePath}` to bind file path and run.
|
||||
- **Wrapping keys:**
|
||||
- **age recipients** (preferred for offline): each tenant can list one or more age public keys. DEK is wrapped once per recipient using age X25519. Store `recipient`, `wrappedKey` (base64), and optional `keyId` in provenance.
|
||||
- **KMS envelope** (Authority/HSM): DEK wrapped with tenant-scoped KMS key alias `export/envelope`. Store `kmsKeyId` (authority URI or external ARN) and `wrappedKey` (base64) plus KMS-provided `algorithm`.
|
||||
|
||||
## Write path (worker)
|
||||
1) Generate DEK (32 bytes) per run; zeroize after use.
|
||||
2) For each encrypted file, derive AAD = `{runId}:{relativePath}`; encrypt with AES-256-GCM (nonce per file). Store `nonce` and `ciphertext`.
|
||||
3) Wrap DEK for all configured recipients:
|
||||
- age: `age --encrypt --recipient <pub>` over DEK bytes → base64.
|
||||
- KMS: `Encrypt`/`WrapKey` with `KeyId=export/envelope` and `EncryptionContext={runId,tenant}` → base64.
|
||||
4) Record wrapping metadata in `provenance.json` under `environment.encryption.recipients[]` preserving deterministic order (age recipients lexicographically by `recipient`, then KMS entries by `kmsKeyId`).
|
||||
5) Include `encryption.mode` (`age` or `aes-gcm+kms`), `aadFormat`, and `nonceFormat` in provenance for verification tooling.
|
||||
|
||||
## Read/verification path
|
||||
1) Select a recipient entry that matches available keys (age private key or KMS key).
|
||||
2) Unwrap DEK:
|
||||
- age: `age --decrypt` → DEK bytes.
|
||||
- KMS: `Decrypt`/`UnwrapKey` with same encryption context.
|
||||
3) For each encrypted file, recompute AAD from `{runId}:{relativePath}`, decrypt with AES-256-GCM using stored `nonce`, verify tag.
|
||||
4) Recompute SHA-256 of decrypted payload and compare with `export.json` entries.
|
||||
|
||||
## Determinism & offline posture
|
||||
- Recipient lists and wrapped keys are ordered deterministically to keep `provenance.json` hashes stable across retries.
|
||||
- age path works fully offline; KMS path requires Authority/HSM availability but stores all metadata to allow later decryption once KMS is reachable.
|
||||
- Use fixed casing and field names: `mode`, `recipients[] {type, recipient|kmsKeyId, wrappedKey, keyId?}` and `aadFormat`.
|
||||
|
||||
## Testing notes
|
||||
- Add regression cases that encrypt/decrypt fixtures with both age and KMS paths, asserting identical manifest/provenance hashes across reruns.
|
||||
- Ensure decryption fails when AAD does not match expected `{runId}:{relativePath}` (prevents path swapping).
|
||||
- Keep tests air-gap friendly: mock KMS wrapper with deterministic stub keys.
|
||||
|
||||
## Rollout guidance
|
||||
- Default to age recipients for Offline Kit deployments; enable KMS wrapping where Authority/HSM is reachable.
|
||||
- Configuration knobs:
|
||||
- `ExportCenter:Encryption:Mode` = `age` | `kms`
|
||||
- `ExportCenter:Encryption:Recipients` = list of age public keys
|
||||
- `ExportCenter:Encryption:KmsKeyId` = tenant-specific key alias (when using KMS)
|
||||
- Documented verification commands should reference this pattern (update CLI/Console guides when payloads change).
|
||||
@@ -11,15 +11,16 @@ The Export Center packages StellaOps evidence and policy overlays into reproduci
|
||||
- Runbook execution for recovery, retention, and compliance.
|
||||
- Coordination with DevOps validation (cosign + `trivy module db import` smoke tests).
|
||||
|
||||
Related documentation:
|
||||
|
||||
- `docs/modules/export-center/overview.md`
|
||||
- `docs/modules/export-center/architecture.md`
|
||||
- `docs/modules/export-center/profiles.md`
|
||||
- `docs/modules/export-center/trivy-adapter.md`
|
||||
- `docs/modules/export-center/mirror-bundles.md`
|
||||
- `docs/modules/export-center/api.md`
|
||||
- `docs/modules/export-center/cli.md`
|
||||
Related documentation:
|
||||
|
||||
- `docs/modules/export-center/overview.md`
|
||||
- `docs/modules/export-center/architecture.md`
|
||||
- `docs/modules/export-center/profiles.md`
|
||||
- `docs/modules/export-center/trivy-adapter.md`
|
||||
- `docs/modules/export-center/mirror-bundles.md`
|
||||
- `docs/modules/export-center/api.md`
|
||||
- `docs/modules/export-center/cli.md`
|
||||
- `docs/modules/export-center/operations/kms-envelope-pattern.md`
|
||||
|
||||
## 2. Contacts & tooling
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ Notify evaluates operator-defined rules against platform events and dispatches c
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./operations/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
|
||||
Notify (Notifications Studio) converts platform events into tenant-scoped alerts with deterministic delivery, offline parity, and a full audit trail. The service is split between the reusable tooling in `src/Notify/*` and the runtime host in `src/Notifier/*` (decision recorded 2025-11-02) so downstream systems can embed the rules engine without inheriting the Studio UI.
|
||||
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_322_docs_modules_notify.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability runbook stub and Grafana placeholder added under `operations/` (offline import); finalize after next demo.
|
||||
- NOTIFY-DOCS-0002 remains blocked pending NOTIFY-SVC-39-001..004 outputs (correlation/digests/simulation/quiet hours).
|
||||
|
||||
## Scope & responsibilities
|
||||
- Apply tenant-scoped rules to events from Scanner, Scheduler, VEX Lens, Attestor, Task Runner, and Zastava.
|
||||
- Render channel-specific payloads (Slack, Teams, Email, webhook) using deterministic templates with localisation safeguards.
|
||||
@@ -40,6 +45,7 @@ Status for these items is tracked in `src/Notifier/StellaOps.Notifier/TASKS.md`
|
||||
- Schema fixtures live in `./resources/schemas`; event and delivery samples live in `./resources/samples` for contract tests and UI mocks.
|
||||
- Offline Kit bundles ship plug-ins, default templates, and seed rules; update manifests under `ops/offline-kit/` when connectors change.
|
||||
- Dashboards and alert references depend on `DEVOPS-NOTIFY-39-002`; coordinate before renaming metrics or labels.
|
||||
- Observability assets: `operations/observability.md` and `operations/dashboards/notify-observability.json` (offline import).
|
||||
- When releasing new rule or connector features, mirror guidance into `docs/notifications/*.md` and checklists in `docs/updates/2025-10-29-notify-docs.md` until the follow-ups are closed.
|
||||
|
||||
## Epic alignment
|
||||
|
||||
10
docs/modules/notify/TASKS.md
Normal file
10
docs/modules/notify/TASKS.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Notify · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| NOTIFY-DOCS-0001 | DONE (2025-11-05) | Docs Guild | README refreshed for Notifications Studio pivot + release notes. |
|
||||
| NOTIFY-OPS-0001 | BLOCKED (2025-11-30) | Ops Guild | Await next demo outputs; observability runbook stub added. |
|
||||
| NOTIFY-ENG-0001 | DONE (2025-11-27) | Module Team | Implementation plan readiness tracker aligned with SPRINT_171..173. |
|
||||
| NOTIFY-DOCS-0002 | BLOCKED (2025-11-30) | Docs Guild | Pending NOTIFY-SVC-39-001..004 correlation/digests/simulation/quiet hours evidence. |
|
||||
|
||||
> Keep this table in lockstep with `docs/implplan/SPRINT_322_docs_modules_notify.md` (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -384,6 +384,7 @@ Authority signs ack tokens using keys configured under `notifications.ackTokens`
|
||||
* `notify.sent_total{channel}` / `notify.failed_total{channel,code}`
|
||||
* `notify.delivery_latency_seconds{channel}` (end‑to‑end)
|
||||
* **Tracing**: spans `ingest`, `match`, `render`, `send`; correlation id = `eventId`.
|
||||
- Runbook + dashboard stub (offline import): `operations/observability.md`, `operations/dashboards/notify-observability.json` (to be populated after next demo).
|
||||
|
||||
**SLO targets**
|
||||
|
||||
|
||||
@@ -56,13 +56,18 @@
|
||||
- **Security:** RBAC tests, tenant isolation, secret reference validation, DSSE signature verification.
|
||||
- **Offline:** export/import round-trips, Offline Kit deployment, manual delivery replay.
|
||||
|
||||
## Definition of done
|
||||
- Notify service, workers, connectors, Console/CLI, observability, and Offline Kit assets shipped with documentation and runbooks.
|
||||
- Compliance checklist appended to docs; ./TASKS.md and ../../TASKS.md updated with progress.
|
||||
|
||||
---
|
||||
|
||||
## Sprint readiness tracker
|
||||
## Definition of done
|
||||
- Notify service, workers, connectors, Console/CLI, observability, and Offline Kit assets shipped with documentation and runbooks.
|
||||
- Compliance checklist appended to docs; ./TASKS.md and ../../TASKS.md updated with progress.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs sprint: `docs/implplan/SPRINT_322_docs_modules_notify.md`; statuses mirrored in `docs/modules/notify/TASKS.md`.
|
||||
- Observability evidence stub: `operations/observability.md` and `operations/dashboards/notify-observability.json` (to be populated after next demo outputs).
|
||||
- NOTIFY-DOCS-0002 remains blocked pending NOTIFY-SVC-39-001..004 (correlation/digests/simulation/quiet hours); keep sprint/TASKS synced when those land.
|
||||
|
||||
---
|
||||
|
||||
## Sprint readiness tracker
|
||||
|
||||
> Last updated: 2025-11-27 (NOTIFY-ENG-0001)
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Notify. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Notify Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/notify/operations/observability.md
Normal file
38
docs/modules/notify/operations/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Notify observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/notify/operations/dashboards/notify-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: enqueue/dequeue rate, delivery latency p95/p99, channel error rate, retry/dead-letter counts, rule evaluation latency, tenant isolation breaches (should stay 0), and notification simulation outcomes.
|
||||
|
||||
## Key metrics
|
||||
- `notify_enqueue_total{channel}` — notifications enqueued by channel.
|
||||
- `notify_delivery_latency_seconds_bucket{channel}` — delivery latency per channel.
|
||||
- `notify_delivery_failures_total{channel,reason}` — failed deliveries.
|
||||
- `notify_retry_total{channel}` and `notify_deadletter_total{channel}` — retries and dead letters.
|
||||
- `notify_rule_eval_duration_seconds_bucket` — rule evaluation latency.
|
||||
- `notify_simulation_total{result}` — simulation outcomes when quiet hours/correlation rules applied.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `notificationId`, `ruleId`, `tenant`, `channel`. Include `quietHoursApplied`, `correlationKey`, `retries` fields.
|
||||
- Traces disabled by default for air-gap; enable by pointing OTLP exporter to on-prem collector.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` check queue backend reachability and channel provider credentials.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Simulation probe: `/api/notify/simulate` with sample rule set to validate correlation/digest wiring once NOTIFY-SVC-39-001..004 land.
|
||||
|
||||
## Alert hints
|
||||
- Delivery latency p99 > 1.5s for email/webhook channels.
|
||||
- Dead-letter queue growth > threshold.
|
||||
- Rule evaluation latency p99 > 500ms.
|
||||
- Correlation/quiet-hours simulation failures once enabled.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `notify`.
|
||||
2) Run `stella notify simulate --rules samples/rules.yaml --dry-run` (once available) and ensure metrics/logs emit locally.
|
||||
3) Fetch `/status` and compare commit/version to offline bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_322_docs_modules_notify.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/notify-observability.json`.
|
||||
@@ -2,8 +2,10 @@
|
||||
|
||||
The Orchestrator schedules, observes, and recovers ingestion and analysis jobs across the StellaOps platform.
|
||||
|
||||
## Latest updates (2025-11-18)
|
||||
- Job leasing now flows through the Task Runner bridge: allocations carry idempotency keys, lease durations, and retry hints; workers acknowledge via claim/ack and emit heartbeats.
|
||||
## Latest updates (2025-11-30)
|
||||
- OpenAPI discovery published at `/.well-known/openapi` with `openapi/orchestrator.json`; includes pagination/idempotency/error-envelope examples and version headers.
|
||||
- Legacy job detail/summary endpoints now emit `Deprecation` + `Link` headers pointing to the stable replacements.
|
||||
- Job leasing flows through the Task Runner bridge: allocations carry idempotency keys, lease durations, and retry hints; workers acknowledge via claim/ack and emit heartbeats.
|
||||
- Event envelopes remain interim pending ORCH-SVC-37-101; include provenance (tenant/project, job type, correlationId, task runner id) in all notifier events.
|
||||
- Authority `orch:quota` / `orch:backfill` scopes require reason/ticket audit fields; include them in runbooks and dashboard overrides.
|
||||
|
||||
|
||||
@@ -5,5 +5,8 @@
|
||||
| ORCH-DOCS-0001 | DONE | Docs Guild | README updated with leasing / task runner bridge notes and interim envelope guidance. |
|
||||
| ORCH-ENG-0001 | DONE | Module Team | Sprint references normalized; notes synced to doc sprint. |
|
||||
| ORCH-OPS-0001 | DONE | Ops Guild | Runbook impacts captured in README; follow-up to update ops docs. |
|
||||
| ORCH-OAS-61-001 | DONE | Orchestrator Service Guild | OpenAPI spec drafted for orchestrator endpoints with pagination/idempotency/error envelopes (2025-11-30). |
|
||||
| ORCH-OAS-61-002 | DONE | Orchestrator Service Guild | `/.well-known/openapi` discovery endpoint aligned to runtime build metadata (2025-11-30). |
|
||||
| ORCH-OAS-63-001 | DONE | Orchestrator Service Guild | Legacy job endpoints emit `Deprecation` + `Link` headers; doc metadata updated (2025-11-30). |
|
||||
|
||||
Status rules: mirror changes in `docs/implplan/SPRINT_0323_0001_0001_docs_modules_orchestrator.md`; use TODO → DOING → DONE/BLOCKED; add brief note if pausing.
|
||||
|
||||
@@ -38,8 +38,9 @@
|
||||
- `POST /api/limits/throttle` — apply throttle (requires elevated scope).
|
||||
- `GET /api/dashboard/metrics` — aggregated metrics for Console dashboards.
|
||||
- Event envelope draft (`docs/modules/orchestrator/event-envelope.md`) defines notifier/webhook/SSE payloads with idempotency keys, provenance, and task runner metadata for job/pack-run events.
|
||||
|
||||
All responses include deterministic timestamps, job digests, and DSSE signature fields for offline reconciliation.
|
||||
- OpenAPI discovery: `/.well-known/openapi` exposes `/openapi/orchestrator.json` (OAS 3.1) with pagination/idempotency/error-envelope examples; legacy job detail/summary endpoints now ship `Deprecation` + `Link` headers that point to their replacements.
|
||||
|
||||
All responses include deterministic timestamps, job digests, and DSSE signature fields for offline reconciliation.
|
||||
|
||||
## 5) Observability
|
||||
|
||||
|
||||
87
docs/modules/taskrunner/architecture.md
Normal file
87
docs/modules/taskrunner/architecture.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# TaskRunner Architecture (v1)
|
||||
|
||||
> Canonical contract for TaskRunner delivery scoped by SPRINT_0157_0001_0002 (TaskRunner Blockers) and SPRINT_0157_0001_0001 (TaskRunner I). Anchored in product advisory **"29-Nov-2025 - Task Pack Orchestration and Automation"** and the Task Pack runbook/spec (`docs/task-packs/*.md`).
|
||||
|
||||
## 1. Purpose and Scope
|
||||
- Execute Task Packs deterministically with approvals, sealed-mode enforcement, and evidence capture.
|
||||
- Provide API/CLI surface for pack submission, status, logs, approvals, artifacts, and cancellation.
|
||||
- Produce provenance: DSSE attestation + evidence bundle for every completed run.
|
||||
- Operate offline/air-gapped with plan-hash binding and sealed-mode network allowlists.
|
||||
|
||||
## 2. Components
|
||||
- **WebService** (`StellaOps.TaskRunner.WebService`) - HTTP API, plan hash validation, SSE log streaming, approval endpoints.
|
||||
- **Worker** (`StellaOps.TaskRunner.Worker`) - run orchestration, retries/backoff, artifact capture, attestation generation.
|
||||
- **Core** (`StellaOps.TaskRunner.Core`) - execution graph builder, simulation engine, step state machine, policy/approval gate abstractions.
|
||||
- **Infrastructure** (`StellaOps.TaskRunner.Infrastructure`) - storage adapters (Mongo, file), artifact/object store clients, evidence bundle writer.
|
||||
|
||||
## 3. Execution Phases
|
||||
1. **Plan** - parse manifest, validate schema, resolve inputs/secrets, build execution graph, compute canonical `planHash` (SHA-256 over normalised graph).
|
||||
2. **Simulation (optional)** - dry-run graph; emit determinstic preview with approvals/policy gates highlighted.
|
||||
3. **Execution** - verify runtime graph matches `planHash`; execute steps honoring `maxParallel`, `continueOnError`, `map`/`parallel` semantics; stream logs/events.
|
||||
4. **Evidence** - capture artifacts + transcripts, emit DSSE attestation binding `planHash`, inputs/outputs, steps, and timestamps; expose artifact listings via API for post-run retrieval.
|
||||
|
||||
## 4. API Surface (v1)
|
||||
- `POST /api/runs` (`packs.run`) - submit pack run; requires manifest/version, inputs, tenant context; returns `runId` + `planHash`.
|
||||
- `GET /api/runs/{runId}` (`packs.read`) - run status (graph, attempts, pending gates).
|
||||
- `GET /api/runs/{runId}/logs` (`packs.read`) - SSE stream of ordered log events.
|
||||
- `GET /api/runs/{runId}/artifacts` (`packs.read`) - list captured artifacts with digests/paths.
|
||||
- `POST /api/runs/{runId}/approve` (`packs.approve`) - record approval gate decision (requires Authority token claims `pack_run_id`, `pack_gate_id`, `pack_plan_hash`).
|
||||
- `POST /api/runs/{runId}/cancel` (`packs.run`) - cancel active run.
|
||||
- TODO (Phase II): `GET /.well-known/openapi` (TASKRUN-OAS-61-002) after OAS publication.
|
||||
|
||||
## 5. Data Model (Mongo, mirrors migration doc)
|
||||
- **pack_runs**: `_id`, `planHash`, `plan`, `failurePolicy`, `requestedAt`, `createdAt`, `updatedAt`, `steps[]`, `tenantId`.
|
||||
- **pack_run_logs**: `_id`, `runId`, `sequence` (monotonic), `timestamp` (UTC), `level`, `eventType`, `message`, `stepId?`, `metadata`.
|
||||
- **pack_artifacts**: `_id`, `runId`, `name`, `type`, `sourcePath?`, `storedPath?`, `status`, `notes?`, `capturedAt`.
|
||||
- Indexes as defined in `docs/modules/taskrunner/migrations/pack-run-collections.md`.
|
||||
|
||||
## 6. Step Types and Semantics
|
||||
- `run` - module invocation; declares `inputs`/`outputs`.
|
||||
- `parallel` - executes nested `steps[]`; honors `maxParallel`.
|
||||
- `map` - expands items into child steps (`stepId[index]::templateId`).
|
||||
- `gate.approval` - human approval checkpoint; enforces TTL/required count; pauses run until satisfied or expired.
|
||||
- `gate.policy` - Policy Engine evaluation; `failAction` decides halt vs. continue.
|
||||
|
||||
## 7. Determinism, Air-Gap, and Security
|
||||
- Plan hash binding: runtime graph must equal planned graph; mismatch aborts run.
|
||||
- All timestamps UTC ISO-8601; ordered logs via `(runId, sequence)` unique index.
|
||||
- Secrets never logged; evidence bundles store only redacted metadata.
|
||||
- Sealed mode: reject non-allowlisted network calls; approvals can be processed offline via request/response bundles.
|
||||
- RBAC scopes: `packs.read`, `packs.write`, `packs.run`, `packs.approve`.
|
||||
- Approval enforcement: service rejects approval decisions when provided `planHash` does not match stored run state (protects against stale/forged tokens).
|
||||
|
||||
## 8. Evidence & Attestation
|
||||
- DSSE attestation payload (`payloadType`: `application/vnd.stellaops.pack-run+json`) includes `runId`, `packName/version`, `planHash`, input/output digests, step statuses, `completedAt`.
|
||||
- Evidence bundle contents: signed manifest, inputs (redacted), outputs, transcripts, DSSE attestation; optional Rekor anchoring when online.
|
||||
|
||||
## 9. Observability (Phase I delivered)
|
||||
- Metrics: step latency, retries, queue depth, resource usage (`TASKRUN-OBS-50/51-001` DONE).
|
||||
- Pending: timeline events (`TASKRUN-OBS-52-001`), evidence snapshots (`TASKRUN-OBS-53-001`), attestations (`TASKRUN-OBS-54-001`), incident mode (`TASKRUN-OBS-55-001`).
|
||||
|
||||
## 10. Integration Points
|
||||
- **Authority** - approval tokens, scope validation, sealed-vault secrets.
|
||||
- **Policy Engine** - `gate.policy` decisions, policy context in evidence.
|
||||
- **Export Center** - evidence bundles and manifests for offline/air-gapped export.
|
||||
- **Orchestrator/CLI** - submission + resume flows; SSE log consumption.
|
||||
|
||||
## 11. Configuration (Mongo example)
|
||||
```json
|
||||
\"TaskRunner\": {
|
||||
\"Storage\": {
|
||||
\"Mode\": \"mongo\",
|
||||
\"Mongo\": {
|
||||
\"ConnectionString\": \"mongodb://127.0.0.1:27017/taskrunner\",
|
||||
\"Database\": \"taskrunner\",
|
||||
\"RunsCollection\": \"pack_runs\",
|
||||
\"LogsCollection\": \"pack_run_logs\",
|
||||
\"ArtifactsCollection\": \"pack_artifacts\",
|
||||
\"ApprovalsCollection\": \"pack_run_approvals\"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 12. References
|
||||
- Product advisory: `docs/product-advisories/29-Nov-2025 - Task Pack Orchestration and Automation.md`.
|
||||
- Task Pack spec + authoring + runbook: `docs/task-packs/spec.md`, `docs/task-packs/authoring-guide.md`, `docs/task-packs/runbook.md`.
|
||||
- Migration detail: `docs/modules/taskrunner/migrations/pack-run-collections.md`.
|
||||
@@ -15,6 +15,7 @@ Telemetry module captures deployment and operations guidance for the shared obse
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./operations/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
|
||||
@@ -2,7 +2,12 @@
|
||||
|
||||
Telemetry module captures deployment and operations guidance for the shared observability stack (collectors, storage, dashboards).
|
||||
|
||||
## Responsibilities
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0330_0001_0001_docs_modules_telemetry.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability runbook stub + dashboard placeholder added under `operations/` (offline import).
|
||||
- Storage/isolation posture references updated; align with platform docs.
|
||||
|
||||
## Responsibilities
|
||||
- Deploy and operate OpenTelemetry collectors for StellaOps services.
|
||||
- Provide storage configuration for Prometheus/Tempo/Loki stacks.
|
||||
- Document smoke tests and offline bootstrapping steps.
|
||||
@@ -22,6 +27,7 @@ Telemetry module captures deployment and operations guidance for the shared obse
|
||||
- Smoke script references (../../ops/devops/telemetry).
|
||||
- Bundle packaging instructions in ops/devops/telemetry.
|
||||
- Sprint 23 console security sign-off (2025-10-27) added the `console-security.json` Grafana board and burn-rate alert pack—ensure environments import the updated dashboards/alerts referenced in `docs/updates/2025-10-27-console-security-signoff.md`.
|
||||
- Observability assets for this sprint: `operations/observability.md` and `operations/dashboards/telemetry-observability.json` (offline import).
|
||||
|
||||
## Related resources
|
||||
- ./operations/collector.md
|
||||
|
||||
9
docs/modules/telemetry/TASKS.md
Normal file
9
docs/modules/telemetry/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Telemetry · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| TELEMETRY-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refreshed for storage/isolation posture; sprint links added. |
|
||||
| TELEMETRY-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana placeholder added under `operations/`. |
|
||||
| TELEMETRY-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0330_0001_0001_docs_modules_telemetry.md`. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -58,7 +58,12 @@
|
||||
- **Security:** redaction verification, RBAC/tenant scoping, sealed-mode tests, signed config verification.
|
||||
- **Offline:** capture bundles, transfer, replay, compliance attestation.
|
||||
|
||||
## Definition of done
|
||||
- Collector profiles, storage backends, incident mode, dashboards, CLI, and offline kit delivered with telemetry and documentation.
|
||||
- Runbooks and SOC handoff packages published; compliance checklists appended.
|
||||
- ./TASKS.md and ../../TASKS.md updated; imposed rule statements confirmed in documentation.
|
||||
## Definition of done
|
||||
- Collector profiles, storage backends, incident mode, dashboards, CLI, and offline kit delivered with telemetry and documentation.
|
||||
- Runbooks and SOC handoff packages published; compliance checklists appended.
|
||||
- ./TASKS.md and ../../TASKS.md updated; imposed rule statements confirmed in documentation.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0330_0001_0001_docs_modules_telemetry.md`; statuses mirrored in `docs/modules/telemetry/TASKS.md`.
|
||||
- Observability evidence lives in `operations/observability.md` with Grafana JSON stub under `operations/dashboards/`.
|
||||
- Keep future doc/ops updates mirrored across sprint, TASKS, and module front doors to avoid drift.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Telemetry. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Telemetry Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/telemetry/operations/observability.md
Normal file
38
docs/modules/telemetry/operations/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Telemetry observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/telemetry/operations/dashboards/telemetry-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: collector uptime, scrape errors, ingestion/backlog per tenant, storage retention headroom, query latency p95/p99, and OTLP export errors.
|
||||
|
||||
## Key metrics
|
||||
- `telemetry_collector_uptime_seconds` — per-collector uptime.
|
||||
- `telemetry_scrape_failures_total{job}` — scrape failures per job.
|
||||
- `telemetry_ingest_backlog` — queued spans/logs/metrics awaiting storage.
|
||||
- `telemetry_storage_retention_percent_used` — storage utilization against retention budget.
|
||||
- `telemetry_query_latency_seconds_bucket{route}` — API/query latency.
|
||||
- `telemetry_otlp_export_failures_total{signal}` — OTLP export failures by signal.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `trace_id` and `tenant`; include `collector_id`, `pipeline`, `exporter` fields.
|
||||
- Traces disabled by default for air-gap; enable by setting OTLP endpoints to on-prem collectors.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (collector + storage gateway) check exporter reachability and disk headroom.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Storage probe: `GET /api/storage/usage` (if available) to confirm retention headroom; otherwise rely on Prometheus metrics.
|
||||
|
||||
## Alert hints
|
||||
- OTLP export failures > 0 over 5m.
|
||||
- Ingest backlog above threshold (configurable per tenant/workload).
|
||||
- Query latency p99 > 1s for `/api/query` routes.
|
||||
- Storage utilization > 85% of retention budget.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `telemetry`.
|
||||
2) Run collector smoke: push sample OTLP spans/logs/metrics to local collector and confirm metrics emit in Prometheus.
|
||||
3) Fetch `/status` and compare commit/version to offline bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0330_0001_0001_docs_modules_telemetry.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/telemetry-observability.json`.
|
||||
@@ -2,7 +2,11 @@
|
||||
|
||||
The Console presents operator dashboards for scans, policies, VEX evidence, runtime posture, and admin workflows.
|
||||
|
||||
## Responsibilities
|
||||
## Latest updates (2025-11-30)
|
||||
- Docs refreshed per `docs/implplan/SPRINT_0331_0001_0001_docs_modules_ui.md`; added observability runbook stub and TASKS mirror.
|
||||
- Access-control guidance from 2025-11-03 remains valid; ensure Authority scopes are verified before enabling uploads.
|
||||
|
||||
## Responsibilities
|
||||
- Render real-time status for ingestion, scanning, policy, and exports via SSE.
|
||||
- Provide policy editor, SBOM explorer, and advisory views with accessibility compliance.
|
||||
- Integrate with Authority for fresh-auth and scope enforcement.
|
||||
@@ -18,14 +22,16 @@ The Console presents operator dashboards for scans, policies, VEX evidence, runt
|
||||
- Authority for DPoP-protected calls.
|
||||
- Telemetry streams for observability dashboards.
|
||||
|
||||
## Operational notes
|
||||
- Auth smoke tests in ./operations/auth-smoke.md.
|
||||
- Console architecture doc for layout and SSE fan-out.
|
||||
- Accessibility and security guides in ../../ui/ & ../../security/.
|
||||
## Operational notes
|
||||
- Auth smoke tests in `operations/auth-smoke.md`.
|
||||
- Observability runbook + dashboard stub in `operations/observability.md` and `operations/dashboards/console-ui-observability.json` (offline import).
|
||||
- Console architecture doc for layout and SSE fan-out.
|
||||
- Accessibility and security guides in ../../ui/ & ../../security/.
|
||||
|
||||
## Related resources
|
||||
- ./operations/auth-smoke.md
|
||||
- ./console-architecture.md
|
||||
## Related resources
|
||||
- ./operations/auth-smoke.md
|
||||
- ./operations/observability.md
|
||||
- ./console-architecture.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-CONSOLE-23-001 … DOCS-CONSOLE-23-003 baseline (done).
|
||||
|
||||
9
docs/modules/ui/TASKS.md
Normal file
9
docs/modules/ui/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Console UI · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| CONSOLE UI-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture updated; sprint links and observability references added. |
|
||||
| CONSOLE UI-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0331_0001_0001_docs_modules_ui.md`. |
|
||||
| CONSOLE UI-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana JSON placeholder added under `operations/`. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both places).
|
||||
@@ -28,8 +28,9 @@
|
||||
* **State**: Angular **Signals** + `@ngrx/signals` store for cross‑page slices.
|
||||
* **Transport**: `fetch` + RxJS interop; **SSE** (EventSource) for progress streams.
|
||||
* **Build**: Angular CLI + Vite builder.
|
||||
* **Testing**: Jest + Testing Library, Playwright for e2e.
|
||||
* **Packaging**: Containerized NGINX (immutable assets, ETag + content hashing).
|
||||
* **Testing**: Jest + Testing Library, Playwright for e2e.
|
||||
* **Packaging**: Containerized NGINX (immutable assets, ETag + content hashing).
|
||||
* **Observability docs**: runbook + Grafana JSON stub in `operations/observability.md` and `operations/dashboards/console-ui-observability.json` (offline import).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -16,10 +16,15 @@
|
||||
- **Epic 6 – Vulnerability Explorer:** surface triage dashboards, findings ledger, and audit exports.
|
||||
- **Epic 8 – Advisory AI:** integrate advisory summaries and remediation hints with strict provenance.
|
||||
- **Epic 9 – Orchestrator Dashboard:** expose job/source monitoring controls.
|
||||
- **Epic 11 – Notifications Studio:** provide notifications workspace with previews and audit trails.
|
||||
- Track supporting tasks (e.g., DOCS-CONSOLE-23-001, CONSOLE-OBS-52-001) in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
- **Epic 11 – Notifications Studio:** provide notifications workspace with previews and audit trails.
|
||||
- Track supporting tasks (e.g., DOCS-CONSOLE-23-001, CONSOLE-OBS-52-001) in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Current doc refresh tracked in `docs/implplan/SPRINT_0331_0001_0001_docs_modules_ui.md`; statuses mirrored in `docs/modules/ui/TASKS.md`.
|
||||
- Observability evidence for latest demo lives in `operations/observability.md` with Grafana JSON stub under `operations/dashboards/`.
|
||||
- Keep future doc/ops changes mirrored across sprint, TASKS, and module front doors to prevent drift.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Console UI. Replace panels when metrics endpoints are wired; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Console UI Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/ui/operations/observability.md
Normal file
38
docs/modules/ui/operations/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Console UI observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/ui/operations/dashboards/console-ui-observability.json` (import locally; no external data sources assumed).
|
||||
- Panels to include: API latency (p95/p99), error rate, WebSocket/SSE connection count, asset load time, bundle size budget, Core Web Vitals (LCP/FID/CLS), and triage view render time.
|
||||
|
||||
## Key metrics
|
||||
- `console_ui_http_request_duration_seconds_bucket{route}` — API call latency.
|
||||
- `console_ui_http_requests_total{status}` — error rate tracking.
|
||||
- `console_ui_websocket_connections` — active live session count.
|
||||
- `console_ui_bundle_bytes{chunk}` — bundle size by chunk (ensures offline kit budget).
|
||||
- `console_ui_core_web_vitals{metric}` — LCP/FID/CLS gauges.
|
||||
- `console_ui_export_duration_seconds_bucket` — export trigger to download completion.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId` (propagated from API) and `tenant`. Include `feature` (triage, findings, policy) and `route` fields.
|
||||
- Traces disabled by default for air-gap; enable by pointing OTLP endpoint to on-prem collector and setting `Telemetry:ExportEnabled=true`.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (UI backend) must return 200; readiness checks asset storage + API gateway reachability.
|
||||
- `/status` exposes build version, commit, feature flags; ensure it matches the offline bundle manifest when shipping sealed kits.
|
||||
- Frontend self-check: open `/health/ui` to verify core bundles are reachable and integrity hashes match manifest.
|
||||
|
||||
## Alert hints
|
||||
- p99 API latency > 1s for `/api/findings` or `/api/policy`.
|
||||
- SSE/WS disconnect rate > 2% over 5m window.
|
||||
- Bundle size > 3.5 MB for main chunk after gzip (offline kit budget breach).
|
||||
- Core Web Vitals: LCP > 2.5s, CLS > 0.1 on internal demo dataset.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `console-ui`.
|
||||
2) Run `npm run build -- --configuration=production` (or offline kit build) and verify bundle hashes against manifest used by `/health/ui`.
|
||||
3) Fetch `/status` and compare commit/version to the static asset manifest embedded in the offline kit.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0331_0001_0001_docs_modules_ui.md`.
|
||||
- Module front doors: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/console-ui-observability.json`.
|
||||
@@ -8,6 +8,7 @@ VEX Lens computes deterministic consensus across conflicting VEX statements whil
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./runbooks/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Review ./architecture.md for consensus algorithm, trust model, and export contracts.
|
||||
|
||||
@@ -27,6 +27,10 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- `VEXLENS-30-005..007` — expose consensus APIs and export flows, aligning docs with future `/docs/vex/consensus-*.md` deliverables.
|
||||
- `DOCS-VEX-30-001..004` — broader documentation set (overview, algorithm, API) tracked in `docs/TASKS.md`; update this README again once those artefacts merge.
|
||||
|
||||
## Latest updates (2025-11-30)
|
||||
- Docs refresh per `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; added observability runbook stub and TASKS mirror.
|
||||
- Observability assets: `runbooks/observability.md` and `runbooks/dashboards/vex-lens-observability.json` (offline import).
|
||||
|
||||
## Integrations & dependencies
|
||||
- **Excititor** supplies signature-verified VEX observations and issuer hints.
|
||||
- **Policy Engine** consumes consensus verdicts for suppression/waiver logic and exposes trust controls to operators.
|
||||
@@ -44,7 +48,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- [`architecture.md`](architecture.md) — implementation-ready blueprint covering inputs, algorithm, APIs, storage, observability, and exports.
|
||||
- [`implementation_plan.md`](implementation_plan.md) — phased delivery roadmap and acceptance criteria.
|
||||
- [`../../vex/aggregation.md`](../../vex/aggregation.md) — Aggregation-Only Contract boundaries for VEX ingestion and downstream consumers.
|
||||
- Sprint tracking in `docs/implplan/SPRINT_200_documentation_process.md` and module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`.
|
||||
- Sprint tracking in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`; doc TASKS mirror in `docs/modules/vex-lens/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- Epic 7 — VEX Consensus Lens: deterministic VEX adjudication, issuer directory, consensus exports.
|
||||
|
||||
9
docs/modules/vex-lens/TASKS.md
Normal file
9
docs/modules/vex-lens/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# VEX Lens · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| VEX-CONSENSUS-LENS-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refresh with consensus workflow and release links; sprint references added. |
|
||||
| VEX-LENS-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana JSON placeholder added under `runbooks/`. |
|
||||
| VEX-LENS-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both places).
|
||||
@@ -71,13 +71,14 @@ All responses include provenance fields (`consensus_digest`, `derived_from`, DSS
|
||||
- Recompute jobs run via Orchestrator; deterministic ordering ensures identical results for the same input set.
|
||||
- Jobs produce SRM-style manifests for recomputation verification.
|
||||
|
||||
## 7) Observability
|
||||
|
||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
|
||||
- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
|
||||
- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
|
||||
|
||||
## 8) Offline & export
|
||||
|
||||
- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
|
||||
- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
|
||||
## 7) Observability
|
||||
|
||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
|
||||
- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
|
||||
- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
|
||||
- Runbook + dashboard stub (offline import): `runbooks/observability.md`, `runbooks/dashboards/vex-lens-observability.json`.
|
||||
|
||||
## 8) Offline & export
|
||||
|
||||
- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
|
||||
- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
|
||||
|
||||
@@ -61,3 +61,8 @@
|
||||
- Lens service, issuer directory, API/CLI/Console components deployed with telemetry and runbooks.
|
||||
- Documentation set (overview, algorithm, issuer directory, API, console, policy trust) updated with imposed rule statements.
|
||||
- ./TASKS.md and ../../TASKS.md reflect current status; Offline Kit parity confirmed.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; statuses mirrored in `docs/modules/vex-lens/TASKS.md`.
|
||||
- Observability evidence lives in `runbooks/observability.md` with Grafana JSON stub under `runbooks/dashboards/`.
|
||||
- Keep future doc/ops updates mirrored across sprint, TASKS, and module front doors to avoid drift.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for VEX Lens. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "VEX Lens Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/vex-lens/runbooks/observability.md
Normal file
38
docs/modules/vex-lens/runbooks/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# VEX Lens observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/vex-lens/runbooks/dashboards/vex-lens-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: consensus latency, conflict backlog, recompute duration, issuer trust changes, export job success rate, and DSSE verification failures.
|
||||
|
||||
## Key metrics
|
||||
- `vex_consensus_latency_seconds_bucket` — latency from observation intake to consensus write.
|
||||
- `vex_conflict_queue_depth` — size of unresolved conflict queue.
|
||||
- `vex_recompute_duration_seconds_bucket{reason}` — recompute times by trigger (issuer update, policy knob, ingestion delta).
|
||||
- `vex_export_duration_seconds_bucket` — export job runtime.
|
||||
- `vex_dsse_verification_failures_total` — failed attestations during export/ingest.
|
||||
- `vex_consensus_conflicts_total{reason}` — conflict counts by reason (status disagreement, scope mismatch, missing provenance).
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId`, `artifactKey`, `advisoryKey`, and `issuer`. Include `trustTier`, `weightBefore`, `weightAfter`, and `justification` fields for audits.
|
||||
- Traces disabled by default for air-gap; enable by setting `Telemetry:ExportEnabled=true` and pointing OTLP endpoint to on-prem collector.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (service) must return 200; readiness checks Mongo + cache + event bus reachability.
|
||||
- `/status` exposes build version, commit, feature flags; verify it matches offline bundle manifest.
|
||||
- Export self-check: run `stella vex export --format json --manifest out/manifest.json` and validate hashes against manifest entries.
|
||||
|
||||
## Alert hints
|
||||
- Consensus latency p99 > 1.5s over 5m.
|
||||
- Conflict queue depth > 500 for any tenant.
|
||||
- DSSE verification failures > 0 in a 10m window.
|
||||
- Export failure rate > 2% over 10m.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `vex-lens`.
|
||||
2) Run export CLI above and verify `manifest.json` hashes via `jq -r '.files[].sha256'`.
|
||||
3) Fetch `/status` and confirm commit/version match the exported manifest and offline kit bundle metadata.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `runbooks/dashboards/vex-lens-observability.json`.
|
||||
@@ -2,8 +2,10 @@
|
||||
|
||||
Vulnerability Explorer delivers policy-aware triage, investigation, and reporting surfaces for effective findings.
|
||||
|
||||
## Latest updates (2025-11-03)
|
||||
- Access controls refresh introduced attachment signing tokens and updated scope guidance (`docs/updates/2025-11-03-vuln-explorer-access-controls.md`). Ensure operator runbooks reference the new Authority scopes (`authority-scopes.md`) and security checklist before enabling attachment uploads.
|
||||
## Latest updates (2025-11-30)
|
||||
- Documentation refresh aligned to sprint 0334: added observability/runbook snapshot and cross-links to OpenAPI draft (`./api.md`) and schemas in `architecture.md`.
|
||||
- New offline-friendly observability runbook at `runbooks/observability.md` plus stub Grafana JSON in `runbooks/dashboards/`.
|
||||
- Retained 2025-11-03 access-control changes; verify Authority scopes before enabling attachment uploads (`docs/updates/2025-11-03-vuln-explorer-access-controls.md`).
|
||||
|
||||
## Responsibilities
|
||||
- Present policy-evaluated findings with advisory, VEX, SBOM, and runtime context.
|
||||
@@ -22,10 +24,11 @@ Vulnerability Explorer delivers policy-aware triage, investigation, and reportin
|
||||
- Scheduler for remediation/verification jobs.
|
||||
- Notify for triage notifications.
|
||||
|
||||
## Operational notes
|
||||
- Audit logging per Epic 6 requirements.
|
||||
- Offline-ready CSV/PDF exports with deterministic hashes.
|
||||
- Dashboards for MTTR and triage throughput.
|
||||
## Operational notes
|
||||
- Audit logging per Epic 6 requirements.
|
||||
- Offline-ready CSV/PDF exports with deterministic hashes.
|
||||
- Dashboards for MTTR and triage throughput.
|
||||
- Observability runbook and dashboard stub: see `runbooks/observability.md` and `runbooks/dashboards/vuln-explorer-observability.json` (import locally).
|
||||
|
||||
## Epic alignment
|
||||
- Epic 6: Vulnerability Explorer.
|
||||
|
||||
9
docs/modules/vuln-explorer/TASKS.md
Normal file
9
docs/modules/vuln-explorer/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Vuln Explorer · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| VULNERABILITY-EXPLORER-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture updated; OpenAPI/schema/sprint links added; runbook evidence captured. |
|
||||
| VULNERABILITY-EXPLORER-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook + dashboard stub added; health/alert guidance documented. |
|
||||
| VULNERABILITY-EXPLORER-ENG-0001 | DONE (2025-11-30) | Module Team | Sprint alignment notes added to implementation_plan; task mirror created. |
|
||||
|
||||
> Status must mirror `/docs/implplan/SPRINT_0334_0001_0001_docs_modules_vuln_explorer.md` (DOING/DONE/BLOCKED updates go to both files).
|
||||
@@ -55,6 +55,7 @@ CLI mirrors these endpoints (`stella findings list|view|update|export`). Console
|
||||
- Logs: structured with `findingId`, `artifactId`, `advisory`, `policyVersion`, `actor`, `actionType`.
|
||||
- Audit exports: `audit_log.jsonl` appended whenever state changes; offline bundles include signed audit log and manifest.
|
||||
- Compliance: accepted risk requires dual approval and stores justification plus expiry reminders (raised through Notify).
|
||||
- Runbook and dashboard stub for demo snapshot: `runbooks/observability.md` and `runbooks/dashboards/vuln-explorer-observability.json` (offline import).
|
||||
|
||||
## 6) Identity & access integration
|
||||
|
||||
|
||||
@@ -68,3 +68,8 @@
|
||||
- Services, UI/CLI, integrations, exports, and observability deployed with runbooks and Offline Kit parity.
|
||||
- Documentation suite (overview, using-console, API, CLI, findings ledger, policy mapping, VEX/SBOM integration, telemetry, security, runbooks, install) updated with imposed rule statement.
|
||||
- ./TASKS.md and ../../TASKS.md reflect active progress; compliance checklists appended where required.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0334_0001_0001_docs_modules_vuln_explorer.md` (Docs/OPS/ENG streams).
|
||||
- Observability runbook evidence lives under `runbooks/observability.md` with stub dashboard JSON for offline demos.
|
||||
- API/OpenAPI references: `api.md` and `openapi/vuln-explorer.v1.yaml` are the current contract drafts; keep Console/CLI generators synced to these.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for offline import. Populate with panel definitions when metrics endpoints are available; see runbooks/observability.md for expected panels.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Vuln Explorer Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/vuln-explorer/runbooks/observability.md
Normal file
38
docs/modules/vuln-explorer/runbooks/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Vuln Explorer observability runbook (demo snapshot · 2025-11-29)
|
||||
|
||||
## Dashboards (offline-friendly)
|
||||
- Grafana JSON: `docs/modules/vuln-explorer/runbooks/dashboards/vuln-explorer-observability.json` (import locally; no external data sources assumed).
|
||||
- Panels: projection lag, open findings by severity/tenant, accepted-risk ageing, API 5xx rate, export duration p95, ledger replay backlog.
|
||||
|
||||
## Key metrics
|
||||
- `vuln_projection_lag_seconds{tenant}` – seconds between latest ledger event and projector head.
|
||||
- `vuln_findings_open_total{severity,tenant}` – count of open findings by severity.
|
||||
- `vuln_export_duration_seconds_bucket` – histogram for export job runtime.
|
||||
- `vuln_projection_backlog_total` – queued events awaiting projection.
|
||||
- `vuln_triage_actions_total{type}` – immutable triage actions (assign, comment, risk_accept, remediation_note).
|
||||
- `vuln_api_request_duration_seconds_bucket{route}` – API latency for `GET /v1/findings*` and `POST /v1/reports`.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId` and `findingId`. Structured fields: `tenant`, `advisoryKey`, `policyVersion`, `projectId`, `route`.
|
||||
- Trace exemplar anchors: `traceparent` headers are copied into logs; exporters stay disabled by default for air-gap. Enable by setting `Telemetry:ExportEnabled=true` and pointing to on-prem Tempo/Jaeger.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (HTTP 200 expected; readiness checks Mongo + cache reachability).
|
||||
- `/status` returns build version, git commit, and enabled features; safe for anonymous fetch in sealed environments.
|
||||
- Ledger replay check: `GET /v1/findings?projectionMode=verify` emits `X-Vuln-Projection-Head` for quick consistency probes.
|
||||
|
||||
## Alert hints (wire to local Alertmanager or watchdog)
|
||||
- Projection lag > 120s for any tenant.
|
||||
- API p99 latency > 800ms for `GET /v1/findings` or `POST /v1/reports`.
|
||||
- Export failure rate > 2% over 10m window.
|
||||
- Accepted-risk approaching expiry within 7d (emit Notify event `vuln.accepted_risk.expiring`).
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally and point to Prometheus scrape job `vuln-explorer`.
|
||||
2) Run `stella vuln export --format json --manifest out/manifest.json` and validate hashes using `jq -r '.files[].sha256'` against generated bundle.
|
||||
3) Use `curl -s "$BASEURL/status" | jq '{commit,version,features}'` to confirm expected build metadata matches the exported bundle manifest.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint alignment: `docs/implplan/SPRINT_0334_0001_0001_docs_modules_vuln_explorer.md`.
|
||||
- API contract draft: `docs/modules/vuln-explorer/api.md` and OpenAPI at `docs/modules/vuln-explorer/openapi/vuln-explorer.v1.yaml`.
|
||||
- Schema references: `docs/modules/vuln-explorer/architecture.md` (ledger model, VEX decision schemas).
|
||||
@@ -15,6 +15,7 @@ Zastava monitors running workloads, verifies supply chain posture, and enforces
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
- [Observability runbook](./operations/observability.md) (offline import friendly)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
|
||||
@@ -2,7 +2,12 @@
|
||||
|
||||
Zastava monitors running workloads, verifies supply chain posture, and enforces runtime policy via Kubernetes admission webhooks.
|
||||
|
||||
## Responsibilities
|
||||
## Latest updates (2025-11-30)
|
||||
- Sprint tracker `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md` and module `TASKS.md` added to mirror status.
|
||||
- Observability runbook stub + dashboard placeholder added under `operations/` (offline import).
|
||||
- Surface.Env/Surface.Secrets adoption remains pending platform contracts; align with platform docs before enabling sealed mode.
|
||||
|
||||
## Responsibilities
|
||||
- Observe node/container activity and emit runtime events.
|
||||
- Validate signatures, SBOM presence, and backend verdicts before allowing containers.
|
||||
- Buffer and replay events during disconnections.
|
||||
@@ -18,10 +23,10 @@ Zastava monitors running workloads, verifies supply chain posture, and enforces
|
||||
- Scanner/Scheduler for remediation triggers.
|
||||
- Notify/UI for runtime alerts and dashboards.
|
||||
|
||||
## Operational notes
|
||||
- Runbook ./operations/runtime.md with Grafana/Prometheus assets.
|
||||
- Offline kit assets bundling webhook charts.
|
||||
- DPoP/mTLS rotation guidance shared with Authority.
|
||||
## Operational notes
|
||||
- Runbook `./operations/observability.md` (stub) plus dashboard placeholder `./operations/dashboards/zastava-observability.json`.
|
||||
- Legacy runtime runbook assets remain under ./operations if present; keep offline kit bundles deterministic.
|
||||
- DPoP/mTLS rotation guidance shared with Authority.
|
||||
|
||||
## Related resources
|
||||
- ./operations/runtime.md
|
||||
|
||||
9
docs/modules/zastava/TASKS.md
Normal file
9
docs/modules/zastava/TASKS.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Zastava · TASKS (status mirror)
|
||||
|
||||
| Task ID | Status | Owner(s) | Notes / Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| ZASTAVA-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refreshed; Surface Env/Secrets and sprint links added. |
|
||||
| ZASTAVA-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md`. |
|
||||
| ZASTAVA-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana JSON placeholder added under `operations/`. |
|
||||
|
||||
> Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both places).
|
||||
@@ -486,11 +486,20 @@ webhooks:
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap
|
||||
|
||||
* **eBPF** option for syscall/library load tracing (kernel‑level, opt‑in).
|
||||
* **Windows containers** support (ETW providers, loaded modules).
|
||||
* **Network posture** checks: listening ports vs policy.
|
||||
* **Live **used‑by‑entrypoint** synthesis**: send compact bitset diff to backend to tighten Usage view.
|
||||
* **Admission dry‑run** dashboards (simulate block lists before enforcing).
|
||||
## 15) Roadmap
|
||||
|
||||
* **eBPF** option for syscall/library load tracing (kernel‑level, opt‑in).
|
||||
* **Windows containers** support (ETW providers, loaded modules).
|
||||
* **Network posture** checks: listening ports vs policy.
|
||||
* **Live **used‑by‑entrypoint** synthesis**: send compact bitset diff to backend to tighten Usage view.
|
||||
* **Admission dry‑run** dashboards (simulate block lists before enforcing).
|
||||
|
||||
---
|
||||
|
||||
## 16) Observability (stub)
|
||||
|
||||
- Runbook + dashboard placeholder for offline import: `operations/observability.md`, `operations/dashboards/zastava-observability.json`.
|
||||
- Metrics to surface: admission latency p95/p99, allow/deny counts, Surface.Env miss rate, Surface.Secrets failures, Surface.FS cache freshness, drift events.
|
||||
- Health endpoints: `/health/liveness`, `/health/readiness`, `/status`, `/surface/fs/cache/status` (see runbook).
|
||||
- Alert hints: deny spikes, latency > 800ms p99, cache freshness lag > 10m, any secrets failure.
|
||||
|
||||
|
||||
@@ -13,7 +13,12 @@
|
||||
- ZASTAVA runtime tasks in ../../TASKS.md.
|
||||
- Webhook smoke tests tracked in src/Zastava/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in `/docs/implplan/SPRINT_*.md`.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
## Sprint alignment (2025-11-30)
|
||||
- Docs refresh tracked in `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md`; statuses mirrored in `docs/modules/zastava/TASKS.md`.
|
||||
- Observability evidence lives in `operations/observability.md` with Grafana JSON stub under `operations/dashboards/`.
|
||||
- Surface.Env/Surface.Secrets contracts remain dependencies; align with platform docs before enabling sealed mode.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_note": "Placeholder Grafana dashboard stub for Zastava. Replace panels when metrics endpoints are available; keep offline-import friendly.",
|
||||
"schemaVersion": 39,
|
||||
"title": "Zastava Observability (stub)",
|
||||
"panels": []
|
||||
}
|
||||
38
docs/modules/zastava/operations/observability.md
Normal file
38
docs/modules/zastava/operations/observability.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Zastava observability runbook (stub · 2025-11-29 demo)
|
||||
|
||||
## Dashboards (offline import)
|
||||
- Grafana JSON: `docs/modules/zastava/operations/dashboards/zastava-observability.json` (import locally; no external data sources assumed).
|
||||
- Planned panels: admission decision rate, webhook latency p95/p99, cache freshness (Surface.FS), Surface.Env key misses, Secrets fetch failures, policy violation counts, and drift events.
|
||||
|
||||
## Key metrics
|
||||
- `zastava_admission_latency_seconds_bucket{webhook}` — admission webhook latency.
|
||||
- `zastava_admission_decisions_total{result}` — allow/deny counts.
|
||||
- `zastava_surface_env_miss_total` — Surface.Env key misses.
|
||||
- `zastava_surface_secrets_failures_total{reason}` — secret retrieval failures.
|
||||
- `zastava_surface_fs_cache_freshness_seconds` — cache age vs Scanner surface metadata.
|
||||
- `zastava_drift_events_total{type}` — drift detections by category.
|
||||
|
||||
## Logs & traces
|
||||
- Correlate by `correlationId`, `tenant`, `cluster`, and `admissionId`. Include `policyVersion`, `surfaceEnvProfile`, and `secretsProvider` fields.
|
||||
- Traces disabled by default for air-gap; enable via `Telemetry:ExportEnabled=true` pointing to on-prem collector.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (webhook + observer) check cache reachability, Secrets provider connectivity, and policy fetch.
|
||||
- `/status` exposes build version, commit, feature flags; verify against offline bundle manifest.
|
||||
- Cache probe: `GET /surface/fs/cache/status` returns freshness and hash for cached surfaces.
|
||||
|
||||
## Alert hints
|
||||
- Admission latency p99 > 800ms.
|
||||
- Deny rate spike > 5% over 10m without policy change.
|
||||
- Surface.Env miss rate > 1% or Secrets failure > 0 over 10m.
|
||||
- Cache freshness > 10m behind Scanner surface metadata.
|
||||
|
||||
## Offline verification steps
|
||||
1) Import Grafana JSON locally; point to Prometheus scrape labeled `zastava`.
|
||||
2) Replay a sealed admission bundle and verify `/status` + cache probe hashes match the manifest in the offline kit.
|
||||
3) Run webhook smoke (`kubectl apply --dry-run=server -f samples/admission-request.yaml`) and confirm metrics increment locally.
|
||||
|
||||
## Evidence locations
|
||||
- Sprint tracker: `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md`.
|
||||
- Module docs: `README.md`, `architecture.md`, `implementation_plan.md`.
|
||||
- Dashboard stub: `operations/dashboards/zastava-observability.json`.
|
||||
Reference in New Issue
Block a user