Align AOC tasks for Excititor and Concelier
This commit is contained in:
master
root
68
.gitignore
vendored
68
.gitignore
vendored
@@ -1,34 +1,34 @@
|
||||
# Build outputs
|
||||
bin/
|
||||
obj/
|
||||
*.pdb
|
||||
*.dll
|
||||
|
||||
# IDE state
|
||||
.vs/
|
||||
*.user
|
||||
*.suo
|
||||
*.userprefs
|
||||
|
||||
# Rider/VSCode
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Packages and logs
|
||||
*.log
|
||||
TestResults/
|
||||
|
||||
.dotnet
|
||||
.DS_Store
|
||||
seed-data/ics-cisa/*.csv
|
||||
seed-data/ics-cisa/*.xlsx
|
||||
seed-data/ics-cisa/*.sha256
|
||||
seed-data/cert-bund/**/*.json
|
||||
seed-data/cert-bund/**/*.sha256
|
||||
|
||||
out/offline-kit/web/**/*
|
||||
**/node_modules/**/*
|
||||
**/.angular/**/*
|
||||
**/.cache/**/*
|
||||
**/dist/**/*
|
||||
tmp/**/*
|
||||
# Build outputs
|
||||
bin/
|
||||
obj/
|
||||
*.pdb
|
||||
*.dll
|
||||
|
||||
# IDE state
|
||||
.vs/
|
||||
*.user
|
||||
*.suo
|
||||
*.userprefs
|
||||
|
||||
# Rider/VSCode
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Packages and logs
|
||||
*.log
|
||||
TestResults/
|
||||
|
||||
.dotnet
|
||||
.DS_Store
|
||||
seed-data/ics-cisa/*.csv
|
||||
seed-data/ics-cisa/*.xlsx
|
||||
seed-data/ics-cisa/*.sha256
|
||||
seed-data/cert-bund/**/*.json
|
||||
seed-data/cert-bund/**/*.sha256
|
||||
|
||||
out/offline-kit/web/**/*
|
||||
**/node_modules/**/*
|
||||
**/.angular/**/*
|
||||
**/.cache/**/*
|
||||
**/dist/**/*
|
||||
tmp/**/*
|
||||
|
||||
412
AGENTS.md
412
AGENTS.md
@@ -1,206 +1,206 @@
|
||||
# 1) What is StellaOps?
|
||||
|
||||
**StellaOps** an next-gen and sovereign container-security toolkit built for high-speed, offline operation, released under AGPL-3.0-or-later.
|
||||
|
||||
Stella Ops is a self-hostable, sovereign container-security platform that makes proof—not promises—default. It binds every container digest to content-addressed SBOMs (SBOM 3.0.0 and CycloneDX 1.6), in-toto/DSSE attestations, and optional Sigstore Rekor transparency, then layers deterministic, replayable scanning with entry-trace and VEX-first decisioning. “Next-gen” means findings are reproducible and explainable, exploitability is modeled in OpenVEX and merged with lattice logic for stable outcomes, and the same workflow runs online or fully air-gapped. “Sovereign” means cryptographic and operational independence: bring-your-own trust roots, regional crypto readiness (eIDAS/FIPS/GOST/SM), offline bundles, and post-quantum-ready modes—so regulated orgs can comply without phoning home.
|
||||
|
||||
Our principles and goals are simple: authenticity & integrity by default, provenance attached to digests, transparency for tamper-evidence, determinism & replay for audits, explainability engineers can act on, and exploitability over enumeration to cut noise. We minimize trust and blast radius with short-lived keys, least-privilege, and content-addressed caches; we stay air-gap friendly with mirrored feeds; and we keep governance honest with reviewable OPA/Rego policy gates and VEX-based waivers. The result is a platform that shortens time-to-truth, makes risk measurable, and lets you ship with confidence—anywhere, under any sovereignty requirement.
|
||||
More documention is available ./docs/*.md files. Read `docs/README.md` to gather information about the available documentation. You could inquiry specific documents as your work requires it
|
||||
|
||||
---
|
||||
|
||||
# 3) Practices
|
||||
|
||||
## 3.1) Naming
|
||||
All modules are .NET projects based on .NET 10 (preview). Exclussion is the UI. It is based on Angular
|
||||
All modules are contained by one or more projects. Each project goes in its dedicated folder. Each project starts with StellaOps.<ModuleName>. In case it is common for for all StellaOps modules it is library or plugin and it is named StellaOps.<LibraryOrPlugin>.
|
||||
|
||||
## 3.2) Key technologies & integrations
|
||||
|
||||
- **Runtime**: .NET 10 (`net10.0`) preview SDK; C# latest preview features. Any dependencies like Microsoft.* should strive to be closests version.
|
||||
- **Nuget**: Try to re-use / cache nugets to /local-nugets
|
||||
- **Data**: MongoDB (canonical store and job/export state). MongoDB driver version should be > 3.0
|
||||
- **Observability**: structured logs, counters, and (optional) OpenTelemetry traces.
|
||||
- **Ops posture**: offline‑first, allowlist for remote hosts, strict schema validation, gated LLM fallback (only where explicitly configured).
|
||||
|
||||
# 4) Modules
|
||||
StellaOps ships as containerised building blocks; each module owns a clear boundary and has its own code folder, deployable image, and deep-dive architecture dossier.
|
||||
|
||||
| Module | Primary path(s) | Key doc |
|
||||
|--------|-----------------|---------|
|
||||
| Authority | `src/Authority/StellaOps.Authority`<br>`src/Authority/StellaOps.Authority.Plugin.*` | `docs/modules/authority/architecture.md` |
|
||||
| Signer | `src/Signer/StellaOps.Signer` | `docs/modules/signer/architecture.md` |
|
||||
| Attestor | `src/Attestor/StellaOps.Attestor`<br>`src/Attestor/StellaOps.Attestor.Verify` | `docs/modules/attestor/architecture.md` |
|
||||
| Concelier | `src/Concelier/StellaOps.Concelier.WebService`<br>`src/Concelier/__Libraries/StellaOps.Concelier.*` | `docs/modules/concelier/architecture.md` |
|
||||
| Excititor | `src/Excititor/StellaOps.Excititor.WebService`<br>`src/Excititor/__Libraries/StellaOps.Excititor.*` | `docs/modules/excititor/architecture.md` |
|
||||
| Policy Engine | `src/Policy/StellaOps.Policy.Engine`<br>`src/Policy/__Libraries/StellaOps.Policy.*` | `docs/modules/policy/architecture.md` |
|
||||
| Scanner | `src/Scanner/StellaOps.Scanner.WebService`<br>`src/Scanner/StellaOps.Scanner.Worker`<br>`src/Scanner/__Libraries/StellaOps.Scanner.*` | `docs/modules/scanner/architecture.md` |
|
||||
| Scheduler | `src/Scheduler/StellaOps.Scheduler.WebService`<br>`src/Scheduler/StellaOps.Scheduler.Worker` | `docs/modules/scheduler/architecture.md` |
|
||||
| CLI | `src/Cli/StellaOps.Cli`<br>`src/Cli/StellaOps.Cli.Core`<br>`src/Cli/StellaOps.Cli.Plugins.*` | `docs/modules/cli/architecture.md` |
|
||||
| UI / Console | `src/UI/StellaOps.UI` | `docs/modules/ui/architecture.md` |
|
||||
| Notify | `src/Notify/StellaOps.Notify.WebService`<br>`src/Notify/StellaOps.Notify.Worker` | `docs/modules/notify/architecture.md` |
|
||||
| Export Center | `src/ExportCenter/StellaOps.ExportCenter.WebService`<br>`src/ExportCenter/StellaOps.ExportCenter.Worker` | `docs/modules/export-center/architecture.md` |
|
||||
| Registry Token Service | `src/Registry/StellaOps.Registry.TokenService`<br>`src/Registry/__Tests/StellaOps.Registry.TokenService.Tests` | `docs/modules/registry/architecture.md` |
|
||||
| Advisory AI | `src/AdvisoryAI/StellaOps.AdvisoryAI` | `docs/modules/advisory-ai/architecture.md` |
|
||||
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | `docs/modules/orchestrator/architecture.md` |
|
||||
| Vulnerability Explorer | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | `docs/modules/vuln-explorer/architecture.md` |
|
||||
| VEX Lens | `src/VexLens/StellaOps.VexLens` | `docs/modules/vex-lens/architecture.md` |
|
||||
| Graph Explorer | `src/Graph/StellaOps.Graph.Api`<br>`src/Graph/StellaOps.Graph.Indexer` | `docs/modules/graph/architecture.md` |
|
||||
| Telemetry Stack | `ops/devops/telemetry` | `docs/modules/telemetry/architecture.md` |
|
||||
| DevOps / Release | `ops/devops` | `docs/modules/devops/architecture.md` |
|
||||
| Platform | *(cross-cutting docs)* | `docs/modules/platform/architecture-overview.md` |
|
||||
| CI Recipes | *(pipeline templates)* | `docs/modules/ci/architecture.md` |
|
||||
| Zastava | `src/Zastava/StellaOps.Zastava.Observer`<br>`src/Zastava/StellaOps.Zastava.Webhook`<br>`src/Zastava/StellaOps.Zastava.Core` | `docs/modules/zastava/architecture.md` |
|
||||
|
||||
## 4.1 Module cheat sheet
|
||||
|
||||
### Authority
|
||||
- **Path:** `src/Authority/StellaOps.Authority`, plugins in `src/Authority/StellaOps.Authority.Plugin.*`.
|
||||
- **Docs:** `docs/modules/authority/architecture.md`.
|
||||
- **Responsibilities:** Issues short-lived, sender-constrained OpToks (DPoP/mTLS) for services, CLI, and UI; exposes OIDC discovery, device-code, and auth-code flows.
|
||||
- **Key traits:** Ed25519/ES256 signing with JWKS rotation, tenant-aware scopes, stateless JWT validation, optional introspection, and structured audit trails.
|
||||
|
||||
### Signer
|
||||
- **Path:** `src/Signer/StellaOps.Signer`.
|
||||
- **Docs:** `docs/modules/signer/architecture.md`.
|
||||
- **Responsibilities:** Authenticates callers, enforces Proof-of-Entitlement, verifies scanner release signatures, and returns DSSE bundles for SBOMs and reports.
|
||||
- **Key traits:** Supports keyless (Fulcio) and keyful (KMS/HSM) signing, applies plan quotas, stores audit trails, and delegates Rekor logging to the Attestor.
|
||||
|
||||
### Attestor
|
||||
- **Path:** `src/Attestor/StellaOps.Attestor`, proof helpers in `src/Attestor/StellaOps.Attestor.Verify`.
|
||||
- **Docs:** `docs/modules/attestor/architecture.md`.
|
||||
- **Responsibilities:** Submits DSSE bundles to Rekor v2, caches `{uuid, index, proof}`, and serves verification bundles to Scanner, UI, CLI, and Export Center.
|
||||
- **Key traits:** mTLS + OpTok enforcement for Signer-only submissions, Mongo/Redis idempotency, optional DSSE archive mirroring, and resilient retry/backoff.
|
||||
|
||||
### Concelier
|
||||
- **Path:** `src/Concelier/StellaOps.Concelier.WebService` with connectors/exporters under `src/Concelier/__Libraries/StellaOps.Concelier.*`.
|
||||
- **Docs:** `docs/modules/concelier/architecture.md`.
|
||||
- **Responsibilities:** Applies the Aggregation-Only Contract to ingest advisories, produce immutable observations, correlate linksets, and publish deterministic exports.
|
||||
- **Key traits:** Restart-time connectors/exporters, Mongo-backed scheduling, canonical JSON/Trivy outputs, Offline Kit parity, and hash-stable manifests.
|
||||
|
||||
### Excititor
|
||||
- **Path:** `src/Excititor/StellaOps.Excititor.WebService`, connectors/adapters in `src/Excititor/__Libraries/StellaOps.Excititor.*`.
|
||||
- **Docs:** `docs/modules/excititor/architecture.md`.
|
||||
- **Responsibilities:** Normalises VEX statements into observations, builds provenance-rich linksets, and surfaces consensus/conflicts for policy suppression.
|
||||
- **Key traits:** Aggregation-only guardrails, restart-time plug-ins, Mongo persistence, deterministic exports, and Offline Kit-ready bundles.
|
||||
|
||||
### Policy Engine
|
||||
- **Path:** `src/Policy/StellaOps.Policy.Engine`, shared libraries under `src/Policy/__Libraries/StellaOps.Policy.*`.
|
||||
- **Docs:** `docs/modules/policy/architecture.md`.
|
||||
- **Responsibilities:** Evaluates `stella-dsl@1` policies, joins SBOM/advisory/VEX evidence, materialises effective findings, and emits explain traces.
|
||||
- **Key traits:** Deterministic evaluation (no wall clock), change-stream driven increments, simulation endpoints, and Authority-scoped tenancy/RBAC enforcement.
|
||||
|
||||
### Scanner.WebService
|
||||
- **Path:** `src/Scanner/StellaOps.Scanner.WebService`.
|
||||
- **Docs:** `docs/modules/scanner/architecture.md`.
|
||||
- **Responsibilities:** Hosts scan/diff/export APIs, enqueues work, serves SBOM and diff artifacts, and publishes DSSE-ready report metadata.
|
||||
- **Key traits:** Minimal APIs with Redis/NATS queue clients, RustFS artifact integration, BOM-index lookups, and DSSE hand-off to Signer/Attestor.
|
||||
|
||||
### Scanner.Worker
|
||||
- **Path:** `src/Scanner/StellaOps.Scanner.Worker` with analyzers/caches in `src/Scanner/__Libraries/StellaOps.Scanner.*`.
|
||||
- **Docs:** `docs/modules/scanner/architecture.md`.
|
||||
- **Responsibilities:** Runs deterministic OS/language/native analyzers per layer, composes inventory and usage SBOM fragments, and streams them back to the catalog.
|
||||
- **Key traits:** Layer/file CAS caching, restart-time analyzer plug-ins under `plugins/scanner/**`, bounded retries with lease renewals, and DSSE-ready outputs.
|
||||
|
||||
### Scheduler
|
||||
- **Path:** `src/Scheduler/StellaOps.Scheduler.WebService`, `src/Scheduler/StellaOps.Scheduler.Worker`.
|
||||
- **Docs:** `docs/modules/scheduler/architecture.md`.
|
||||
- **Responsibilities:** Detects advisory/VEX deltas, selects impacted assets via BOM index, and schedules analysis-only runs toward Scanner and Policy Engine.
|
||||
- **Key traits:** Mongo impact cursors, Redis/NATS orchestration, webhook fan-out (Policy/Notify/Runtime), and deterministic evaluation windows.
|
||||
|
||||
### CLI
|
||||
- **Path:** `src/Cli/StellaOps.Cli`, helpers in `src/Cli/StellaOps.Cli.Core`, plug-ins in `src/Cli/StellaOps.Cli.Plugins.*`.
|
||||
- **Docs:** `docs/modules/cli/architecture.md`.
|
||||
- **Responsibilities:** Provides deterministic verbs for scan/diff/export/report, Buildx SBOM orchestration, policy/VEX administration, and offline kit workflows.
|
||||
- **Key traits:** Native AOT binaries, device-code/client-credential login with DPoP storage, golden-output tests, and restart-time plug-in manifests in `plugins/cli/**`.
|
||||
|
||||
### UI
|
||||
- **Path:** `src/UI/StellaOps.UI`.
|
||||
- **Docs:** `docs/modules/ui/architecture.md`.
|
||||
- **Responsibilities:** Angular SPA for scans, policy authoring, VEX evidence exploration, runtime posture, and admin tooling via backend APIs.
|
||||
- **Key traits:** Angular Signals with `@ngrx/signals`, typed API clients handling DPoP + SSE, Tailwind theming, and immutable content-hashed bundles.
|
||||
|
||||
### Notify
|
||||
- **Path:** `src/Notify/StellaOps.Notify.WebService`, `src/Notify/StellaOps.Notify.Worker`, connectors in `src/Notify/__Libraries`.
|
||||
- **Docs:** `docs/modules/notify/architecture.md`.
|
||||
- **Responsibilities:** Evaluates notification rules on platform events, renders channel-specific payloads, and delivers messages with throttling/digests.
|
||||
- **Key traits:** Tenant-scoped rule engine, idempotent delivery queues, secrets referenced rather than stored, and comprehensive audit/metrics coverage.
|
||||
|
||||
### Export Center
|
||||
- **Path:** `src/ExportCenter/StellaOps.ExportCenter.WebService`, `src/ExportCenter/StellaOps.ExportCenter.Worker`, adapters in `src/ExportCenter/StellaOps.ExportCenter.*`.
|
||||
- **Docs:** `docs/modules/export-center/architecture.md`.
|
||||
- **Responsibilities:** Packages reproducible evidence bundles (JSON, Trivy, mirror) with provenance, signing, and distribution manifests for offline or mirror deployments.
|
||||
- **Key traits:** Profile-driven exports, Orchestrator-backed job leases, Mongo/object storage staging, and cosign-compatible provenance/signature emission.
|
||||
|
||||
### Registry Token Service
|
||||
- **Path:** `src/Registry/StellaOps.Registry.TokenService`, with integration tests in `src/Registry/__Tests/StellaOps.Registry.TokenService.Tests`.
|
||||
- **Docs:** `docs/modules/registry/operations/token-service.md`.
|
||||
- **Responsibilities:** Issues scoped pull tokens for container/image registries, enforces licence/plan constraints, and publishes audit telemetry for token usage.
|
||||
- **Key traits:** Authority-issued OpTok validation, Mongo-backed issuance ledger, deterministic checksum manifests for Offline Kit bundles, and emergency revoke/rotation tooling.
|
||||
|
||||
### Zastava
|
||||
- **Path:** `src/Zastava/StellaOps.Zastava.Observer`, `src/Zastava/StellaOps.Zastava.Webhook`, shared contracts in `src/Zastava/StellaOps.Zastava.Core`.
|
||||
- **Docs:** `docs/modules/zastava/architecture.md`.
|
||||
- **Responsibilities:** Observes running workloads, emits runtime posture events, and enforces admission-time policy (signed images, SBOM availability, policy verdict).
|
||||
- **Key traits:** Authority-issued OpToks with DPoP/mTLS, ND-JSON batching with local buffering, delta-scan triggers on drift, and Kubernetes webhook enforcement.
|
||||
|
||||
---
|
||||
|
||||
### 4.1.4) Glossary (quick)
|
||||
|
||||
- **OVAL** — Vendor/distro security definition format; authoritative for OS packages.
|
||||
- **NEVRA / EVR** — RPM and Debian version semantics for OS packages.
|
||||
- **PURL / SemVer** — Coordinates and version semantics for OSS ecosystems.
|
||||
- **KEV** — Known Exploited Vulnerabilities (flag only).
|
||||
|
||||
---
|
||||
# 5) Your role as StellaOps contributor
|
||||
|
||||
You acting as information technology engineer that will take different type of roles in goal achieving StellaOps production implementation
|
||||
In order you to work - you have to be supplied with directory that contains `AGENTS.md`,`TASKS.md` files. There will you have more information about the role you have, the scope of your work and the tasks you will have.
|
||||
|
||||
Boundaries:
|
||||
- You operate only in the working directories I gave you, unless there is dependencies that makes you to work on dependency in shared directory. Then you ask for confirmation.
|
||||
|
||||
You main characteristics:
|
||||
- Keep endpoints small, deterministic, and cancellation-aware.
|
||||
- Improve logs/metrics as per tasks.
|
||||
- Update `TASKS.md` when moving tasks forward.
|
||||
- When you are done with all task you state explicitly you are done.
|
||||
- Impersonate the role described on working directory `AGENTS.md` you will read, if role is not available - take role of the CTO of the StellaOps in early stages.
|
||||
- You always strive for best practices
|
||||
- You always strive for re-usability
|
||||
- When in doubt of design decision - you ask then act
|
||||
- You are autonomus - meaning that you will work for long time alone and achieve maximum without stopping for stupid questions
|
||||
- You operate on the same directory where other agents will work. In case you need to work on directory that is dependency on provided `AGENTS.md`,`TASKS.md` files you have to ask for confirmation first.
|
||||
|
||||
## 5.1) Type of contributions
|
||||
|
||||
- **BE‑Base (Platform & Pipeline)**
|
||||
Owns DI, plugin host, job scheduler/coordinator, configuration binding, minimal API endpoints, and Mongo bootstrapping.
|
||||
- **BE‑Conn‑X (Connectors)**
|
||||
One agent per source family (NVD, Red Hat, Ubuntu, Debian, SUSE, GHSA, OSV, PSIRTs, CERTs, KEV, ICS). Implements fetch/parse/map with incremental watermarks.
|
||||
- **BE‑Merge (Canonical Merge & Dedupe)**
|
||||
Identity graph, precedence policies, canonical JSON serializer, and deterministic hashing (`merge_event`).
|
||||
- **BE‑Export (JSON & Trivy DB)**
|
||||
Deterministic export trees, Trivy DB packaging, optional ORAS push, and offline bundle.
|
||||
- **QA (Validation & Observability)**
|
||||
Schema tests, fixture goldens, determinism checks, metrics/logs/traces, e2e reproducibility runs.
|
||||
- **DevEx/Docs**
|
||||
Maintains this agent framework, templates, and per‑directory guides; assists parallelization and reviews.
|
||||
|
||||
|
||||
## 5.2) Work rules (important)
|
||||
|
||||
- **Directory ownership**: Each agent works **only inside its module directory**. Cross‑module edits require a brief handshake in issues/PR description.
|
||||
- **Scoping**: Use each module’s `AGENTS.md` and `TASKS.md` to plan; autonomous agents must read `src/AGENTS.md` and the module docs before acting.
|
||||
- **Determinism**: Sort keys, normalize timestamps to UTC ISO‑8601, avoid non‑deterministic data in exports and tests.
|
||||
- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md and in ./SPRINTS.md file.
|
||||
- **Coordination**: In case task is discovered as blocked on other team or task, according TASKS.md files that dependency is on needs to be changed by adding new tasks describing the requirement. the current task must be updated as completed. In case task changes, scope or requirements or rules - other documentations needs be updated accordingly.
|
||||
- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and the relevant TASKS.md file. Always check the AGENTS.md in the relevant TASKS.md directory.
|
||||
- **Tests**: Add/extend fixtures and unit tests per change; never regress determinism or precedence.
|
||||
- **Test layout**: Use module-specific projects in `StellaOps.Concelier.<Component>.Tests`; shared fixtures/harnesses live in `StellaOps.Concelier.Testing`.
|
||||
- **Execution autonomous**: In case you need to continue with more than one options just continue sequentially, unless the continue requires design decision.
|
||||
- **Additional references**: When a task mentions historical epics, consult the corresponding module guides or domain playbooks under `docs/modules/**`, `docs/api/`, `docs/risk/`, or `docs/airgap/` for the latest specification.
|
||||
|
||||
---
|
||||
# 1) What is StellaOps?
|
||||
|
||||
**StellaOps** an next-gen and sovereign container-security toolkit built for high-speed, offline operation, released under AGPL-3.0-or-later.
|
||||
|
||||
Stella Ops is a self-hostable, sovereign container-security platform that makes proof—not promises—default. It binds every container digest to content-addressed SBOMs (SBOM 3.0.0 and CycloneDX 1.6), in-toto/DSSE attestations, and optional Sigstore Rekor transparency, then layers deterministic, replayable scanning with entry-trace and VEX-first decisioning. “Next-gen” means findings are reproducible and explainable, exploitability is modeled in OpenVEX and merged with lattice logic for stable outcomes, and the same workflow runs online or fully air-gapped. “Sovereign” means cryptographic and operational independence: bring-your-own trust roots, regional crypto readiness (eIDAS/FIPS/GOST/SM), offline bundles, and post-quantum-ready modes—so regulated orgs can comply without phoning home.
|
||||
|
||||
Our principles and goals are simple: authenticity & integrity by default, provenance attached to digests, transparency for tamper-evidence, determinism & replay for audits, explainability engineers can act on, and exploitability over enumeration to cut noise. We minimize trust and blast radius with short-lived keys, least-privilege, and content-addressed caches; we stay air-gap friendly with mirrored feeds; and we keep governance honest with reviewable OPA/Rego policy gates and VEX-based waivers. The result is a platform that shortens time-to-truth, makes risk measurable, and lets you ship with confidence—anywhere, under any sovereignty requirement.
|
||||
More documention is available ./docs/*.md files. Read `docs/README.md` to gather information about the available documentation. You could inquiry specific documents as your work requires it
|
||||
|
||||
---
|
||||
|
||||
# 3) Practices
|
||||
|
||||
## 3.1) Naming
|
||||
All modules are .NET projects based on .NET 10 (preview). Exclussion is the UI. It is based on Angular
|
||||
All modules are contained by one or more projects. Each project goes in its dedicated folder. Each project starts with StellaOps.<ModuleName>. In case it is common for for all StellaOps modules it is library or plugin and it is named StellaOps.<LibraryOrPlugin>.
|
||||
|
||||
## 3.2) Key technologies & integrations
|
||||
|
||||
- **Runtime**: .NET 10 (`net10.0`) preview SDK; C# latest preview features. Any dependencies like Microsoft.* should strive to be closests version.
|
||||
- **Nuget**: Try to re-use / cache nugets to /local-nugets
|
||||
- **Data**: MongoDB (canonical store and job/export state). MongoDB driver version should be > 3.0
|
||||
- **Observability**: structured logs, counters, and (optional) OpenTelemetry traces.
|
||||
- **Ops posture**: offline‑first, allowlist for remote hosts, strict schema validation, gated LLM fallback (only where explicitly configured).
|
||||
|
||||
# 4) Modules
|
||||
StellaOps ships as containerised building blocks; each module owns a clear boundary and has its own code folder, deployable image, and deep-dive architecture dossier.
|
||||
|
||||
| Module | Primary path(s) | Key doc |
|
||||
|--------|-----------------|---------|
|
||||
| Authority | `src/Authority/StellaOps.Authority`<br>`src/Authority/StellaOps.Authority.Plugin.*` | `docs/modules/authority/architecture.md` |
|
||||
| Signer | `src/Signer/StellaOps.Signer` | `docs/modules/signer/architecture.md` |
|
||||
| Attestor | `src/Attestor/StellaOps.Attestor`<br>`src/Attestor/StellaOps.Attestor.Verify` | `docs/modules/attestor/architecture.md` |
|
||||
| Concelier | `src/Concelier/StellaOps.Concelier.WebService`<br>`src/Concelier/__Libraries/StellaOps.Concelier.*` | `docs/modules/concelier/architecture.md` |
|
||||
| Excititor | `src/Excititor/StellaOps.Excititor.WebService`<br>`src/Excititor/__Libraries/StellaOps.Excititor.*` | `docs/modules/excititor/architecture.md` |
|
||||
| Policy Engine | `src/Policy/StellaOps.Policy.Engine`<br>`src/Policy/__Libraries/StellaOps.Policy.*` | `docs/modules/policy/architecture.md` |
|
||||
| Scanner | `src/Scanner/StellaOps.Scanner.WebService`<br>`src/Scanner/StellaOps.Scanner.Worker`<br>`src/Scanner/__Libraries/StellaOps.Scanner.*` | `docs/modules/scanner/architecture.md` |
|
||||
| Scheduler | `src/Scheduler/StellaOps.Scheduler.WebService`<br>`src/Scheduler/StellaOps.Scheduler.Worker` | `docs/modules/scheduler/architecture.md` |
|
||||
| CLI | `src/Cli/StellaOps.Cli`<br>`src/Cli/StellaOps.Cli.Core`<br>`src/Cli/StellaOps.Cli.Plugins.*` | `docs/modules/cli/architecture.md` |
|
||||
| UI / Console | `src/UI/StellaOps.UI` | `docs/modules/ui/architecture.md` |
|
||||
| Notify | `src/Notify/StellaOps.Notify.WebService`<br>`src/Notify/StellaOps.Notify.Worker` | `docs/modules/notify/architecture.md` |
|
||||
| Export Center | `src/ExportCenter/StellaOps.ExportCenter.WebService`<br>`src/ExportCenter/StellaOps.ExportCenter.Worker` | `docs/modules/export-center/architecture.md` |
|
||||
| Registry Token Service | `src/Registry/StellaOps.Registry.TokenService`<br>`src/Registry/__Tests/StellaOps.Registry.TokenService.Tests` | `docs/modules/registry/architecture.md` |
|
||||
| Advisory AI | `src/AdvisoryAI/StellaOps.AdvisoryAI` | `docs/modules/advisory-ai/architecture.md` |
|
||||
| Orchestrator | `src/Orchestrator/StellaOps.Orchestrator` | `docs/modules/orchestrator/architecture.md` |
|
||||
| Vulnerability Explorer | `src/VulnExplorer/StellaOps.VulnExplorer.Api` | `docs/modules/vuln-explorer/architecture.md` |
|
||||
| VEX Lens | `src/VexLens/StellaOps.VexLens` | `docs/modules/vex-lens/architecture.md` |
|
||||
| Graph Explorer | `src/Graph/StellaOps.Graph.Api`<br>`src/Graph/StellaOps.Graph.Indexer` | `docs/modules/graph/architecture.md` |
|
||||
| Telemetry Stack | `ops/devops/telemetry` | `docs/modules/telemetry/architecture.md` |
|
||||
| DevOps / Release | `ops/devops` | `docs/modules/devops/architecture.md` |
|
||||
| Platform | *(cross-cutting docs)* | `docs/modules/platform/architecture-overview.md` |
|
||||
| CI Recipes | *(pipeline templates)* | `docs/modules/ci/architecture.md` |
|
||||
| Zastava | `src/Zastava/StellaOps.Zastava.Observer`<br>`src/Zastava/StellaOps.Zastava.Webhook`<br>`src/Zastava/StellaOps.Zastava.Core` | `docs/modules/zastava/architecture.md` |
|
||||
|
||||
## 4.1 Module cheat sheet
|
||||
|
||||
### Authority
|
||||
- **Path:** `src/Authority/StellaOps.Authority`, plugins in `src/Authority/StellaOps.Authority.Plugin.*`.
|
||||
- **Docs:** `docs/modules/authority/architecture.md`.
|
||||
- **Responsibilities:** Issues short-lived, sender-constrained OpToks (DPoP/mTLS) for services, CLI, and UI; exposes OIDC discovery, device-code, and auth-code flows.
|
||||
- **Key traits:** Ed25519/ES256 signing with JWKS rotation, tenant-aware scopes, stateless JWT validation, optional introspection, and structured audit trails.
|
||||
|
||||
### Signer
|
||||
- **Path:** `src/Signer/StellaOps.Signer`.
|
||||
- **Docs:** `docs/modules/signer/architecture.md`.
|
||||
- **Responsibilities:** Authenticates callers, enforces Proof-of-Entitlement, verifies scanner release signatures, and returns DSSE bundles for SBOMs and reports.
|
||||
- **Key traits:** Supports keyless (Fulcio) and keyful (KMS/HSM) signing, applies plan quotas, stores audit trails, and delegates Rekor logging to the Attestor.
|
||||
|
||||
### Attestor
|
||||
- **Path:** `src/Attestor/StellaOps.Attestor`, proof helpers in `src/Attestor/StellaOps.Attestor.Verify`.
|
||||
- **Docs:** `docs/modules/attestor/architecture.md`.
|
||||
- **Responsibilities:** Submits DSSE bundles to Rekor v2, caches `{uuid, index, proof}`, and serves verification bundles to Scanner, UI, CLI, and Export Center.
|
||||
- **Key traits:** mTLS + OpTok enforcement for Signer-only submissions, Mongo/Redis idempotency, optional DSSE archive mirroring, and resilient retry/backoff.
|
||||
|
||||
### Concelier
|
||||
- **Path:** `src/Concelier/StellaOps.Concelier.WebService` with connectors/exporters under `src/Concelier/__Libraries/StellaOps.Concelier.*`.
|
||||
- **Docs:** `docs/modules/concelier/architecture.md`.
|
||||
- **Responsibilities:** Applies the Aggregation-Only Contract to ingest advisories, produce immutable observations, correlate linksets, and publish deterministic exports.
|
||||
- **Key traits:** Restart-time connectors/exporters, Mongo-backed scheduling, canonical JSON/Trivy outputs, Offline Kit parity, and hash-stable manifests.
|
||||
|
||||
### Excititor
|
||||
- **Path:** `src/Excititor/StellaOps.Excititor.WebService`, connectors/adapters in `src/Excititor/__Libraries/StellaOps.Excititor.*`.
|
||||
- **Docs:** `docs/modules/excititor/architecture.md`.
|
||||
- **Responsibilities:** Normalises VEX statements into observations, builds provenance-rich linksets, and surfaces consensus/conflicts for policy suppression.
|
||||
- **Key traits:** Aggregation-only guardrails, restart-time plug-ins, Mongo persistence, deterministic exports, and Offline Kit-ready bundles.
|
||||
|
||||
### Policy Engine
|
||||
- **Path:** `src/Policy/StellaOps.Policy.Engine`, shared libraries under `src/Policy/__Libraries/StellaOps.Policy.*`.
|
||||
- **Docs:** `docs/modules/policy/architecture.md`.
|
||||
- **Responsibilities:** Evaluates `stella-dsl@1` policies, joins SBOM/advisory/VEX evidence, materialises effective findings, and emits explain traces.
|
||||
- **Key traits:** Deterministic evaluation (no wall clock), change-stream driven increments, simulation endpoints, and Authority-scoped tenancy/RBAC enforcement.
|
||||
|
||||
### Scanner.WebService
|
||||
- **Path:** `src/Scanner/StellaOps.Scanner.WebService`.
|
||||
- **Docs:** `docs/modules/scanner/architecture.md`.
|
||||
- **Responsibilities:** Hosts scan/diff/export APIs, enqueues work, serves SBOM and diff artifacts, and publishes DSSE-ready report metadata.
|
||||
- **Key traits:** Minimal APIs with Redis/NATS queue clients, RustFS artifact integration, BOM-index lookups, and DSSE hand-off to Signer/Attestor.
|
||||
|
||||
### Scanner.Worker
|
||||
- **Path:** `src/Scanner/StellaOps.Scanner.Worker` with analyzers/caches in `src/Scanner/__Libraries/StellaOps.Scanner.*`.
|
||||
- **Docs:** `docs/modules/scanner/architecture.md`.
|
||||
- **Responsibilities:** Runs deterministic OS/language/native analyzers per layer, composes inventory and usage SBOM fragments, and streams them back to the catalog.
|
||||
- **Key traits:** Layer/file CAS caching, restart-time analyzer plug-ins under `plugins/scanner/**`, bounded retries with lease renewals, and DSSE-ready outputs.
|
||||
|
||||
### Scheduler
|
||||
- **Path:** `src/Scheduler/StellaOps.Scheduler.WebService`, `src/Scheduler/StellaOps.Scheduler.Worker`.
|
||||
- **Docs:** `docs/modules/scheduler/architecture.md`.
|
||||
- **Responsibilities:** Detects advisory/VEX deltas, selects impacted assets via BOM index, and schedules analysis-only runs toward Scanner and Policy Engine.
|
||||
- **Key traits:** Mongo impact cursors, Redis/NATS orchestration, webhook fan-out (Policy/Notify/Runtime), and deterministic evaluation windows.
|
||||
|
||||
### CLI
|
||||
- **Path:** `src/Cli/StellaOps.Cli`, helpers in `src/Cli/StellaOps.Cli.Core`, plug-ins in `src/Cli/StellaOps.Cli.Plugins.*`.
|
||||
- **Docs:** `docs/modules/cli/architecture.md`.
|
||||
- **Responsibilities:** Provides deterministic verbs for scan/diff/export/report, Buildx SBOM orchestration, policy/VEX administration, and offline kit workflows.
|
||||
- **Key traits:** Native AOT binaries, device-code/client-credential login with DPoP storage, golden-output tests, and restart-time plug-in manifests in `plugins/cli/**`.
|
||||
|
||||
### UI
|
||||
- **Path:** `src/UI/StellaOps.UI`.
|
||||
- **Docs:** `docs/modules/ui/architecture.md`.
|
||||
- **Responsibilities:** Angular SPA for scans, policy authoring, VEX evidence exploration, runtime posture, and admin tooling via backend APIs.
|
||||
- **Key traits:** Angular Signals with `@ngrx/signals`, typed API clients handling DPoP + SSE, Tailwind theming, and immutable content-hashed bundles.
|
||||
|
||||
### Notify
|
||||
- **Path:** `src/Notify/StellaOps.Notify.WebService`, `src/Notify/StellaOps.Notify.Worker`, connectors in `src/Notify/__Libraries`.
|
||||
- **Docs:** `docs/modules/notify/architecture.md`.
|
||||
- **Responsibilities:** Evaluates notification rules on platform events, renders channel-specific payloads, and delivers messages with throttling/digests.
|
||||
- **Key traits:** Tenant-scoped rule engine, idempotent delivery queues, secrets referenced rather than stored, and comprehensive audit/metrics coverage.
|
||||
|
||||
### Export Center
|
||||
- **Path:** `src/ExportCenter/StellaOps.ExportCenter.WebService`, `src/ExportCenter/StellaOps.ExportCenter.Worker`, adapters in `src/ExportCenter/StellaOps.ExportCenter.*`.
|
||||
- **Docs:** `docs/modules/export-center/architecture.md`.
|
||||
- **Responsibilities:** Packages reproducible evidence bundles (JSON, Trivy, mirror) with provenance, signing, and distribution manifests for offline or mirror deployments.
|
||||
- **Key traits:** Profile-driven exports, Orchestrator-backed job leases, Mongo/object storage staging, and cosign-compatible provenance/signature emission.
|
||||
|
||||
### Registry Token Service
|
||||
- **Path:** `src/Registry/StellaOps.Registry.TokenService`, with integration tests in `src/Registry/__Tests/StellaOps.Registry.TokenService.Tests`.
|
||||
- **Docs:** `docs/modules/registry/operations/token-service.md`.
|
||||
- **Responsibilities:** Issues scoped pull tokens for container/image registries, enforces licence/plan constraints, and publishes audit telemetry for token usage.
|
||||
- **Key traits:** Authority-issued OpTok validation, Mongo-backed issuance ledger, deterministic checksum manifests for Offline Kit bundles, and emergency revoke/rotation tooling.
|
||||
|
||||
### Zastava
|
||||
- **Path:** `src/Zastava/StellaOps.Zastava.Observer`, `src/Zastava/StellaOps.Zastava.Webhook`, shared contracts in `src/Zastava/StellaOps.Zastava.Core`.
|
||||
- **Docs:** `docs/modules/zastava/architecture.md`.
|
||||
- **Responsibilities:** Observes running workloads, emits runtime posture events, and enforces admission-time policy (signed images, SBOM availability, policy verdict).
|
||||
- **Key traits:** Authority-issued OpToks with DPoP/mTLS, ND-JSON batching with local buffering, delta-scan triggers on drift, and Kubernetes webhook enforcement.
|
||||
|
||||
---
|
||||
|
||||
### 4.1.4) Glossary (quick)
|
||||
|
||||
- **OVAL** — Vendor/distro security definition format; authoritative for OS packages.
|
||||
- **NEVRA / EVR** — RPM and Debian version semantics for OS packages.
|
||||
- **PURL / SemVer** — Coordinates and version semantics for OSS ecosystems.
|
||||
- **KEV** — Known Exploited Vulnerabilities (flag only).
|
||||
|
||||
---
|
||||
# 5) Your role as StellaOps contributor
|
||||
|
||||
You acting as information technology engineer that will take different type of roles in goal achieving StellaOps production implementation
|
||||
In order you to work - you have to be supplied with directory that contains `AGENTS.md`,`TASKS.md` files. There will you have more information about the role you have, the scope of your work and the tasks you will have.
|
||||
|
||||
Boundaries:
|
||||
- You operate only in the working directories I gave you, unless there is dependencies that makes you to work on dependency in shared directory. Then you ask for confirmation.
|
||||
|
||||
You main characteristics:
|
||||
- Keep endpoints small, deterministic, and cancellation-aware.
|
||||
- Improve logs/metrics as per tasks.
|
||||
- Update `TASKS.md` when moving tasks forward.
|
||||
- When you are done with all task you state explicitly you are done.
|
||||
- Impersonate the role described on working directory `AGENTS.md` you will read, if role is not available - take role of the CTO of the StellaOps in early stages.
|
||||
- You always strive for best practices
|
||||
- You always strive for re-usability
|
||||
- When in doubt of design decision - you ask then act
|
||||
- You are autonomus - meaning that you will work for long time alone and achieve maximum without stopping for stupid questions
|
||||
- You operate on the same directory where other agents will work. In case you need to work on directory that is dependency on provided `AGENTS.md`,`TASKS.md` files you have to ask for confirmation first.
|
||||
|
||||
## 5.1) Type of contributions
|
||||
|
||||
- **BE‑Base (Platform & Pipeline)**
|
||||
Owns DI, plugin host, job scheduler/coordinator, configuration binding, minimal API endpoints, and Mongo bootstrapping.
|
||||
- **BE‑Conn‑X (Connectors)**
|
||||
One agent per source family (NVD, Red Hat, Ubuntu, Debian, SUSE, GHSA, OSV, PSIRTs, CERTs, KEV, ICS). Implements fetch/parse/map with incremental watermarks.
|
||||
- **BE‑Merge (Canonical Merge & Dedupe)**
|
||||
Identity graph, precedence policies, canonical JSON serializer, and deterministic hashing (`merge_event`).
|
||||
- **BE‑Export (JSON & Trivy DB)**
|
||||
Deterministic export trees, Trivy DB packaging, optional ORAS push, and offline bundle.
|
||||
- **QA (Validation & Observability)**
|
||||
Schema tests, fixture goldens, determinism checks, metrics/logs/traces, e2e reproducibility runs.
|
||||
- **DevEx/Docs**
|
||||
Maintains this agent framework, templates, and per‑directory guides; assists parallelization and reviews.
|
||||
|
||||
|
||||
## 5.2) Work rules (important)
|
||||
|
||||
- **Directory ownership**: Each agent works **only inside its module directory**. Cross‑module edits require a brief handshake in issues/PR description.
|
||||
- **Scoping**: Use each module’s `AGENTS.md` and `TASKS.md` to plan; autonomous agents must read `src/AGENTS.md` and the module docs before acting.
|
||||
- **Determinism**: Sort keys, normalize timestamps to UTC ISO‑8601, avoid non‑deterministic data in exports and tests.
|
||||
- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md and in ./SPRINTS.md file.
|
||||
- **Coordination**: In case task is discovered as blocked on other team or task, according TASKS.md files that dependency is on needs to be changed by adding new tasks describing the requirement. the current task must be updated as completed. In case task changes, scope or requirements or rules - other documentations needs be updated accordingly.
|
||||
- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and the relevant TASKS.md file. Always check the AGENTS.md in the relevant TASKS.md directory.
|
||||
- **Tests**: Add/extend fixtures and unit tests per change; never regress determinism or precedence.
|
||||
- **Test layout**: Use module-specific projects in `StellaOps.Concelier.<Component>.Tests`; shared fixtures/harnesses live in `StellaOps.Concelier.Testing`.
|
||||
- **Execution autonomous**: In case you need to continue with more than one options just continue sequentially, unless the continue requires design decision.
|
||||
- **Additional references**: When a task mentions historical epics, consult the corresponding module guides or domain playbooks under `docs/modules/**`, `docs/api/`, `docs/risk/`, or `docs/airgap/` for the latest specification.
|
||||
|
||||
---
|
||||
|
||||
66
README.md
66
README.md
@@ -1,33 +1,33 @@
|
||||
# StellaOps Concelier & CLI
|
||||
|
||||
This repository hosts the StellaOps Concelier service, its plug-in ecosystem, and the
|
||||
first-party CLI (`stellaops-cli`). Concelier ingests vulnerability advisories from
|
||||
authoritative sources, stores them in MongoDB, and exports deterministic JSON and
|
||||
Trivy DB artefacts. The CLI drives scanner distribution, scan execution, and job
|
||||
control against the Concelier API.
|
||||
|
||||
## Quickstart
|
||||
|
||||
1. Prepare a MongoDB instance and (optionally) install `trivy-db`/`oras`.
|
||||
2. Copy `etc/concelier.yaml.sample` to `etc/concelier.yaml` and update the storage + telemetry
|
||||
settings.
|
||||
3. Copy `etc/authority.yaml.sample` to `etc/authority.yaml`, review the issuer, token
|
||||
lifetimes, and plug-in descriptors, then edit the companion manifests under
|
||||
`etc/authority.plugins/*.yaml` to match your deployment.
|
||||
4. Start the web service with `dotnet run --project src/Concelier/StellaOps.Concelier.WebService`.
|
||||
5. Configure the CLI via environment variables (e.g. `STELLAOPS_BACKEND_URL`) and trigger
|
||||
jobs with `dotnet run --project src/Cli/StellaOps.Cli -- db merge`.
|
||||
|
||||
Detailed operator guidance is available in `docs/10_CONCELIER_CLI_QUICKSTART.md`. API and
|
||||
command reference material lives in `docs/09_API_CLI_REFERENCE.md`.
|
||||
|
||||
Pipeline note: deployment workflows should template `etc/concelier.yaml` during CI/CD,
|
||||
injecting environment-specific Mongo credentials and telemetry endpoints. Upcoming
|
||||
releases will add Microsoft OAuth (Entra ID) authentication support—track the quickstart
|
||||
for integration steps once available.
|
||||
|
||||
## Documentation
|
||||
|
||||
- `docs/README.md` now consolidates the platform index and points to the updated high-level architecture.
|
||||
- Module architecture dossiers now live under `docs/modules/<module>/`. The most relevant here are `docs/modules/concelier/ARCHITECTURE.md` (service layout, merge engine, exports) and `docs/modules/cli/ARCHITECTURE.md` (command surface, AOT packaging, auth flows). Related services such as the Signer, Attestor, Authority, Scanner, UI, Excititor, Zastava, and DevOps pipeline each have their own dossier in the same hierarchy.
|
||||
- Offline operation guidance moved to `docs/24_OFFLINE_KIT.md`, which details bundle composition, verification, and delta workflows. Concelier-specific connector operations stay in `docs/modules/concelier/operations/connectors/*.md` with companion runbooks in `docs/modules/concelier/operations/`.
|
||||
# StellaOps Concelier & CLI
|
||||
|
||||
This repository hosts the StellaOps Concelier service, its plug-in ecosystem, and the
|
||||
first-party CLI (`stellaops-cli`). Concelier ingests vulnerability advisories from
|
||||
authoritative sources, stores them in MongoDB, and exports deterministic JSON and
|
||||
Trivy DB artefacts. The CLI drives scanner distribution, scan execution, and job
|
||||
control against the Concelier API.
|
||||
|
||||
## Quickstart
|
||||
|
||||
1. Prepare a MongoDB instance and (optionally) install `trivy-db`/`oras`.
|
||||
2. Copy `etc/concelier.yaml.sample` to `etc/concelier.yaml` and update the storage + telemetry
|
||||
settings.
|
||||
3. Copy `etc/authority.yaml.sample` to `etc/authority.yaml`, review the issuer, token
|
||||
lifetimes, and plug-in descriptors, then edit the companion manifests under
|
||||
`etc/authority.plugins/*.yaml` to match your deployment.
|
||||
4. Start the web service with `dotnet run --project src/Concelier/StellaOps.Concelier.WebService`.
|
||||
5. Configure the CLI via environment variables (e.g. `STELLAOPS_BACKEND_URL`) and trigger
|
||||
jobs with `dotnet run --project src/Cli/StellaOps.Cli -- db merge`.
|
||||
|
||||
Detailed operator guidance is available in `docs/10_CONCELIER_CLI_QUICKSTART.md`. API and
|
||||
command reference material lives in `docs/09_API_CLI_REFERENCE.md`.
|
||||
|
||||
Pipeline note: deployment workflows should template `etc/concelier.yaml` during CI/CD,
|
||||
injecting environment-specific Mongo credentials and telemetry endpoints. Upcoming
|
||||
releases will add Microsoft OAuth (Entra ID) authentication support—track the quickstart
|
||||
for integration steps once available.
|
||||
|
||||
## Documentation
|
||||
|
||||
- `docs/README.md` now consolidates the platform index and points to the updated high-level architecture.
|
||||
- Module architecture dossiers now live under `docs/modules/<module>/`. The most relevant here are `docs/modules/concelier/ARCHITECTURE.md` (service layout, merge engine, exports) and `docs/modules/cli/ARCHITECTURE.md` (command surface, AOT packaging, auth flows). Related services such as the Signer, Attestor, Authority, Scanner, UI, Excititor, Zastava, and DevOps pipeline each have their own dossier in the same hierarchy.
|
||||
- Offline operation guidance moved to `docs/24_OFFLINE_KIT.md`, which details bundle composition, verification, and delta workflows. Concelier-specific connector operations stay in `docs/modules/concelier/operations/connectors/*.md` with companion runbooks in `docs/modules/concelier/operations/`.
|
||||
|
||||
102
deploy/README.md
102
deploy/README.md
@@ -1,51 +1,51 @@
|
||||
# Deployment Profiles
|
||||
|
||||
This directory contains deterministic deployment bundles for the core Stella Ops stack. All manifests reference immutable image digests and map 1:1 to the release manifests stored under `deploy/releases/`.
|
||||
|
||||
## Structure
|
||||
|
||||
- `releases/` – canonical release manifests (edge, stable, airgap) used to source image digests.
|
||||
- `compose/` – Docker Compose bundles for dev/stage/airgap targets plus `.env` seed files.
|
||||
- `compose/docker-compose.mirror.yaml` – managed mirror bundle for `*.stella-ops.org` with gateway cache and multi-tenant auth.
|
||||
- `compose/docker-compose.telemetry.yaml` – optional OpenTelemetry collector overlay (mutual TLS, OTLP pipelines).
|
||||
- `compose/docker-compose.telemetry-storage.yaml` – optional Prometheus/Tempo/Loki stack for observability backends.
|
||||
- `helm/stellaops/` – multi-profile Helm chart with values files for dev/stage/airgap.
|
||||
- `telemetry/` – shared OpenTelemetry collector configuration and certificate artefacts (generated via tooling).
|
||||
- `tools/validate-profiles.sh` – helper that runs `docker compose config` and `helm lint/template` for every profile.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Update or add a release manifest under `releases/` with the new digests.
|
||||
2. Mirror the digests into the Compose and Helm profiles that correspond to that channel.
|
||||
3. Run `deploy/tools/validate-profiles.sh` (requires Docker CLI and Helm) to ensure the bundles lint and template cleanly.
|
||||
4. If telemetry ingest is required for the release, generate development certificates using
|
||||
`./ops/devops/telemetry/generate_dev_tls.sh` and run the collector smoke test with
|
||||
`python ./ops/devops/telemetry/smoke_otel_collector.py` to verify the OTLP endpoints.
|
||||
5. Commit the change alongside any documentation updates (e.g. install guide cross-links).
|
||||
|
||||
Maintaining the digest linkage keeps offline/air-gapped installs reproducible and avoids tag drift between environments.
|
||||
|
||||
### Additional tooling
|
||||
|
||||
- `deploy/tools/check-channel-alignment.py` – verifies that Helm/Compose profiles reference the exact images listed in a release manifest. Run it for each channel before promoting a release.
|
||||
- `ops/devops/telemetry/generate_dev_tls.sh` – produces local CA/server/client certificates for Compose-based collector testing.
|
||||
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traffic and asserts the collector accepted traces, metrics, and logs.
|
||||
- `ops/devops/telemetry/package_offline_bundle.py` – packages telemetry assets (config/Helm/Compose) into a signed tarball for air-gapped installs.
|
||||
- `docs/modules/devops/runbooks/deployment-upgrade.md` – end-to-end instructions for upgrade, rollback, and channel promotion workflows (Helm + Compose).
|
||||
|
||||
## CI smoke checks
|
||||
|
||||
The `.gitea/workflows/build-test-deploy.yml` pipeline includes a `notify-smoke` stage that validates scanner event propagation after staging deployments. Configure the following repository secrets (or environment-level secrets) so the job can connect to Redis and the Notify API:
|
||||
|
||||
- `NOTIFY_SMOKE_REDIS_DSN` – Redis connection string (`redis://user:pass@host:port/db`).
|
||||
- `NOTIFY_SMOKE_NOTIFY_BASEURL` – Base URL for the staging Notify WebService (e.g. `https://notify.stage.stella-ops.internal`).
|
||||
- `NOTIFY_SMOKE_NOTIFY_TOKEN` – OAuth bearer token (service account) with permission to read deliveries.
|
||||
- `NOTIFY_SMOKE_NOTIFY_TENANT` – Tenant identifier used for the smoke validation requests.
|
||||
- *(Optional)* `NOTIFY_SMOKE_NOTIFY_TENANT_HEADER` – Override for the tenant header name (defaults to `X-StellaOps-Tenant`).
|
||||
|
||||
Define the following repository variables (or secrets) to drive the assertions performed by the smoke check:
|
||||
|
||||
- `NOTIFY_SMOKE_EXPECT_KINDS` – Comma-separated event kinds the checker must observe (for example `scanner.report.ready,scanner.scan.completed`).
|
||||
- `NOTIFY_SMOKE_LOOKBACK_MINUTES` – Time window (in minutes) used when scanning the Redis stream for recent events (for example `30`).
|
||||
|
||||
All of the above values are required—the workflow fails fast with a descriptive error if any are missing or empty. Provide the variables at the organisation or repository scope before enabling the smoke stage.
|
||||
# Deployment Profiles
|
||||
|
||||
This directory contains deterministic deployment bundles for the core Stella Ops stack. All manifests reference immutable image digests and map 1:1 to the release manifests stored under `deploy/releases/`.
|
||||
|
||||
## Structure
|
||||
|
||||
- `releases/` – canonical release manifests (edge, stable, airgap) used to source image digests.
|
||||
- `compose/` – Docker Compose bundles for dev/stage/airgap targets plus `.env` seed files.
|
||||
- `compose/docker-compose.mirror.yaml` – managed mirror bundle for `*.stella-ops.org` with gateway cache and multi-tenant auth.
|
||||
- `compose/docker-compose.telemetry.yaml` – optional OpenTelemetry collector overlay (mutual TLS, OTLP pipelines).
|
||||
- `compose/docker-compose.telemetry-storage.yaml` – optional Prometheus/Tempo/Loki stack for observability backends.
|
||||
- `helm/stellaops/` – multi-profile Helm chart with values files for dev/stage/airgap.
|
||||
- `telemetry/` – shared OpenTelemetry collector configuration and certificate artefacts (generated via tooling).
|
||||
- `tools/validate-profiles.sh` – helper that runs `docker compose config` and `helm lint/template` for every profile.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Update or add a release manifest under `releases/` with the new digests.
|
||||
2. Mirror the digests into the Compose and Helm profiles that correspond to that channel.
|
||||
3. Run `deploy/tools/validate-profiles.sh` (requires Docker CLI and Helm) to ensure the bundles lint and template cleanly.
|
||||
4. If telemetry ingest is required for the release, generate development certificates using
|
||||
`./ops/devops/telemetry/generate_dev_tls.sh` and run the collector smoke test with
|
||||
`python ./ops/devops/telemetry/smoke_otel_collector.py` to verify the OTLP endpoints.
|
||||
5. Commit the change alongside any documentation updates (e.g. install guide cross-links).
|
||||
|
||||
Maintaining the digest linkage keeps offline/air-gapped installs reproducible and avoids tag drift between environments.
|
||||
|
||||
### Additional tooling
|
||||
|
||||
- `deploy/tools/check-channel-alignment.py` – verifies that Helm/Compose profiles reference the exact images listed in a release manifest. Run it for each channel before promoting a release.
|
||||
- `ops/devops/telemetry/generate_dev_tls.sh` – produces local CA/server/client certificates for Compose-based collector testing.
|
||||
- `ops/devops/telemetry/smoke_otel_collector.py` – sends OTLP traffic and asserts the collector accepted traces, metrics, and logs.
|
||||
- `ops/devops/telemetry/package_offline_bundle.py` – packages telemetry assets (config/Helm/Compose) into a signed tarball for air-gapped installs.
|
||||
- `docs/modules/devops/runbooks/deployment-upgrade.md` – end-to-end instructions for upgrade, rollback, and channel promotion workflows (Helm + Compose).
|
||||
|
||||
## CI smoke checks
|
||||
|
||||
The `.gitea/workflows/build-test-deploy.yml` pipeline includes a `notify-smoke` stage that validates scanner event propagation after staging deployments. Configure the following repository secrets (or environment-level secrets) so the job can connect to Redis and the Notify API:
|
||||
|
||||
- `NOTIFY_SMOKE_REDIS_DSN` – Redis connection string (`redis://user:pass@host:port/db`).
|
||||
- `NOTIFY_SMOKE_NOTIFY_BASEURL` – Base URL for the staging Notify WebService (e.g. `https://notify.stage.stella-ops.internal`).
|
||||
- `NOTIFY_SMOKE_NOTIFY_TOKEN` – OAuth bearer token (service account) with permission to read deliveries.
|
||||
- `NOTIFY_SMOKE_NOTIFY_TENANT` – Tenant identifier used for the smoke validation requests.
|
||||
- *(Optional)* `NOTIFY_SMOKE_NOTIFY_TENANT_HEADER` – Override for the tenant header name (defaults to `X-StellaOps-Tenant`).
|
||||
|
||||
Define the following repository variables (or secrets) to drive the assertions performed by the smoke check:
|
||||
|
||||
- `NOTIFY_SMOKE_EXPECT_KINDS` – Comma-separated event kinds the checker must observe (for example `scanner.report.ready,scanner.scan.completed`).
|
||||
- `NOTIFY_SMOKE_LOOKBACK_MINUTES` – Time window (in minutes) used when scanning the Redis stream for recent events (for example `30`).
|
||||
|
||||
All of the above values are required—the workflow fails fast with a descriptive error if any are missing or empty. Provide the variables at the organisation or repository scope before enabling the smoke stage.
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
# Telemetry Storage Stack
|
||||
|
||||
Configuration snippets for the default StellaOps observability backends used in
|
||||
staging and production environments. The stack comprises:
|
||||
|
||||
- **Prometheus** for metrics (scraping the collector's Prometheus exporter)
|
||||
- **Tempo** for traces (OTLP ingest via mTLS)
|
||||
- **Loki** for logs (HTTP ingest with tenant isolation)
|
||||
|
||||
## Files
|
||||
|
||||
| Path | Description |
|
||||
| ---- | ----------- |
|
||||
| `prometheus.yaml` | Scrape configuration for the collector (mTLS + bearer token placeholder). |
|
||||
| `tempo.yaml` | Tempo configuration with multitenancy enabled and local storage paths. |
|
||||
| `loki.yaml` | Loki configuration enabling per-tenant overrides and boltdb-shipper storage. |
|
||||
# Telemetry Storage Stack
|
||||
|
||||
Configuration snippets for the default StellaOps observability backends used in
|
||||
staging and production environments. The stack comprises:
|
||||
|
||||
- **Prometheus** for metrics (scraping the collector's Prometheus exporter)
|
||||
- **Tempo** for traces (OTLP ingest via mTLS)
|
||||
- **Loki** for logs (HTTP ingest with tenant isolation)
|
||||
|
||||
## Files
|
||||
|
||||
| Path | Description |
|
||||
| ---- | ----------- |
|
||||
| `prometheus.yaml` | Scrape configuration for the collector (mTLS + bearer token placeholder). |
|
||||
| `tempo.yaml` | Tempo configuration with multitenancy enabled and local storage paths. |
|
||||
| `loki.yaml` | Loki configuration enabling per-tenant overrides and boltdb-shipper storage. |
|
||||
| `tenants/tempo-overrides.yaml` | Example tenant overrides for Tempo (retention, limits). |
|
||||
| `tenants/loki-overrides.yaml` | Example tenant overrides for Loki (rate limits, retention). |
|
||||
| `auth/` | Placeholder directory for Prometheus bearer token files (e.g., `token`). |
|
||||
@@ -32,5 +32,5 @@ ensure TLS, multitenancy, and override references remain intact.
|
||||
- Both Tempo and Loki require mutual TLS.
|
||||
- Prometheus uses mTLS plus a bearer token that should be minted by Authority.
|
||||
- Update the overrides files to enforce per-tenant retention/ingestion limits.
|
||||
|
||||
For comprehensive deployment steps see `docs/modules/telemetry/operations/storage.md`.
|
||||
|
||||
For comprehensive deployment steps see `docs/modules/telemetry/operations/storage.md`.
|
||||
|
||||
@@ -1,354 +1,354 @@
|
||||
# 10 · Concelier + CLI Quickstart
|
||||
|
||||
This guide walks through configuring the Concelier web service and the `stellaops-cli`
|
||||
tool so an operator can ingest advisories, merge them, and publish exports from a
|
||||
single workstation. It focuses on deployment-facing surfaces only (configuration,
|
||||
runtime wiring, CLI usage) and leaves connector/internal customization for later.
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
- .NET SDK **10.0.100-preview** (matches `global.json`)
|
||||
- MongoDB instance reachable from the host (local Docker or managed)
|
||||
- `trivy-db` binary on `PATH` for Trivy exports (and `oras` if publishing to OCI)
|
||||
- Plugin assemblies present in `StellaOps.Concelier.PluginBinaries/` (already included in the repo)
|
||||
- Optional: Docker/Podman runtime if you plan to run scanners locally
|
||||
|
||||
> **Tip** – air-gapped installs should preload `trivy-db` and `oras` binaries into the
|
||||
> runner image since Concelier never fetches them dynamically.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Configure Concelier
|
||||
|
||||
1. Copy the sample config to the expected location (CI/CD pipelines can stamp values
|
||||
into this file during deployment—see the “Deployment automation” note below):
|
||||
|
||||
```bash
|
||||
mkdir -p etc
|
||||
cp etc/concelier.yaml.sample etc/concelier.yaml
|
||||
```
|
||||
|
||||
2. Edit `etc/concelier.yaml` and update the MongoDB DSN (and optional database name).
|
||||
The default template configures plug-in discovery to look in `StellaOps.Concelier.PluginBinaries/`
|
||||
and disables remote telemetry exporters by default.
|
||||
|
||||
3. (Optional) Override settings via environment variables. All keys are prefixed with
|
||||
`CONCELIER_`. Example:
|
||||
|
||||
```bash
|
||||
export CONCELIER_STORAGE__DSN="mongodb://user:pass@mongo:27017/concelier"
|
||||
export CONCELIER_TELEMETRY__ENABLETRACING=false
|
||||
```
|
||||
|
||||
4. Start the web service from the repository root:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Concelier/StellaOps.Concelier.WebService
|
||||
```
|
||||
|
||||
On startup Concelier validates the options, boots MongoDB indexes, loads plug-ins,
|
||||
and exposes:
|
||||
|
||||
- `GET /health` – returns service status and telemetry settings
|
||||
- `GET /ready` – performs a MongoDB `ping`
|
||||
- `GET /jobs` + `POST /jobs/{kind}` – inspect and trigger connector/export jobs
|
||||
|
||||
> **Security note** – authentication now ships via StellaOps Authority. Keep
|
||||
> `authority.allowAnonymousFallback: true` only during the staged rollout and
|
||||
> disable it before **2025-12-31 UTC** so tokens become mandatory.
|
||||
|
||||
Rollout checkpoints for the two Authority toggles:
|
||||
|
||||
| Phase | `authority.enabled` | `authority.allowAnonymousFallback` | Goal | Observability focus |
|
||||
| ----- | ------------------- | ---------------------------------- | ---- | ------------------- |
|
||||
| **Validation (staging)** | `true` | `true` | Verify token issuance, CLI scopes, and audit log noise without breaking cron jobs. | Watch `Concelier.Authorization.Audit` for `bypass=True` events and scope gaps; confirm CLI `auth status` succeeds. |
|
||||
| **Cutover rehearsal** | `true` | `false` | Exercise production-style enforcement before the deadline; ensure only approved maintenance ranges remain in `bypassNetworks`. | Expect some HTTP 401s; verify `web.jobs.triggered` metrics flatten for unauthenticated calls and audit logs highlight missing tokens. |
|
||||
| **Enforced (steady state)** | `true` | `false` | Production baseline after the 2025-12-31 UTC cutoff. | Alert on new `bypass=True` entries and on repeated 401 bursts; correlate with Authority availability dashboards. |
|
||||
|
||||
### Authority companion configuration (preview)
|
||||
|
||||
1. Copy the Authority sample configuration:
|
||||
|
||||
```bash
|
||||
cp etc/authority.yaml.sample etc/authority.yaml
|
||||
```
|
||||
|
||||
2. Update the issuer URL, token lifetimes, and plug-in descriptors to match your
|
||||
environment. Authority expects per-plugin manifests in `etc/authority.plugins/`;
|
||||
sample `standard.yaml` and `ldap.yaml` files are provided as starting points.
|
||||
For air-gapped installs keep the default plug-in binary directory
|
||||
(`../StellaOps.Authority.PluginBinaries`) so packaged plug-ins load without outbound access.
|
||||
|
||||
3. Environment variables prefixed with `STELLAOPS_AUTHORITY_` override individual
|
||||
fields. Example:
|
||||
|
||||
```bash
|
||||
export STELLAOPS_AUTHORITY__ISSUER="https://authority.stella-ops.local"
|
||||
export STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0="/srv/authority/plugins"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Configure the CLI
|
||||
|
||||
The CLI reads configuration from JSON/YAML files *and* environment variables. The
|
||||
defaults live in `src/Cli/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
|
||||
|
||||
| Setting | Environment variable | Default | Purpose |
|
||||
| ------- | -------------------- | ------- | ------- |
|
||||
| `BackendUrl` | `STELLAOPS_BACKEND_URL` | _empty_ | Base URL of the Concelier web service |
|
||||
| `ApiKey` | `API_KEY` | _empty_ | Reserved for legacy key auth; leave empty when using Authority |
|
||||
| `ScannerCacheDirectory` | `STELLAOPS_SCANNER_CACHE_DIRECTORY` | `scanners` | Local cache folder |
|
||||
| `ResultsDirectory` | `STELLAOPS_RESULTS_DIRECTORY` | `results` | Where scan outputs are written |
|
||||
| `Authority.Url` | `STELLAOPS_AUTHORITY_URL` | _empty_ | StellaOps Authority issuer/token endpoint |
|
||||
| `Authority.ClientId` | `STELLAOPS_AUTHORITY_CLIENT_ID` | _empty_ | Client identifier for the CLI |
|
||||
| `Authority.ClientSecret` | `STELLAOPS_AUTHORITY_CLIENT_SECRET` | _empty_ | Client secret (omit when using username/password grant) |
|
||||
| `Authority.Username` | `STELLAOPS_AUTHORITY_USERNAME` | _empty_ | Username for password grant flows |
|
||||
| `Authority.Password` | `STELLAOPS_AUTHORITY_PASSWORD` | _empty_ | Password for password grant flows |
|
||||
| `Authority.Scope` | `STELLAOPS_AUTHORITY_SCOPE` | `concelier.jobs.trigger advisory:ingest` | Space-separated OAuth scopes requested for backend operations |
|
||||
| `Authority.TokenCacheDirectory` | `STELLAOPS_AUTHORITY_TOKEN_CACHE_DIR` | `~/.stellaops/tokens` | Directory that persists cached tokens |
|
||||
| `Authority.Resilience.EnableRetries` | `STELLAOPS_AUTHORITY_ENABLE_RETRIES` | `true` | Toggle Polly retry handler for Authority HTTP calls |
|
||||
| `Authority.Resilience.RetryDelays` | `STELLAOPS_AUTHORITY_RETRY_DELAYS` | `1s,2s,5s` | Comma- or space-separated backoff delays (hh:mm:ss) |
|
||||
| `Authority.Resilience.AllowOfflineCacheFallback` | `STELLAOPS_AUTHORITY_ALLOW_OFFLINE_CACHE_FALLBACK` | `true` | Allow CLI to reuse cached discovery/JWKS metadata when Authority is offline |
|
||||
| `Authority.Resilience.OfflineCacheTolerance` | `STELLAOPS_AUTHORITY_OFFLINE_CACHE_TOLERANCE` | `00:10:00` | Additional tolerance window applied to cached metadata |
|
||||
|
||||
Example bootstrap:
|
||||
|
||||
```bash
|
||||
export STELLAOPS_BACKEND_URL="http://localhost:5000"
|
||||
export STELLAOPS_RESULTS_DIRECTORY="$HOME/.stellaops/results"
|
||||
export STELLAOPS_AUTHORITY_URL="https://authority.local"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_ID="concelier-cli"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_SECRET="s3cr3t"
|
||||
export STELLAOPS_AUTHORITY_SCOPE="concelier.jobs.trigger advisory:ingest advisory:read"
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
|
||||
# Acquire a bearer token and confirm cache state
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth login
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth status
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth whoami
|
||||
```
|
||||
|
||||
Refer to `docs/dev/32_AUTH_CLIENT_GUIDE.md` for deeper guidance on tuning retry/offline settings and rollout checklists.
|
||||
|
||||
To persist configuration, you can create `stellaops-cli.yaml` next to the binary or
|
||||
rely on environment variables for ephemeral runners.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Operating Workflow
|
||||
|
||||
1. **Trigger connector fetch stages**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage fetch
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage parse
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage map
|
||||
```
|
||||
|
||||
Use `--mode resume` when continuing from a previous window:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
|
||||
```
|
||||
|
||||
2. **Merge canonical advisories**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
```
|
||||
|
||||
3. **Produce exports**
|
||||
|
||||
```bash
|
||||
# JSON tree (vuln-list style)
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format json
|
||||
|
||||
# Trivy DB (delta example)
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format trivy-db --delta
|
||||
```
|
||||
|
||||
Concelier always produces a deterministic OCI layout. The first run after a clean
|
||||
bootstrap emits a **full** baseline; subsequent `--delta` runs reuse the previous
|
||||
baseline’s blobs when only JSON manifests change. If the exporter detects that a
|
||||
prior delta is still active (i.e., `LastDeltaDigest` is recorded) it automatically
|
||||
upgrades the next run to a full export and resets the baseline so operators never
|
||||
chain deltas indefinitely. The CLI exposes `--publish-full/--publish-delta` (for
|
||||
ORAS pushes) and `--include-full/--include-delta` (for offline bundles) should you
|
||||
need to override the defaults interactively.
|
||||
|
||||
**Smoke-check delta reuse:** after the first baseline completes, run the export a
|
||||
second time with `--delta` and verify that the new directory reports `mode=delta`
|
||||
while reusing the previous layer blob.
|
||||
|
||||
```bash
|
||||
export_root=${CONCELIER_EXPORT_ROOT:-exports/trivy}
|
||||
base=$(ls -1d "$export_root"/* | sort | tail -n2 | head -n1)
|
||||
delta=$(ls -1d "$export_root"/* | sort | tail -n1)
|
||||
|
||||
jq -r '.mode,.baseExportId' "$delta/metadata.json"
|
||||
|
||||
base_manifest=$(jq -r '.manifests[0].digest' "$base/index.json")
|
||||
delta_manifest=$(jq -r '.manifests[0].digest' "$delta/index.json")
|
||||
printf 'baseline manifest: %s\ndelta manifest: %s\n' "$base_manifest" "$delta_manifest"
|
||||
|
||||
layer_digest=$(jq -r '.layers[0].digest' "$base/blobs/sha256/${base_manifest#sha256:}")
|
||||
cmp "$base/blobs/sha256/${layer_digest#sha256:}" \
|
||||
"$delta/blobs/sha256/${layer_digest#sha256:}"
|
||||
```
|
||||
|
||||
`cmp` returning exit code `0` confirms the delta export reuses the baseline’s
|
||||
`db.tar.gz` layer instead of rebuilding it.
|
||||
|
||||
4. **Verify guard compliance**
|
||||
|
||||
```bash
|
||||
export STELLA_TENANT="${STELLA_TENANT:-tenant-a}"
|
||||
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 24h \
|
||||
--format table \
|
||||
--tenant "$STELLA_TENANT"
|
||||
|
||||
# Optional: capture JSON evidence for pipelines/audits
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 7d \
|
||||
--limit 100 \
|
||||
--format json \
|
||||
--export artifacts/aoc-verify.json \
|
||||
--tenant "$STELLA_TENANT"
|
||||
```
|
||||
|
||||
The CLI exits with `0` when no violations are detected. Guard failures map
|
||||
to `ERR_AOC_00x` codes (`11…17`), while truncated results return `18`. Use
|
||||
`--sources`/`--codes` to focus on noisy connectors and feed the exported JSON
|
||||
into dashboards or evidence lockers for compliance reviews.
|
||||
|
||||
5. **Pre-flight individual payloads**
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-9999.json \
|
||||
--tenant "$STELLA_TENANT" \
|
||||
--format json \
|
||||
--output artifacts/redhat-dry-run.json
|
||||
```
|
||||
|
||||
Exit code `0` confirms the candidate document is AOC compliant. Any guard
|
||||
violation is emitted as deterministic `ERR_AOC_00x` exit codes (`11…17`);
|
||||
reuse the exported JSON in PRs or incident timelines to show offending paths.
|
||||
|
||||
6. **Manage scanners (optional)**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scanner download --channel stable
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan upload --file results/scan-001.json
|
||||
```
|
||||
|
||||
Add `--verbose` to any command for structured console logs. All commands honour
|
||||
`Ctrl+C` cancellation and exit with non-zero status codes when the backend returns
|
||||
a problem document.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Verification Checklist
|
||||
|
||||
- Concelier `/health` returns `"status":"healthy"` and Storage bootstrap is marked
|
||||
complete after startup.
|
||||
- CLI commands return HTTP 202 with a `Location` header (job tracking URL) when
|
||||
triggering Concelier jobs.
|
||||
- Export artefacts are materialised under the configured output directories and
|
||||
their manifests record digests.
|
||||
- MongoDB contains the expected `document`, `dto`, `advisory`, and `export_state`
|
||||
collections after a run.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Deployment Automation
|
||||
|
||||
- Treat `etc/concelier.yaml.sample` as the canonical template. CI/CD should copy it to
|
||||
the deployment artifact and replace placeholders (DSN, telemetry endpoints, cron
|
||||
overrides) with environment-specific secrets.
|
||||
- Keep secret material (Mongo credentials, OTLP tokens) outside of the repository;
|
||||
inject them via secret stores or pipeline variables at stamp time.
|
||||
- When building container images, include `trivy-db` (and `oras` if used) so air-gapped
|
||||
clusters do not need outbound downloads at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Next Steps
|
||||
|
||||
- Enable authority-backed authentication in non-production first. Set
|
||||
`authority.enabled: true` while keeping `authority.allowAnonymousFallback: true`
|
||||
to observe logs, then flip it to `false` before 2025-12-31 UTC to enforce tokens.
|
||||
- Automate the workflow above via CI/CD (compose stack or Kubernetes CronJobs).
|
||||
- Pair with the Concelier connector teams when enabling additional sources so their
|
||||
module-specific requirements are pulled in safely.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Authority Integration
|
||||
|
||||
- Concelier now authenticates callers through StellaOps Authority using OAuth 2.0
|
||||
resource server flows. Populate the `authority` block in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
authority:
|
||||
enabled: true
|
||||
allowAnonymousFallback: false # keep true only during the staged rollout window
|
||||
issuer: "https://authority.example.org"
|
||||
audiences:
|
||||
- "api://concelier"
|
||||
requiredScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
requiredTenants:
|
||||
- "tenant-default"
|
||||
clientId: "concelier-jobs"
|
||||
clientSecretFile: "../secrets/concelier-jobs.secret"
|
||||
clientScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
bypassNetworks:
|
||||
- "127.0.0.1/32"
|
||||
- "::1/128"
|
||||
```
|
||||
|
||||
- Store the client secret outside of source control. Either provide it via
|
||||
`authority.clientSecret` (environment variable `CONCELIER_AUTHORITY__CLIENTSECRET`)
|
||||
or point `authority.clientSecretFile` to a file mounted at runtime.
|
||||
- Cron jobs running on the same host can keep using the API thanks to the loopback
|
||||
bypass mask. Add additional CIDR ranges as needed; every bypass is logged.
|
||||
- Export the same configuration to Kubernetes or systemd by setting environment
|
||||
variables such as:
|
||||
|
||||
```bash
|
||||
export CONCELIER_AUTHORITY__ENABLED=true
|
||||
export CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=false
|
||||
export CONCELIER_AUTHORITY__ISSUER="https://authority.example.org"
|
||||
export CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
export CONCELIER_AUTHORITY__CLIENTSECRETFILE="/var/run/secrets/concelier/authority-client"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
export CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
```
|
||||
|
||||
- CLI commands already pass `Authorization` headers when credentials are supplied.
|
||||
Configure the CLI with matching Authority settings (`docs/09_API_CLI_REFERENCE.md`)
|
||||
so that automation can obtain tokens with the same client credentials. Concelier
|
||||
logs every job request with the client ID, subject (if present), scopes, and
|
||||
a `bypass` flag so operators can audit cron traffic.
|
||||
- **Rollout checklist.**
|
||||
1. Stage the integration with fallback enabled (`allowAnonymousFallback=true`) and confirm CLI/token issuance using `stella auth status`.
|
||||
2. Follow the rehearsal pattern (`allowAnonymousFallback=false`) while monitoring `Concelier.Authorization.Audit` and `web.jobs.triggered`/`web.jobs.trigger.failed` metrics.
|
||||
3. Lock in enforcement, review the audit runbook (`docs/modules/concelier/operations/authority-audit-runbook.md`), and document the bypass CIDR approvals in your change log.
|
||||
# 10 · Concelier + CLI Quickstart
|
||||
|
||||
This guide walks through configuring the Concelier web service and the `stellaops-cli`
|
||||
tool so an operator can ingest advisories, merge them, and publish exports from a
|
||||
single workstation. It focuses on deployment-facing surfaces only (configuration,
|
||||
runtime wiring, CLI usage) and leaves connector/internal customization for later.
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
- .NET SDK **10.0.100-preview** (matches `global.json`)
|
||||
- MongoDB instance reachable from the host (local Docker or managed)
|
||||
- `trivy-db` binary on `PATH` for Trivy exports (and `oras` if publishing to OCI)
|
||||
- Plugin assemblies present in `StellaOps.Concelier.PluginBinaries/` (already included in the repo)
|
||||
- Optional: Docker/Podman runtime if you plan to run scanners locally
|
||||
|
||||
> **Tip** – air-gapped installs should preload `trivy-db` and `oras` binaries into the
|
||||
> runner image since Concelier never fetches them dynamically.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Configure Concelier
|
||||
|
||||
1. Copy the sample config to the expected location (CI/CD pipelines can stamp values
|
||||
into this file during deployment—see the “Deployment automation” note below):
|
||||
|
||||
```bash
|
||||
mkdir -p etc
|
||||
cp etc/concelier.yaml.sample etc/concelier.yaml
|
||||
```
|
||||
|
||||
2. Edit `etc/concelier.yaml` and update the MongoDB DSN (and optional database name).
|
||||
The default template configures plug-in discovery to look in `StellaOps.Concelier.PluginBinaries/`
|
||||
and disables remote telemetry exporters by default.
|
||||
|
||||
3. (Optional) Override settings via environment variables. All keys are prefixed with
|
||||
`CONCELIER_`. Example:
|
||||
|
||||
```bash
|
||||
export CONCELIER_STORAGE__DSN="mongodb://user:pass@mongo:27017/concelier"
|
||||
export CONCELIER_TELEMETRY__ENABLETRACING=false
|
||||
```
|
||||
|
||||
4. Start the web service from the repository root:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Concelier/StellaOps.Concelier.WebService
|
||||
```
|
||||
|
||||
On startup Concelier validates the options, boots MongoDB indexes, loads plug-ins,
|
||||
and exposes:
|
||||
|
||||
- `GET /health` – returns service status and telemetry settings
|
||||
- `GET /ready` – performs a MongoDB `ping`
|
||||
- `GET /jobs` + `POST /jobs/{kind}` – inspect and trigger connector/export jobs
|
||||
|
||||
> **Security note** – authentication now ships via StellaOps Authority. Keep
|
||||
> `authority.allowAnonymousFallback: true` only during the staged rollout and
|
||||
> disable it before **2025-12-31 UTC** so tokens become mandatory.
|
||||
|
||||
Rollout checkpoints for the two Authority toggles:
|
||||
|
||||
| Phase | `authority.enabled` | `authority.allowAnonymousFallback` | Goal | Observability focus |
|
||||
| ----- | ------------------- | ---------------------------------- | ---- | ------------------- |
|
||||
| **Validation (staging)** | `true` | `true` | Verify token issuance, CLI scopes, and audit log noise without breaking cron jobs. | Watch `Concelier.Authorization.Audit` for `bypass=True` events and scope gaps; confirm CLI `auth status` succeeds. |
|
||||
| **Cutover rehearsal** | `true` | `false` | Exercise production-style enforcement before the deadline; ensure only approved maintenance ranges remain in `bypassNetworks`. | Expect some HTTP 401s; verify `web.jobs.triggered` metrics flatten for unauthenticated calls and audit logs highlight missing tokens. |
|
||||
| **Enforced (steady state)** | `true` | `false` | Production baseline after the 2025-12-31 UTC cutoff. | Alert on new `bypass=True` entries and on repeated 401 bursts; correlate with Authority availability dashboards. |
|
||||
|
||||
### Authority companion configuration (preview)
|
||||
|
||||
1. Copy the Authority sample configuration:
|
||||
|
||||
```bash
|
||||
cp etc/authority.yaml.sample etc/authority.yaml
|
||||
```
|
||||
|
||||
2. Update the issuer URL, token lifetimes, and plug-in descriptors to match your
|
||||
environment. Authority expects per-plugin manifests in `etc/authority.plugins/`;
|
||||
sample `standard.yaml` and `ldap.yaml` files are provided as starting points.
|
||||
For air-gapped installs keep the default plug-in binary directory
|
||||
(`../StellaOps.Authority.PluginBinaries`) so packaged plug-ins load without outbound access.
|
||||
|
||||
3. Environment variables prefixed with `STELLAOPS_AUTHORITY_` override individual
|
||||
fields. Example:
|
||||
|
||||
```bash
|
||||
export STELLAOPS_AUTHORITY__ISSUER="https://authority.stella-ops.local"
|
||||
export STELLAOPS_AUTHORITY__PLUGINDIRECTORIES__0="/srv/authority/plugins"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Configure the CLI
|
||||
|
||||
The CLI reads configuration from JSON/YAML files *and* environment variables. The
|
||||
defaults live in `src/Cli/StellaOps.Cli/appsettings.json` and expect overrides at runtime.
|
||||
|
||||
| Setting | Environment variable | Default | Purpose |
|
||||
| ------- | -------------------- | ------- | ------- |
|
||||
| `BackendUrl` | `STELLAOPS_BACKEND_URL` | _empty_ | Base URL of the Concelier web service |
|
||||
| `ApiKey` | `API_KEY` | _empty_ | Reserved for legacy key auth; leave empty when using Authority |
|
||||
| `ScannerCacheDirectory` | `STELLAOPS_SCANNER_CACHE_DIRECTORY` | `scanners` | Local cache folder |
|
||||
| `ResultsDirectory` | `STELLAOPS_RESULTS_DIRECTORY` | `results` | Where scan outputs are written |
|
||||
| `Authority.Url` | `STELLAOPS_AUTHORITY_URL` | _empty_ | StellaOps Authority issuer/token endpoint |
|
||||
| `Authority.ClientId` | `STELLAOPS_AUTHORITY_CLIENT_ID` | _empty_ | Client identifier for the CLI |
|
||||
| `Authority.ClientSecret` | `STELLAOPS_AUTHORITY_CLIENT_SECRET` | _empty_ | Client secret (omit when using username/password grant) |
|
||||
| `Authority.Username` | `STELLAOPS_AUTHORITY_USERNAME` | _empty_ | Username for password grant flows |
|
||||
| `Authority.Password` | `STELLAOPS_AUTHORITY_PASSWORD` | _empty_ | Password for password grant flows |
|
||||
| `Authority.Scope` | `STELLAOPS_AUTHORITY_SCOPE` | `concelier.jobs.trigger advisory:ingest` | Space-separated OAuth scopes requested for backend operations |
|
||||
| `Authority.TokenCacheDirectory` | `STELLAOPS_AUTHORITY_TOKEN_CACHE_DIR` | `~/.stellaops/tokens` | Directory that persists cached tokens |
|
||||
| `Authority.Resilience.EnableRetries` | `STELLAOPS_AUTHORITY_ENABLE_RETRIES` | `true` | Toggle Polly retry handler for Authority HTTP calls |
|
||||
| `Authority.Resilience.RetryDelays` | `STELLAOPS_AUTHORITY_RETRY_DELAYS` | `1s,2s,5s` | Comma- or space-separated backoff delays (hh:mm:ss) |
|
||||
| `Authority.Resilience.AllowOfflineCacheFallback` | `STELLAOPS_AUTHORITY_ALLOW_OFFLINE_CACHE_FALLBACK` | `true` | Allow CLI to reuse cached discovery/JWKS metadata when Authority is offline |
|
||||
| `Authority.Resilience.OfflineCacheTolerance` | `STELLAOPS_AUTHORITY_OFFLINE_CACHE_TOLERANCE` | `00:10:00` | Additional tolerance window applied to cached metadata |
|
||||
|
||||
Example bootstrap:
|
||||
|
||||
```bash
|
||||
export STELLAOPS_BACKEND_URL="http://localhost:5000"
|
||||
export STELLAOPS_RESULTS_DIRECTORY="$HOME/.stellaops/results"
|
||||
export STELLAOPS_AUTHORITY_URL="https://authority.local"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_ID="concelier-cli"
|
||||
export STELLAOPS_AUTHORITY_CLIENT_SECRET="s3cr3t"
|
||||
export STELLAOPS_AUTHORITY_SCOPE="concelier.jobs.trigger advisory:ingest advisory:read"
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
|
||||
# Acquire a bearer token and confirm cache state
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth login
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth status
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- auth whoami
|
||||
```
|
||||
|
||||
Refer to `docs/dev/32_AUTH_CLIENT_GUIDE.md` for deeper guidance on tuning retry/offline settings and rollout checklists.
|
||||
|
||||
To persist configuration, you can create `stellaops-cli.yaml` next to the binary or
|
||||
rely on environment variables for ephemeral runners.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Operating Workflow
|
||||
|
||||
1. **Trigger connector fetch stages**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage fetch
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage parse
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source osv --stage map
|
||||
```
|
||||
|
||||
Use `--mode resume` when continuing from a previous window:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db fetch --source redhat --stage fetch --mode resume
|
||||
```
|
||||
|
||||
2. **Merge canonical advisories**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db merge
|
||||
```
|
||||
|
||||
3. **Produce exports**
|
||||
|
||||
```bash
|
||||
# JSON tree (vuln-list style)
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format json
|
||||
|
||||
# Trivy DB (delta example)
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- db export --format trivy-db --delta
|
||||
```
|
||||
|
||||
Concelier always produces a deterministic OCI layout. The first run after a clean
|
||||
bootstrap emits a **full** baseline; subsequent `--delta` runs reuse the previous
|
||||
baseline’s blobs when only JSON manifests change. If the exporter detects that a
|
||||
prior delta is still active (i.e., `LastDeltaDigest` is recorded) it automatically
|
||||
upgrades the next run to a full export and resets the baseline so operators never
|
||||
chain deltas indefinitely. The CLI exposes `--publish-full/--publish-delta` (for
|
||||
ORAS pushes) and `--include-full/--include-delta` (for offline bundles) should you
|
||||
need to override the defaults interactively.
|
||||
|
||||
**Smoke-check delta reuse:** after the first baseline completes, run the export a
|
||||
second time with `--delta` and verify that the new directory reports `mode=delta`
|
||||
while reusing the previous layer blob.
|
||||
|
||||
```bash
|
||||
export_root=${CONCELIER_EXPORT_ROOT:-exports/trivy}
|
||||
base=$(ls -1d "$export_root"/* | sort | tail -n2 | head -n1)
|
||||
delta=$(ls -1d "$export_root"/* | sort | tail -n1)
|
||||
|
||||
jq -r '.mode,.baseExportId' "$delta/metadata.json"
|
||||
|
||||
base_manifest=$(jq -r '.manifests[0].digest' "$base/index.json")
|
||||
delta_manifest=$(jq -r '.manifests[0].digest' "$delta/index.json")
|
||||
printf 'baseline manifest: %s\ndelta manifest: %s\n' "$base_manifest" "$delta_manifest"
|
||||
|
||||
layer_digest=$(jq -r '.layers[0].digest' "$base/blobs/sha256/${base_manifest#sha256:}")
|
||||
cmp "$base/blobs/sha256/${layer_digest#sha256:}" \
|
||||
"$delta/blobs/sha256/${layer_digest#sha256:}"
|
||||
```
|
||||
|
||||
`cmp` returning exit code `0` confirms the delta export reuses the baseline’s
|
||||
`db.tar.gz` layer instead of rebuilding it.
|
||||
|
||||
4. **Verify guard compliance**
|
||||
|
||||
```bash
|
||||
export STELLA_TENANT="${STELLA_TENANT:-tenant-a}"
|
||||
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 24h \
|
||||
--format table \
|
||||
--tenant "$STELLA_TENANT"
|
||||
|
||||
# Optional: capture JSON evidence for pipelines/audits
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- aoc verify \
|
||||
--since 7d \
|
||||
--limit 100 \
|
||||
--format json \
|
||||
--export artifacts/aoc-verify.json \
|
||||
--tenant "$STELLA_TENANT"
|
||||
```
|
||||
|
||||
The CLI exits with `0` when no violations are detected. Guard failures map
|
||||
to `ERR_AOC_00x` codes (`11…17`), while truncated results return `18`. Use
|
||||
`--sources`/`--codes` to focus on noisy connectors and feed the exported JSON
|
||||
into dashboards or evidence lockers for compliance reviews.
|
||||
|
||||
5. **Pre-flight individual payloads**
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-9999.json \
|
||||
--tenant "$STELLA_TENANT" \
|
||||
--format json \
|
||||
--output artifacts/redhat-dry-run.json
|
||||
```
|
||||
|
||||
Exit code `0` confirms the candidate document is AOC compliant. Any guard
|
||||
violation is emitted as deterministic `ERR_AOC_00x` exit codes (`11…17`);
|
||||
reuse the exported JSON in PRs or incident timelines to show offending paths.
|
||||
|
||||
6. **Manage scanners (optional)**
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scanner download --channel stable
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan run --entry scanners/latest/Scanner.dll --target ./sboms
|
||||
dotnet run --project src/Cli/StellaOps.Cli -- scan upload --file results/scan-001.json
|
||||
```
|
||||
|
||||
Add `--verbose` to any command for structured console logs. All commands honour
|
||||
`Ctrl+C` cancellation and exit with non-zero status codes when the backend returns
|
||||
a problem document.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Verification Checklist
|
||||
|
||||
- Concelier `/health` returns `"status":"healthy"` and Storage bootstrap is marked
|
||||
complete after startup.
|
||||
- CLI commands return HTTP 202 with a `Location` header (job tracking URL) when
|
||||
triggering Concelier jobs.
|
||||
- Export artefacts are materialised under the configured output directories and
|
||||
their manifests record digests.
|
||||
- MongoDB contains the expected `document`, `dto`, `advisory`, and `export_state`
|
||||
collections after a run.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Deployment Automation
|
||||
|
||||
- Treat `etc/concelier.yaml.sample` as the canonical template. CI/CD should copy it to
|
||||
the deployment artifact and replace placeholders (DSN, telemetry endpoints, cron
|
||||
overrides) with environment-specific secrets.
|
||||
- Keep secret material (Mongo credentials, OTLP tokens) outside of the repository;
|
||||
inject them via secret stores or pipeline variables at stamp time.
|
||||
- When building container images, include `trivy-db` (and `oras` if used) so air-gapped
|
||||
clusters do not need outbound downloads at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Next Steps
|
||||
|
||||
- Enable authority-backed authentication in non-production first. Set
|
||||
`authority.enabled: true` while keeping `authority.allowAnonymousFallback: true`
|
||||
to observe logs, then flip it to `false` before 2025-12-31 UTC to enforce tokens.
|
||||
- Automate the workflow above via CI/CD (compose stack or Kubernetes CronJobs).
|
||||
- Pair with the Concelier connector teams when enabling additional sources so their
|
||||
module-specific requirements are pulled in safely.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Authority Integration
|
||||
|
||||
- Concelier now authenticates callers through StellaOps Authority using OAuth 2.0
|
||||
resource server flows. Populate the `authority` block in `concelier.yaml`:
|
||||
|
||||
```yaml
|
||||
authority:
|
||||
enabled: true
|
||||
allowAnonymousFallback: false # keep true only during the staged rollout window
|
||||
issuer: "https://authority.example.org"
|
||||
audiences:
|
||||
- "api://concelier"
|
||||
requiredScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
requiredTenants:
|
||||
- "tenant-default"
|
||||
clientId: "concelier-jobs"
|
||||
clientSecretFile: "../secrets/concelier-jobs.secret"
|
||||
clientScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
bypassNetworks:
|
||||
- "127.0.0.1/32"
|
||||
- "::1/128"
|
||||
```
|
||||
|
||||
- Store the client secret outside of source control. Either provide it via
|
||||
`authority.clientSecret` (environment variable `CONCELIER_AUTHORITY__CLIENTSECRET`)
|
||||
or point `authority.clientSecretFile` to a file mounted at runtime.
|
||||
- Cron jobs running on the same host can keep using the API thanks to the loopback
|
||||
bypass mask. Add additional CIDR ranges as needed; every bypass is logged.
|
||||
- Export the same configuration to Kubernetes or systemd by setting environment
|
||||
variables such as:
|
||||
|
||||
```bash
|
||||
export CONCELIER_AUTHORITY__ENABLED=true
|
||||
export CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=false
|
||||
export CONCELIER_AUTHORITY__ISSUER="https://authority.example.org"
|
||||
export CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
export CONCELIER_AUTHORITY__CLIENTSECRETFILE="/var/run/secrets/concelier/authority-client"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
export CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
export CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
export CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
```
|
||||
|
||||
- CLI commands already pass `Authorization` headers when credentials are supplied.
|
||||
Configure the CLI with matching Authority settings (`docs/09_API_CLI_REFERENCE.md`)
|
||||
so that automation can obtain tokens with the same client credentials. Concelier
|
||||
logs every job request with the client ID, subject (if present), scopes, and
|
||||
a `bypass` flag so operators can audit cron traffic.
|
||||
- **Rollout checklist.**
|
||||
1. Stage the integration with fallback enabled (`allowAnonymousFallback=true`) and confirm CLI/token issuance using `stella auth status`.
|
||||
2. Follow the rehearsal pattern (`allowAnonymousFallback=false`) while monitoring `Concelier.Authorization.Audit` and `web.jobs.triggered`/`web.jobs.trigger.failed` metrics.
|
||||
3. Lock in enforcement, review the audit runbook (`docs/modules/concelier/operations/authority-audit-runbook.md`), and document the bypass CIDR approvals in your change log.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,170 +1,170 @@
|
||||
# 12 - Performance Workbook
|
||||
|
||||
*Purpose* – define **repeatable, data‑driven** benchmarks that guard Stella Ops’ core pledge:
|
||||
> *“P95 vulnerability feedback in ≤ 5 seconds.”*
|
||||
|
||||
---
|
||||
|
||||
## 0 Benchmark Scope
|
||||
|
||||
| Area | Included | Excluded |
|
||||
|------------------|----------------------------------|---------------------------|
|
||||
| SBOM‑first scan | Trivy engine w/ warmed DB | Full image unpack ≥ 300 MB |
|
||||
| Delta SBOM ⭑ | Missing‑layer lookup & merge | Multi‑arch images |
|
||||
| Policy eval ⭑ | YAML → JSON → rule match | Rego (until GA) |
|
||||
| Feed merge | NVD JSON 2023–2025 | GHSA GraphQL (plugin) |
|
||||
| Quota wait‑path | 5 s soft‑wait, 60 s hard‑wait behaviour | Paid tiers (unlimited) |
|
||||
| API latency | REST `/scan`, `/layers/missing` | UI SPA calls |
|
||||
|
||||
⭑ = new in July 2025.
|
||||
|
||||
---
|
||||
|
||||
## 1 Hardware Baseline (Reference Rig)
|
||||
|
||||
| Element | Spec |
|
||||
|-------------|------------------------------------|
|
||||
| CPU | 8 vCPU (Intel Ice‑Lake equiv.) |
|
||||
| Memory | 16 GiB |
|
||||
| Disk | NVMe SSD, 3 GB/s R/W |
|
||||
| Network | 1 Gbit virt. switch |
|
||||
| Container | Docker 25.0 + overlay2 |
|
||||
| OS | Ubuntu 22.04 LTS (kernel 6.8) |
|
||||
|
||||
*All P95 targets assume a **single‑node** deployment on this rig unless stated.*
|
||||
|
||||
---
|
||||
|
||||
## 2 Phase Targets & Gates
|
||||
|
||||
| Phase (ID) | Target P95 | Gate (CI) | Rationale |
|
||||
|-----------------------|-----------:|-----------|----------------------------------------|
|
||||
| **SBOM_FIRST** | ≤ 5 s | `hard` | Core UX promise. |
|
||||
| **IMAGE_UNPACK** | ≤ 10 s | `soft` | Fallback path for legacy flows. |
|
||||
| **DELTA_SBOM** ⭑ | ≤ 1 s | `hard` | Needed to stay sub‑5 s for big bases. |
|
||||
| **POLICY_EVAL** ⭑ | ≤ 50 ms | `hard` | Keeps gate latency invisible to users. |
|
||||
| **QUOTA_WAIT** ⭑ | *soft* ≤ 5 s<br>*hard* ≤ 60 s | `hard` | Ensures graceful Free‑tier throttling. |
|
||||
| **SCHED_RESCAN** | ≤ 30 s | `soft` | Nightly batch – not user‑facing. |
|
||||
| **FEED_MERGE** | ≤ 60 s | `soft` | Off‑peak cron @ 01:00. |
|
||||
| **API_P95** | ≤ 200 ms | `hard` | UI snappiness. |
|
||||
|
||||
*Gate* legend — `hard`: break CI if regression > 3 × target,
|
||||
`soft`: raise warning & issue ticket.
|
||||
|
||||
---
|
||||
|
||||
## 3 Test Harness
|
||||
|
||||
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
|
||||
* **Language analyzers microbench** – `dotnet run --project src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/Bench/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
|
||||
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
|
||||
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
|
||||
|
||||
> **Note** – harness mounts `/var/cache/trivy` tmpfs to avoid disk noise.
|
||||
|
||||
---
|
||||
|
||||
## 4 Current Results (July 2025)
|
||||
|
||||
| Phase | Samples | Mean (s) | P95 (s) | Target OK? |
|
||||
|---------------|--------:|---------:|--------:|-----------:|
|
||||
| SBOM_FIRST | 100 | 3.7 | 4.9 | ✅ |
|
||||
| IMAGE_UNPACK | 50 | 6.4 | 9.2 | ✅ |
|
||||
| **DELTA_SBOM**| 100 | 0.46 | 0.83 | ✅ |
|
||||
| **POLICY_EVAL** | 1 000 | 0.021 | 0.041 | ✅ |
|
||||
| **QUOTA_WAIT** | 80 | 4.0* | 4.9* | ✅ |
|
||||
| SCHED_RESCAN | 10 | 18.3 | 24.9 | ✅ |
|
||||
| FEED_MERGE | 3 | 38.1 | 41.0 | ✅ |
|
||||
| API_P95 | 20 000 | 0.087 | 0.143 | ✅ |
|
||||
|
||||
*Data files:* `bench-artifacts/2025‑07‑14/phase‑stats.json`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Δ‑SBOM Micro‑Benchmark Detail
|
||||
|
||||
### 5.1 Scenario
|
||||
|
||||
1. Base image `python:3.12-slim` already scanned (all layers cached).
|
||||
2. Application layer (`COPY . /app`) triggers new digest.
|
||||
3. `Stella CLI` lists **7** layers, backend replies *6 hit*, *1 miss*.
|
||||
4. Builder scans **only 1 layer** (~9 MiB, 217 files) & uploads delta.
|
||||
|
||||
### 5.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|---------------------|----------:|
|
||||
| `/layers/missing` | 13 |
|
||||
| Trivy single layer | 655 |
|
||||
| Upload delta blob | 88 |
|
||||
| Backend merge + CVE | 74 |
|
||||
| **Total wall‑time** | **830 ms** |
|
||||
|
||||
---
|
||||
|
||||
## 6 Quota Wait‑Path Benchmark Detail
|
||||
|
||||
### 6.1 Scenario
|
||||
|
||||
1. Free‑tier token reaches **scan #200** – dashboard shows yellow banner.
|
||||
|
||||
### 6.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|------------------------------------|----------:|
|
||||
| `/quota/check` Redis LUA INCR | 0.8 |
|
||||
| Soft wait sleep (server) | 5 000 |
|
||||
| Hard wait sleep (server) | 60 000 |
|
||||
| End‑to‑end wall‑time (soft‑hit) | 5 003 |
|
||||
| End‑to‑end wall‑time (hard‑hit) | 60 004 |
|
||||
|
||||
---
|
||||
## 7 Policy Eval Bench
|
||||
|
||||
### 7.1 Setup
|
||||
|
||||
* Policy YAML: **28** rules, mix severity & package conditions.
|
||||
* Input: scan result JSON with **1 026** findings.
|
||||
* Evaluator: custom rules engine (Go structs → map look‑ups).
|
||||
|
||||
### 7.2 Latency Histogram
|
||||
|
||||
```
|
||||
0‑10 ms ▇▇▇▇▇▇▇▇▇▇ 38 %
|
||||
10‑20 ms ▇▇▇▇▇▇▇▇▇▇ 42 %
|
||||
20‑40 ms ▇▇▇▇▇▇ 17 %
|
||||
40‑50 ms ▇ 3 %
|
||||
```
|
||||
|
||||
P99 = 48 ms. Meets 50 ms gate.
|
||||
|
||||
---
|
||||
|
||||
## 8 Trend Snapshot
|
||||
|
||||
# 12 - Performance Workbook
|
||||
|
||||
*Purpose* – define **repeatable, data‑driven** benchmarks that guard Stella Ops’ core pledge:
|
||||
> *“P95 vulnerability feedback in ≤ 5 seconds.”*
|
||||
|
||||
---
|
||||
|
||||
## 0 Benchmark Scope
|
||||
|
||||
| Area | Included | Excluded |
|
||||
|------------------|----------------------------------|---------------------------|
|
||||
| SBOM‑first scan | Trivy engine w/ warmed DB | Full image unpack ≥ 300 MB |
|
||||
| Delta SBOM ⭑ | Missing‑layer lookup & merge | Multi‑arch images |
|
||||
| Policy eval ⭑ | YAML → JSON → rule match | Rego (until GA) |
|
||||
| Feed merge | NVD JSON 2023–2025 | GHSA GraphQL (plugin) |
|
||||
| Quota wait‑path | 5 s soft‑wait, 60 s hard‑wait behaviour | Paid tiers (unlimited) |
|
||||
| API latency | REST `/scan`, `/layers/missing` | UI SPA calls |
|
||||
|
||||
⭑ = new in July 2025.
|
||||
|
||||
---
|
||||
|
||||
## 1 Hardware Baseline (Reference Rig)
|
||||
|
||||
| Element | Spec |
|
||||
|-------------|------------------------------------|
|
||||
| CPU | 8 vCPU (Intel Ice‑Lake equiv.) |
|
||||
| Memory | 16 GiB |
|
||||
| Disk | NVMe SSD, 3 GB/s R/W |
|
||||
| Network | 1 Gbit virt. switch |
|
||||
| Container | Docker 25.0 + overlay2 |
|
||||
| OS | Ubuntu 22.04 LTS (kernel 6.8) |
|
||||
|
||||
*All P95 targets assume a **single‑node** deployment on this rig unless stated.*
|
||||
|
||||
---
|
||||
|
||||
## 2 Phase Targets & Gates
|
||||
|
||||
| Phase (ID) | Target P95 | Gate (CI) | Rationale |
|
||||
|-----------------------|-----------:|-----------|----------------------------------------|
|
||||
| **SBOM_FIRST** | ≤ 5 s | `hard` | Core UX promise. |
|
||||
| **IMAGE_UNPACK** | ≤ 10 s | `soft` | Fallback path for legacy flows. |
|
||||
| **DELTA_SBOM** ⭑ | ≤ 1 s | `hard` | Needed to stay sub‑5 s for big bases. |
|
||||
| **POLICY_EVAL** ⭑ | ≤ 50 ms | `hard` | Keeps gate latency invisible to users. |
|
||||
| **QUOTA_WAIT** ⭑ | *soft* ≤ 5 s<br>*hard* ≤ 60 s | `hard` | Ensures graceful Free‑tier throttling. |
|
||||
| **SCHED_RESCAN** | ≤ 30 s | `soft` | Nightly batch – not user‑facing. |
|
||||
| **FEED_MERGE** | ≤ 60 s | `soft` | Off‑peak cron @ 01:00. |
|
||||
| **API_P95** | ≤ 200 ms | `hard` | UI snappiness. |
|
||||
|
||||
*Gate* legend — `hard`: break CI if regression > 3 × target,
|
||||
`soft`: raise warning & issue ticket.
|
||||
|
||||
---
|
||||
|
||||
## 3 Test Harness
|
||||
|
||||
* **Runner** – `perf/run.sh`, accepts `--phase` and `--samples`.
|
||||
* **Language analyzers microbench** – `dotnet run --project src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/StellaOps.Bench.ScannerAnalyzers.csproj -- --repo-root . --out src/Bench/StellaOps.Bench/Scanner.Analyzers/baseline.csv --json out/bench/scanner-analyzers/latest.json --prom out/bench/scanner-analyzers/latest.prom --commit $(git rev-parse HEAD)` produces CSV + JSON + Prometheus gauges for analyzer scenarios. Runs fail if `max_ms` regresses ≥ 20 % against `baseline.csv` or if thresholds are exceeded.
|
||||
* **Metrics** – Prometheus + `jq` extracts; aggregated via `scripts/aggregate.ts`.
|
||||
* **CI** – GitLab CI job *benchmark* publishes JSON to `bench‑artifacts/`.
|
||||
* **Visualisation** – Grafana dashboard *Stella‑Perf* (provisioned JSON).
|
||||
|
||||
> **Note** – harness mounts `/var/cache/trivy` tmpfs to avoid disk noise.
|
||||
|
||||
---
|
||||
|
||||
## 4 Current Results (July 2025)
|
||||
|
||||
| Phase | Samples | Mean (s) | P95 (s) | Target OK? |
|
||||
|---------------|--------:|---------:|--------:|-----------:|
|
||||
| SBOM_FIRST | 100 | 3.7 | 4.9 | ✅ |
|
||||
| IMAGE_UNPACK | 50 | 6.4 | 9.2 | ✅ |
|
||||
| **DELTA_SBOM**| 100 | 0.46 | 0.83 | ✅ |
|
||||
| **POLICY_EVAL** | 1 000 | 0.021 | 0.041 | ✅ |
|
||||
| **QUOTA_WAIT** | 80 | 4.0* | 4.9* | ✅ |
|
||||
| SCHED_RESCAN | 10 | 18.3 | 24.9 | ✅ |
|
||||
| FEED_MERGE | 3 | 38.1 | 41.0 | ✅ |
|
||||
| API_P95 | 20 000 | 0.087 | 0.143 | ✅ |
|
||||
|
||||
*Data files:* `bench-artifacts/2025‑07‑14/phase‑stats.json`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Δ‑SBOM Micro‑Benchmark Detail
|
||||
|
||||
### 5.1 Scenario
|
||||
|
||||
1. Base image `python:3.12-slim` already scanned (all layers cached).
|
||||
2. Application layer (`COPY . /app`) triggers new digest.
|
||||
3. `Stella CLI` lists **7** layers, backend replies *6 hit*, *1 miss*.
|
||||
4. Builder scans **only 1 layer** (~9 MiB, 217 files) & uploads delta.
|
||||
|
||||
### 5.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|---------------------|----------:|
|
||||
| `/layers/missing` | 13 |
|
||||
| Trivy single layer | 655 |
|
||||
| Upload delta blob | 88 |
|
||||
| Backend merge + CVE | 74 |
|
||||
| **Total wall‑time** | **830 ms** |
|
||||
|
||||
---
|
||||
|
||||
## 6 Quota Wait‑Path Benchmark Detail
|
||||
|
||||
### 6.1 Scenario
|
||||
|
||||
1. Free‑tier token reaches **scan #200** – dashboard shows yellow banner.
|
||||
|
||||
### 6.2 Key Timings
|
||||
|
||||
| Step | Time (ms) |
|
||||
|------------------------------------|----------:|
|
||||
| `/quota/check` Redis LUA INCR | 0.8 |
|
||||
| Soft wait sleep (server) | 5 000 |
|
||||
| Hard wait sleep (server) | 60 000 |
|
||||
| End‑to‑end wall‑time (soft‑hit) | 5 003 |
|
||||
| End‑to‑end wall‑time (hard‑hit) | 60 004 |
|
||||
|
||||
---
|
||||
## 7 Policy Eval Bench
|
||||
|
||||
### 7.1 Setup
|
||||
|
||||
* Policy YAML: **28** rules, mix severity & package conditions.
|
||||
* Input: scan result JSON with **1 026** findings.
|
||||
* Evaluator: custom rules engine (Go structs → map look‑ups).
|
||||
|
||||
### 7.2 Latency Histogram
|
||||
|
||||
```
|
||||
0‑10 ms ▇▇▇▇▇▇▇▇▇▇ 38 %
|
||||
10‑20 ms ▇▇▇▇▇▇▇▇▇▇ 42 %
|
||||
20‑40 ms ▇▇▇▇▇▇ 17 %
|
||||
40‑50 ms ▇ 3 %
|
||||
```
|
||||
|
||||
P99 = 48 ms. Meets 50 ms gate.
|
||||
|
||||
---
|
||||
|
||||
## 8 Trend Snapshot
|
||||
|
||||
> _Perf trend spark‑line screenshot pending upload._
|
||||
|
||||
> **Grafana/Alerting** – Import `docs/modules/scanner/operations/analyzers-grafana-dashboard.json` and point it at the Prometheus datasource storing `scanner_analyzer_bench_*` metrics. Configure an alert on `scanner_analyzer_bench_regression_ratio` ≥ 1.20 (default limit); the bundled Stat panel surfaces breached scenarios (non-zero values). On-call runbook: `docs/modules/scanner/operations/analyzers.md`.
|
||||
|
||||
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
|
||||
|
||||
---
|
||||
|
||||
## 9 Action Items
|
||||
|
||||
1. **Image Unpack** – Evaluate zstd for layer decompress; aim to shave 1 s.
|
||||
2. **Feed Merge** – Parallelise regional XML feed parse (plugin) once stable.
|
||||
3. **Rego Support** – Prototype OPA side‑car; target ≤ 100 ms eval.
|
||||
4. **Concurrency** – Stress‑test 100 rps on 4‑node Redis cluster (Q4‑2025).
|
||||
|
||||
---
|
||||
|
||||
## 10 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|-------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | Added Δ‑SBOM & Policy Eval phases; updated targets & current results. |
|
||||
| 2025‑07‑12 | First public workbook (SBOM‑first, image‑unpack, feed merge). |
|
||||
|
||||
---
|
||||
|
||||
> **Grafana/Alerting** – Import `docs/modules/scanner/operations/analyzers-grafana-dashboard.json` and point it at the Prometheus datasource storing `scanner_analyzer_bench_*` metrics. Configure an alert on `scanner_analyzer_bench_regression_ratio` ≥ 1.20 (default limit); the bundled Stat panel surfaces breached scenarios (non-zero values). On-call runbook: `docs/modules/scanner/operations/analyzers.md`.
|
||||
|
||||
_Plot generated weekly by `scripts/update‑trend.py`; shows last 12 weeks P95 per phase._
|
||||
|
||||
---
|
||||
|
||||
## 9 Action Items
|
||||
|
||||
1. **Image Unpack** – Evaluate zstd for layer decompress; aim to shave 1 s.
|
||||
2. **Feed Merge** – Parallelise regional XML feed parse (plugin) once stable.
|
||||
3. **Rego Support** – Prototype OPA side‑car; target ≤ 100 ms eval.
|
||||
4. **Concurrency** – Stress‑test 100 rps on 4‑node Redis cluster (Q4‑2025).
|
||||
|
||||
---
|
||||
|
||||
## 10 Change Log
|
||||
|
||||
| Date | Note |
|
||||
|------------|-------------------------------------------------------------------------|
|
||||
| 2025‑07‑14 | Added Δ‑SBOM & Policy Eval phases; updated targets & current results. |
|
||||
| 2025‑07‑12 | First public workbook (SBOM‑first, image‑unpack, feed merge). |
|
||||
|
||||
---
|
||||
|
||||
@@ -1,230 +1,230 @@
|
||||
# 13 · Release Engineering Playbook — Stella Ops
|
||||
|
||||
|
||||
A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.
|
||||
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).
|
||||
|
||||
---
|
||||
|
||||
## 0 Release Philosophy
|
||||
|
||||
* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.
|
||||
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.
|
||||
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
|
||||
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 1 Versioning & Branching
|
||||
|
||||
| Branch | Purpose | Auto‑publish? |
|
||||
| ------------- | ------------------------------ | --------------------------------------- |
|
||||
| `main` | Always‑green development trunk | `nightly-*` images |
|
||||
| `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` |
|
||||
| Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart |
|
||||
|
||||
* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
|
||||
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).
|
||||
|
||||
---
|
||||
|
||||
## 2 CI/CD Overview (GitLab CI + GitLab Runner)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[push / MR] --> Lint
|
||||
Lint --> Unit
|
||||
Unit --> Build
|
||||
Build --> Test-Container
|
||||
Test-Container --> SBOM
|
||||
SBOM --> Sign
|
||||
Sign --> Publish
|
||||
Publish --> E2E
|
||||
Publish --> Notify
|
||||
```
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
| Stage | Key tasks |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
| **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. |
|
||||
| **Unit** | `dotnet test`, `go test`, Jest UI tests. |
|
||||
| **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. |
|
||||
| **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
|
||||
| **Test‑Container** | Spin up compose file, run smoke APIs. |
|
||||
| **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. |
|
||||
| **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). |
|
||||
| **Publish** | Push to `registry.git.stella-ops.org`. |
|
||||
| **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. |
|
||||
| **Notify** | Report to Mattermost & GitLab Slack app. |
|
||||
| **OfflineToken** | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context |
|
||||
|
||||
*All stages run in parallel where possible; max wall‑time < 15 min.*
|
||||
|
||||
**Implementation note.** `.gitea/workflows/release.yml` executes
|
||||
`ops/devops/release/build_release.py` to build multi-arch images, attach
|
||||
CycloneDX SBOMs and SLSA provenance with Cosign, and emit
|
||||
`out/release/release.yaml` for downstream packaging (Helm, Compose, Offline Kit).
|
||||
The `build-test-deploy` workflow also runs
|
||||
`python ops/devops/release/test_verify_release.py` so release verifier
|
||||
regressions fail fast during every CI pass.
|
||||
|
||||
---
|
||||
|
||||
## 3 Container Image Strategy
|
||||
|
||||
| Image | Registry Tag | Contents |
|
||||
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
|
||||
| **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. |
|
||||
| **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. |
|
||||
| **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. |
|
||||
| **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. |
|
||||
| **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
|
||||
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. |
|
||||
| **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). |
|
||||
|
||||
*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*
|
||||
|
||||
---
|
||||
|
||||
## 4 📌 Offline Update Kit (OUK) Build & Distribution
|
||||
|
||||
**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.
|
||||
|
||||
### 4.1 CLI Tool
|
||||
|
||||
*Go binary `ouk` lives in `src/Tools/ouk/`.*
|
||||
|
||||
```sh
|
||||
ouk fetch \
|
||||
--nvd --osv \
|
||||
--trivy-db --date $(date -I) \
|
||||
--output ouk-$(date +%Y%m%d).tar.gz \
|
||||
--sign cosign.key
|
||||
```
|
||||
|
||||
### 4.2 Pipeline Hook
|
||||
|
||||
* Runs on **first Friday** each month (cron).
|
||||
* Generates tarball, signs it, uploads to **GitLab Release asset**.
|
||||
* SHA‑256 + signature published alongside.
|
||||
* Release job must emit `out/release/debug/` with `debug-manifest.json` and `.sha256` so `ops/offline-kit/mirror_debug_store.py` can mirror symbols into the Offline Kit (see `DEVOPS-REL-17-004`).
|
||||
|
||||
### 4.3 Activation Flow (runtime)
|
||||
|
||||
1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.
|
||||
2. Backend verifies Cosign signature & digest.
|
||||
3. Files extracted into `var/lib/stella/db`.
|
||||
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
|
||||
5. Audit event `ouk_update` stored.
|
||||
|
||||
### 4.4 Token Detail
|
||||
|
||||
client.jwt placed under /root/ inside the tarball.
|
||||
CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
|
||||
---
|
||||
|
||||
## 5 Artifact Signing & Transparency
|
||||
|
||||
| Artefact | Signer | Tool/Notes |
|
||||
| ------------ | --------------- | ---------------------------------- |
|
||||
| Git tags | GPG (`0x90C4…`) | `git tag -s` |
|
||||
| Containers | Cosign key pair | `cosign sign` |
|
||||
| Helm Charts | prov file | `helm package --sign` |
|
||||
| OUK tarballs | Cosign | `cosign sign-blob` |
|
||||
| Debug store | — | `debug/debug-manifest.json` hashed |
|
||||
|
||||
**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.
|
||||
|
||||
---
|
||||
|
||||
## 6 Release Checklist
|
||||
|
||||
1. CI pipeline green.
|
||||
2. Bump `VERSION` file.
|
||||
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.
|
||||
4. GitLab CI auto‑publishes images & charts.
|
||||
5. Draft GitLab **Release Notes** using `src/Tools/release-notes-gen`.
|
||||
6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`.
|
||||
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step):
|
||||
`python ops/devops/release/test_verify_release.py`
|
||||
8. Mirror the release debug store into the Offline Kit staging tree and re-check the manifest:
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
jq '.artifacts | length' out/offline-kit/debug/debug-manifest.json
|
||||
readelf -n /app/... | grep -i 'Build ID'
|
||||
```
|
||||
Validate that the hash from `readelf` matches the `.build-id/<aa>/<rest>.debug` path created by the script.
|
||||
9. Smoke-test OUK tarball in offline lab.
|
||||
10. Announce in `#stella-release` Mattermost channel.
|
||||
|
||||
---
|
||||
|
||||
## 7 Hot‑fix Procedure
|
||||
|
||||
* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.
|
||||
* Apply minimal patch, add regression test.
|
||||
* CI pipeline (with reduced stages) must pass.
|
||||
* Tag `X.Y.Z+1`.
|
||||
* Publish only container + Helm chart; OUK not rebuilt.
|
||||
* Cherry‑pick back to `main`.
|
||||
|
||||
---
|
||||
|
||||
## 8 Deprecation & End‑of‑Life Policy
|
||||
|
||||
| Feature | Deprecation notice | Removal earliest |
|
||||
| ------------------------ | ------------------ | ---------------- |
|
||||
| Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 |
|
||||
| Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 |
|
||||
| In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 |
|
||||
|
||||
*At least 6 months notice; removal requires major version bump.*
|
||||
|
||||
---
|
||||
|
||||
## 9 📌 Non‑Commercial Usage Rules (English canonical)
|
||||
|
||||
1. **Free for internal security assessments** (company or personal).
|
||||
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).
|
||||
3. If you distribute a fork with UI or backend modifications **you must**:
|
||||
* Publish the complete modified source code.
|
||||
* Retain the original Stella Ops attribution in UI footer and CLI `--version`.
|
||||
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
|
||||
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.
|
||||
|
||||
---
|
||||
|
||||
## 10 Best Practices Snapshot 📌
|
||||
|
||||
* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.
|
||||
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.
|
||||
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).
|
||||
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.
|
||||
* `cosign policy` ensures only images signed by the project key run in production.
|
||||
|
||||
---
|
||||
|
||||
## 11 Contributing to Release Engineering
|
||||
|
||||
* Fork & create MR to `infra/release-*`.
|
||||
* All infra changes require green **`integration-e2e-offline`** job.
|
||||
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.
|
||||
|
||||
---
|
||||
|
||||
## 12 Change Log (high‑level)
|
||||
|
||||
| Version | Date | Note |
|
||||
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
|
||||
| v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. |
|
||||
| v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy |
|
||||
| v1.0 | 2025‑07‑09 | Initial playbook |
|
||||
|
||||
---
|
||||
|
||||
*(End of Release Engineering Playbook v1.1)*
|
||||
# 13 · Release Engineering Playbook — Stella Ops
|
||||
|
||||
|
||||
A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.
|
||||
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).
|
||||
|
||||
---
|
||||
|
||||
## 0 Release Philosophy
|
||||
|
||||
* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.
|
||||
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.
|
||||
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
|
||||
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 1 Versioning & Branching
|
||||
|
||||
| Branch | Purpose | Auto‑publish? |
|
||||
| ------------- | ------------------------------ | --------------------------------------- |
|
||||
| `main` | Always‑green development trunk | `nightly-*` images |
|
||||
| `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` |
|
||||
| Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart |
|
||||
|
||||
* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
|
||||
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).
|
||||
|
||||
---
|
||||
|
||||
## 2 CI/CD Overview (GitLab CI + GitLab Runner)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[push / MR] --> Lint
|
||||
Lint --> Unit
|
||||
Unit --> Build
|
||||
Build --> Test-Container
|
||||
Test-Container --> SBOM
|
||||
SBOM --> Sign
|
||||
Sign --> Publish
|
||||
Publish --> E2E
|
||||
Publish --> Notify
|
||||
```
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
| Stage | Key tasks |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
| **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. |
|
||||
| **Unit** | `dotnet test`, `go test`, Jest UI tests. |
|
||||
| **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. |
|
||||
| **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
|
||||
| **Test‑Container** | Spin up compose file, run smoke APIs. |
|
||||
| **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. |
|
||||
| **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). |
|
||||
| **Publish** | Push to `registry.git.stella-ops.org`. |
|
||||
| **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. |
|
||||
| **Notify** | Report to Mattermost & GitLab Slack app. |
|
||||
| **OfflineToken** | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context |
|
||||
|
||||
*All stages run in parallel where possible; max wall‑time < 15 min.*
|
||||
|
||||
**Implementation note.** `.gitea/workflows/release.yml` executes
|
||||
`ops/devops/release/build_release.py` to build multi-arch images, attach
|
||||
CycloneDX SBOMs and SLSA provenance with Cosign, and emit
|
||||
`out/release/release.yaml` for downstream packaging (Helm, Compose, Offline Kit).
|
||||
The `build-test-deploy` workflow also runs
|
||||
`python ops/devops/release/test_verify_release.py` so release verifier
|
||||
regressions fail fast during every CI pass.
|
||||
|
||||
---
|
||||
|
||||
## 3 Container Image Strategy
|
||||
|
||||
| Image | Registry Tag | Contents |
|
||||
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
|
||||
| **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. |
|
||||
| **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. |
|
||||
| **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. |
|
||||
| **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. |
|
||||
| **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
|
||||
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. |
|
||||
| **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). |
|
||||
|
||||
*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*
|
||||
|
||||
---
|
||||
|
||||
## 4 📌 Offline Update Kit (OUK) Build & Distribution
|
||||
|
||||
**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.
|
||||
|
||||
### 4.1 CLI Tool
|
||||
|
||||
*Go binary `ouk` lives in `src/Tools/ouk/`.*
|
||||
|
||||
```sh
|
||||
ouk fetch \
|
||||
--nvd --osv \
|
||||
--trivy-db --date $(date -I) \
|
||||
--output ouk-$(date +%Y%m%d).tar.gz \
|
||||
--sign cosign.key
|
||||
```
|
||||
|
||||
### 4.2 Pipeline Hook
|
||||
|
||||
* Runs on **first Friday** each month (cron).
|
||||
* Generates tarball, signs it, uploads to **GitLab Release asset**.
|
||||
* SHA‑256 + signature published alongside.
|
||||
* Release job must emit `out/release/debug/` with `debug-manifest.json` and `.sha256` so `ops/offline-kit/mirror_debug_store.py` can mirror symbols into the Offline Kit (see `DEVOPS-REL-17-004`).
|
||||
|
||||
### 4.3 Activation Flow (runtime)
|
||||
|
||||
1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.
|
||||
2. Backend verifies Cosign signature & digest.
|
||||
3. Files extracted into `var/lib/stella/db`.
|
||||
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
|
||||
5. Audit event `ouk_update` stored.
|
||||
|
||||
### 4.4 Token Detail
|
||||
|
||||
client.jwt placed under /root/ inside the tarball.
|
||||
CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
|
||||
---
|
||||
|
||||
## 5 Artifact Signing & Transparency
|
||||
|
||||
| Artefact | Signer | Tool/Notes |
|
||||
| ------------ | --------------- | ---------------------------------- |
|
||||
| Git tags | GPG (`0x90C4…`) | `git tag -s` |
|
||||
| Containers | Cosign key pair | `cosign sign` |
|
||||
| Helm Charts | prov file | `helm package --sign` |
|
||||
| OUK tarballs | Cosign | `cosign sign-blob` |
|
||||
| Debug store | — | `debug/debug-manifest.json` hashed |
|
||||
|
||||
**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.
|
||||
|
||||
---
|
||||
|
||||
## 6 Release Checklist
|
||||
|
||||
1. CI pipeline green.
|
||||
2. Bump `VERSION` file.
|
||||
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.
|
||||
4. GitLab CI auto‑publishes images & charts.
|
||||
5. Draft GitLab **Release Notes** using `src/Tools/release-notes-gen`.
|
||||
6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`.
|
||||
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step):
|
||||
`python ops/devops/release/test_verify_release.py`
|
||||
8. Mirror the release debug store into the Offline Kit staging tree and re-check the manifest:
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
jq '.artifacts | length' out/offline-kit/debug/debug-manifest.json
|
||||
readelf -n /app/... | grep -i 'Build ID'
|
||||
```
|
||||
Validate that the hash from `readelf` matches the `.build-id/<aa>/<rest>.debug` path created by the script.
|
||||
9. Smoke-test OUK tarball in offline lab.
|
||||
10. Announce in `#stella-release` Mattermost channel.
|
||||
|
||||
---
|
||||
|
||||
## 7 Hot‑fix Procedure
|
||||
|
||||
* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.
|
||||
* Apply minimal patch, add regression test.
|
||||
* CI pipeline (with reduced stages) must pass.
|
||||
* Tag `X.Y.Z+1`.
|
||||
* Publish only container + Helm chart; OUK not rebuilt.
|
||||
* Cherry‑pick back to `main`.
|
||||
|
||||
---
|
||||
|
||||
## 8 Deprecation & End‑of‑Life Policy
|
||||
|
||||
| Feature | Deprecation notice | Removal earliest |
|
||||
| ------------------------ | ------------------ | ---------------- |
|
||||
| Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 |
|
||||
| Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 |
|
||||
| In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 |
|
||||
|
||||
*At least 6 months notice; removal requires major version bump.*
|
||||
|
||||
---
|
||||
|
||||
## 9 📌 Non‑Commercial Usage Rules (English canonical)
|
||||
|
||||
1. **Free for internal security assessments** (company or personal).
|
||||
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).
|
||||
3. If you distribute a fork with UI or backend modifications **you must**:
|
||||
* Publish the complete modified source code.
|
||||
* Retain the original Stella Ops attribution in UI footer and CLI `--version`.
|
||||
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
|
||||
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.
|
||||
|
||||
---
|
||||
|
||||
## 10 Best Practices Snapshot 📌
|
||||
|
||||
* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.
|
||||
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.
|
||||
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).
|
||||
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.
|
||||
* `cosign policy` ensures only images signed by the project key run in production.
|
||||
|
||||
---
|
||||
|
||||
## 11 Contributing to Release Engineering
|
||||
|
||||
* Fork & create MR to `infra/release-*`.
|
||||
* All infra changes require green **`integration-e2e-offline`** job.
|
||||
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.
|
||||
|
||||
---
|
||||
|
||||
## 12 Change Log (high‑level)
|
||||
|
||||
| Version | Date | Note |
|
||||
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
|
||||
| v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. |
|
||||
| v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy |
|
||||
| v1.0 | 2025‑07‑09 | Initial playbook |
|
||||
|
||||
---
|
||||
|
||||
*(End of Release Engineering Playbook v1.1)*
|
||||
|
||||
@@ -1,202 +1,202 @@
|
||||
# 17 · Security Hardening Guide — **Stella Ops**
|
||||
*(v2.0 — 12 Jul 2025)*
|
||||
|
||||
> **Audience** — Site‑reliability and platform teams deploying **the open‑source Core** in production or restricted networks.
|
||||
---
|
||||
|
||||
## 0 Table of Contents
|
||||
|
||||
1. Threat model (summary)
|
||||
2. Host‑OS baseline
|
||||
3. Container & runtime hardening
|
||||
4. Network‑plane guidance
|
||||
5. Secrets & key management
|
||||
6. Image, SBOM & plug‑in supply‑chain controls
|
||||
7. Logging, monitoring & audit
|
||||
8. Update & patch strategy
|
||||
9. Incident‑response workflow
|
||||
10. Pen‑testing & continuous assurance
|
||||
11. Contacts & vulnerability disclosure
|
||||
12. Change log
|
||||
|
||||
---
|
||||
|
||||
## 1 Threat model (summary)
|
||||
|
||||
| Asset | Threats | Mitigations |
|
||||
| -------------------- | --------------------- | ---------------------------------------------------------------------- |
|
||||
| SBOMs & scan results | Disclosure, tamper | TLS‑in‑transit, read‑only Redis volume, RBAC, Cosign‑verified plug‑ins |
|
||||
| Backend container | RCE, code‑injection | Distroless image, non‑root UID, read‑only FS, seccomp + `CAP_DROP:ALL` |
|
||||
| Update artefacts | Supply‑chain attack | Cosign‑signed images & SBOMs, enforced by admission controller |
|
||||
| Admin credentials | Phishing, brute force | OAuth 2.0 with 12‑h token TTL, optional mTLS |
|
||||
|
||||
---
|
||||
|
||||
## 2 Host‑OS baseline checklist
|
||||
|
||||
| Item | Recommended setting |
|
||||
| ------------- | --------------------------------------------------------- |
|
||||
| OS | Ubuntu 22.04 LTS (kernel ≥ 5.15) or Alma 9 |
|
||||
| Patches | `unattended‑upgrades` or vendor‑equivalent enabled |
|
||||
| Filesystem | `noexec,nosuid` on `/tmp`, `/var/tmp` |
|
||||
| Docker Engine | v24.*, API socket root‑owned (`0660`) |
|
||||
| Auditd | Watch `/etc/docker`, `/usr/bin/docker*` and Compose files |
|
||||
| Time sync | `chrony` or `systemd‑timesyncd` |
|
||||
|
||||
---
|
||||
|
||||
## 3 Container & runtime hardening
|
||||
|
||||
### 3.1 Docker Compose reference (`compose-core.yml`)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
backend:
|
||||
image: registry.stella-ops.org/stella-ops/stella-ops:<PINNED_TAG_OR_DIGEST>
|
||||
user: "101:101" # non‑root
|
||||
read_only: true
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
- "seccomp:./seccomp-backend.json"
|
||||
cap_drop: [ALL]
|
||||
tmpfs:
|
||||
- /tmp:size=64m,exec,nosymlink
|
||||
environment:
|
||||
- ASPNETCORE_URLS=https://+:8080
|
||||
- TLSPROVIDER=OpenSslGost
|
||||
depends_on: [redis]
|
||||
networks: [core-net]
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "https://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
redis:
|
||||
image: redis:7.2-alpine
|
||||
command: ["redis-server", "--requirepass", "${REDIS_PASS}", "--rename-command", "FLUSHALL", ""]
|
||||
user: "redis"
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
tmpfs:
|
||||
- /data
|
||||
networks: [core-net]
|
||||
|
||||
networks:
|
||||
core-net:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
No dedicated “Redis” or “Mongo” sub‑nets are declared; the single bridge network suffices for the default stack.
|
||||
|
||||
### 3.2 Kubernetes deployment highlights
|
||||
|
||||
Use a separate NetworkPolicy that only allows egress from backend to Redis :6379.
|
||||
securityContext: runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, drop all capabilities.
|
||||
PodDisruptionBudget of minAvailable: 1.
|
||||
Optionally add CosignVerified=true label enforced by an admission controller (e.g. Kyverno or Connaisseur).
|
||||
|
||||
## 4 Network‑plane guidance
|
||||
|
||||
| Plane | Recommendation |
|
||||
| ------------------ | -------------------------------------------------------------------------- |
|
||||
| North‑south | Terminate TLS 1.2+ (OpenSSL‑GOST default). Use LetsEncrypt or internal CA. |
|
||||
| East‑west | Compose bridge or K8s ClusterIP only; no public Redis/Mongo ports. |
|
||||
| Ingress controller | Limit methods to GET, POST, PATCH (no TRACE). |
|
||||
| Rate‑limits | 40 rps default; tune ScannerPool.Workers and ingress limit‑req to match. |
|
||||
|
||||
## 5 Secrets & key management
|
||||
|
||||
| Secret | Storage | Rotation |
|
||||
| --------------------------------- | ---------------------------------- | ----------------------------- |
|
||||
| **Client‑JWT (offline)** | `/var/lib/stella/tokens/client.jwt` (root : 600) | **30 days** – provided by each OUK |
|
||||
| REDIS_PASS | Docker/K8s secret | 90 days |
|
||||
| OAuth signing key | /keys/jwt.pem (read‑only mount) | 180 days |
|
||||
| Cosign public key | /keys/cosign.pub baked into image; | change on every major release |
|
||||
| Trivy DB mirror token (if remote) | Secret + read‑only | 30 days |
|
||||
|
||||
Never bake secrets into images; always inject at runtime.
|
||||
|
||||
> **Operational tip:** schedule a cron reminding ops 5 days before
|
||||
> `client.jwt` expiry. The backend also emits a Prometheus metric
|
||||
> `stella_quota_token_days_remaining`.
|
||||
|
||||
## 6 Image, SBOM & plug‑in supply‑chain controls
|
||||
|
||||
* Images — Pull by digest not latest; verify:
|
||||
|
||||
```bash
|
||||
cosign verify ghcr.io/stellaops/backend@sha256:<DIGEST> \
|
||||
--key https://stella-ops.org/keys/cosign.pub
|
||||
```
|
||||
|
||||
* SBOM — Each release ships an SPDX file; store alongside images for audit.
|
||||
* Third‑party plug‑ins — Place in /plugins/; backend will:
|
||||
* Validate Cosign signature.
|
||||
* Check [StellaPluginVersion("major.minor")].
|
||||
* Refuse to start if Security.DisablePluginUnsigned=false (default).
|
||||
|
||||
## 7 Logging, monitoring & audit
|
||||
|
||||
| Control | Implementation |
|
||||
| ------------ | ----------------------------------------------------------------- |
|
||||
| Log format | Serilog JSON; ship via Fluent‑Bit to ELK or Loki |
|
||||
| Metrics | Prometheus /metrics endpoint; default Grafana dashboard in infra/ |
|
||||
| Audit events | Redis stream audit; export daily to SIEM |
|
||||
| Alert rules | Feed age ≥ 48 h, P95 wall‑time > 5 s, Redis used memory > 75 % |
|
||||
|
||||
### 7.1 Concelier authorization audits
|
||||
|
||||
- Enable the Authority integration for Concelier (`authority.enabled=true`). Keep
|
||||
`authority.allowAnonymousFallback` set to `true` only during migration and plan
|
||||
to disable it before **2025-12-31 UTC** so the `/jobs*` surface always demands
|
||||
a bearer token.
|
||||
- Store the Authority client secret using Docker/Kubernetes secrets and point
|
||||
`authority.clientSecretFile` at the mounted path; the value is read at startup
|
||||
and never logged.
|
||||
- Watch the `Concelier.Authorization.Audit` logger. Each entry contains the HTTP
|
||||
status, subject, client ID, scopes, remote IP, and a boolean `bypass` flag
|
||||
showing whether a network bypass CIDR allowed the request. Configure your SIEM
|
||||
to alert when unauthenticated requests (`status=401`) appear with
|
||||
`bypass=true`, or when unexpected scopes invoke job triggers.
|
||||
Detailed monitoring and response guidance lives in `docs/modules/concelier/operations/authority-audit-runbook.md`.
|
||||
|
||||
## 8 Update & patch strategy
|
||||
|
||||
| Layer | Cadence | Method |
|
||||
| -------------------- | -------------------------------------------------------- | ------------------------------ |
|
||||
| Backend & CLI images | Monthly or CVE‑driven docker pull + docker compose up -d |
|
||||
| Trivy DB | 24 h scheduler via Concelier (vulnerability ingest/merge/export service) | configurable via Concelier scheduler options |
|
||||
| Docker Engine | vendor LTS | distro package manager |
|
||||
| Host OS | security repos enabled | unattended‑upgrades |
|
||||
|
||||
## 9 Incident‑response workflow
|
||||
|
||||
* Detect — PagerDuty alert from Prometheus or SIEM.
|
||||
* Contain — Stop affected Backend container; isolate Redis RDB snapshot.
|
||||
* Eradicate — Pull verified images, redeploy, rotate secrets.
|
||||
* Recover — Restore RDB, replay SBOMs if history lost.
|
||||
* Review — Post‑mortem within 72 h; create follow‑up issues.
|
||||
* Escalate P1 incidents to <security@stella‑ops.org> (24 × 7).
|
||||
|
||||
|
||||
## 10 Pen‑testing & continuous assurance
|
||||
|
||||
| Control | Frequency | Tool/Runner |
|
||||
|----------------------|-----------------------|-------------------------------------------|
|
||||
| OWASP ZAP baseline | Each merge to `main` | GitHub Action `zap-baseline-scan` |
|
||||
| Dependency scanning | Per pull request | Trivy FS + Dependabot |
|
||||
| External red‑team | Annual or pre‑GA | CREST‑accredited third‑party |
|
||||
|
||||
## 11 Vulnerability disclosure & contact
|
||||
|
||||
* Preferred channel: security@stella‑ops.org (GPG key on website).
|
||||
* Coordinated disclosure reward: public credit and swag (no monetary bounty at this time).
|
||||
|
||||
## 12 Change log
|
||||
|
||||
| Version | Date | Notes |
|
||||
| ------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.0 | 2025‑07‑12 | Full overhaul: host‑OS baseline, supply‑chain signing, removal of unnecessary sub‑nets, role‑based contact e‑mail, K8s guidance. |
|
||||
| v1.1 | 2025‑07‑09 | Minor fence fixes. |
|
||||
| v1.0 | 2025‑07‑09 | Original draft. |
|
||||
# 17 · Security Hardening Guide — **Stella Ops**
|
||||
*(v2.0 — 12 Jul 2025)*
|
||||
|
||||
> **Audience** — Site‑reliability and platform teams deploying **the open‑source Core** in production or restricted networks.
|
||||
---
|
||||
|
||||
## 0 Table of Contents
|
||||
|
||||
1. Threat model (summary)
|
||||
2. Host‑OS baseline
|
||||
3. Container & runtime hardening
|
||||
4. Network‑plane guidance
|
||||
5. Secrets & key management
|
||||
6. Image, SBOM & plug‑in supply‑chain controls
|
||||
7. Logging, monitoring & audit
|
||||
8. Update & patch strategy
|
||||
9. Incident‑response workflow
|
||||
10. Pen‑testing & continuous assurance
|
||||
11. Contacts & vulnerability disclosure
|
||||
12. Change log
|
||||
|
||||
---
|
||||
|
||||
## 1 Threat model (summary)
|
||||
|
||||
| Asset | Threats | Mitigations |
|
||||
| -------------------- | --------------------- | ---------------------------------------------------------------------- |
|
||||
| SBOMs & scan results | Disclosure, tamper | TLS‑in‑transit, read‑only Redis volume, RBAC, Cosign‑verified plug‑ins |
|
||||
| Backend container | RCE, code‑injection | Distroless image, non‑root UID, read‑only FS, seccomp + `CAP_DROP:ALL` |
|
||||
| Update artefacts | Supply‑chain attack | Cosign‑signed images & SBOMs, enforced by admission controller |
|
||||
| Admin credentials | Phishing, brute force | OAuth 2.0 with 12‑h token TTL, optional mTLS |
|
||||
|
||||
---
|
||||
|
||||
## 2 Host‑OS baseline checklist
|
||||
|
||||
| Item | Recommended setting |
|
||||
| ------------- | --------------------------------------------------------- |
|
||||
| OS | Ubuntu 22.04 LTS (kernel ≥ 5.15) or Alma 9 |
|
||||
| Patches | `unattended‑upgrades` or vendor‑equivalent enabled |
|
||||
| Filesystem | `noexec,nosuid` on `/tmp`, `/var/tmp` |
|
||||
| Docker Engine | v24.*, API socket root‑owned (`0660`) |
|
||||
| Auditd | Watch `/etc/docker`, `/usr/bin/docker*` and Compose files |
|
||||
| Time sync | `chrony` or `systemd‑timesyncd` |
|
||||
|
||||
---
|
||||
|
||||
## 3 Container & runtime hardening
|
||||
|
||||
### 3.1 Docker Compose reference (`compose-core.yml`)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
backend:
|
||||
image: registry.stella-ops.org/stella-ops/stella-ops:<PINNED_TAG_OR_DIGEST>
|
||||
user: "101:101" # non‑root
|
||||
read_only: true
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
- "seccomp:./seccomp-backend.json"
|
||||
cap_drop: [ALL]
|
||||
tmpfs:
|
||||
- /tmp:size=64m,exec,nosymlink
|
||||
environment:
|
||||
- ASPNETCORE_URLS=https://+:8080
|
||||
- TLSPROVIDER=OpenSslGost
|
||||
depends_on: [redis]
|
||||
networks: [core-net]
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "https://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
redis:
|
||||
image: redis:7.2-alpine
|
||||
command: ["redis-server", "--requirepass", "${REDIS_PASS}", "--rename-command", "FLUSHALL", ""]
|
||||
user: "redis"
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
tmpfs:
|
||||
- /data
|
||||
networks: [core-net]
|
||||
|
||||
networks:
|
||||
core-net:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
No dedicated “Redis” or “Mongo” sub‑nets are declared; the single bridge network suffices for the default stack.
|
||||
|
||||
### 3.2 Kubernetes deployment highlights
|
||||
|
||||
Use a separate NetworkPolicy that only allows egress from backend to Redis :6379.
|
||||
securityContext: runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, drop all capabilities.
|
||||
PodDisruptionBudget of minAvailable: 1.
|
||||
Optionally add CosignVerified=true label enforced by an admission controller (e.g. Kyverno or Connaisseur).
|
||||
|
||||
## 4 Network‑plane guidance
|
||||
|
||||
| Plane | Recommendation |
|
||||
| ------------------ | -------------------------------------------------------------------------- |
|
||||
| North‑south | Terminate TLS 1.2+ (OpenSSL‑GOST default). Use LetsEncrypt or internal CA. |
|
||||
| East‑west | Compose bridge or K8s ClusterIP only; no public Redis/Mongo ports. |
|
||||
| Ingress controller | Limit methods to GET, POST, PATCH (no TRACE). |
|
||||
| Rate‑limits | 40 rps default; tune ScannerPool.Workers and ingress limit‑req to match. |
|
||||
|
||||
## 5 Secrets & key management
|
||||
|
||||
| Secret | Storage | Rotation |
|
||||
| --------------------------------- | ---------------------------------- | ----------------------------- |
|
||||
| **Client‑JWT (offline)** | `/var/lib/stella/tokens/client.jwt` (root : 600) | **30 days** – provided by each OUK |
|
||||
| REDIS_PASS | Docker/K8s secret | 90 days |
|
||||
| OAuth signing key | /keys/jwt.pem (read‑only mount) | 180 days |
|
||||
| Cosign public key | /keys/cosign.pub baked into image; | change on every major release |
|
||||
| Trivy DB mirror token (if remote) | Secret + read‑only | 30 days |
|
||||
|
||||
Never bake secrets into images; always inject at runtime.
|
||||
|
||||
> **Operational tip:** schedule a cron reminding ops 5 days before
|
||||
> `client.jwt` expiry. The backend also emits a Prometheus metric
|
||||
> `stella_quota_token_days_remaining`.
|
||||
|
||||
## 6 Image, SBOM & plug‑in supply‑chain controls
|
||||
|
||||
* Images — Pull by digest not latest; verify:
|
||||
|
||||
```bash
|
||||
cosign verify ghcr.io/stellaops/backend@sha256:<DIGEST> \
|
||||
--key https://stella-ops.org/keys/cosign.pub
|
||||
```
|
||||
|
||||
* SBOM — Each release ships an SPDX file; store alongside images for audit.
|
||||
* Third‑party plug‑ins — Place in /plugins/; backend will:
|
||||
* Validate Cosign signature.
|
||||
* Check [StellaPluginVersion("major.minor")].
|
||||
* Refuse to start if Security.DisablePluginUnsigned=false (default).
|
||||
|
||||
## 7 Logging, monitoring & audit
|
||||
|
||||
| Control | Implementation |
|
||||
| ------------ | ----------------------------------------------------------------- |
|
||||
| Log format | Serilog JSON; ship via Fluent‑Bit to ELK or Loki |
|
||||
| Metrics | Prometheus /metrics endpoint; default Grafana dashboard in infra/ |
|
||||
| Audit events | Redis stream audit; export daily to SIEM |
|
||||
| Alert rules | Feed age ≥ 48 h, P95 wall‑time > 5 s, Redis used memory > 75 % |
|
||||
|
||||
### 7.1 Concelier authorization audits
|
||||
|
||||
- Enable the Authority integration for Concelier (`authority.enabled=true`). Keep
|
||||
`authority.allowAnonymousFallback` set to `true` only during migration and plan
|
||||
to disable it before **2025-12-31 UTC** so the `/jobs*` surface always demands
|
||||
a bearer token.
|
||||
- Store the Authority client secret using Docker/Kubernetes secrets and point
|
||||
`authority.clientSecretFile` at the mounted path; the value is read at startup
|
||||
and never logged.
|
||||
- Watch the `Concelier.Authorization.Audit` logger. Each entry contains the HTTP
|
||||
status, subject, client ID, scopes, remote IP, and a boolean `bypass` flag
|
||||
showing whether a network bypass CIDR allowed the request. Configure your SIEM
|
||||
to alert when unauthenticated requests (`status=401`) appear with
|
||||
`bypass=true`, or when unexpected scopes invoke job triggers.
|
||||
Detailed monitoring and response guidance lives in `docs/modules/concelier/operations/authority-audit-runbook.md`.
|
||||
|
||||
## 8 Update & patch strategy
|
||||
|
||||
| Layer | Cadence | Method |
|
||||
| -------------------- | -------------------------------------------------------- | ------------------------------ |
|
||||
| Backend & CLI images | Monthly or CVE‑driven docker pull + docker compose up -d |
|
||||
| Trivy DB | 24 h scheduler via Concelier (vulnerability ingest/merge/export service) | configurable via Concelier scheduler options |
|
||||
| Docker Engine | vendor LTS | distro package manager |
|
||||
| Host OS | security repos enabled | unattended‑upgrades |
|
||||
|
||||
## 9 Incident‑response workflow
|
||||
|
||||
* Detect — PagerDuty alert from Prometheus or SIEM.
|
||||
* Contain — Stop affected Backend container; isolate Redis RDB snapshot.
|
||||
* Eradicate — Pull verified images, redeploy, rotate secrets.
|
||||
* Recover — Restore RDB, replay SBOMs if history lost.
|
||||
* Review — Post‑mortem within 72 h; create follow‑up issues.
|
||||
* Escalate P1 incidents to <security@stella‑ops.org> (24 × 7).
|
||||
|
||||
|
||||
## 10 Pen‑testing & continuous assurance
|
||||
|
||||
| Control | Frequency | Tool/Runner |
|
||||
|----------------------|-----------------------|-------------------------------------------|
|
||||
| OWASP ZAP baseline | Each merge to `main` | GitHub Action `zap-baseline-scan` |
|
||||
| Dependency scanning | Per pull request | Trivy FS + Dependabot |
|
||||
| External red‑team | Annual or pre‑GA | CREST‑accredited third‑party |
|
||||
|
||||
## 11 Vulnerability disclosure & contact
|
||||
|
||||
* Preferred channel: security@stella‑ops.org (GPG key on website).
|
||||
* Coordinated disclosure reward: public credit and swag (no monetary bounty at this time).
|
||||
|
||||
## 12 Change log
|
||||
|
||||
| Version | Date | Notes |
|
||||
| ------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.0 | 2025‑07‑12 | Full overhaul: host‑OS baseline, supply‑chain signing, removal of unnecessary sub‑nets, role‑based contact e‑mail, K8s guidance. |
|
||||
| v1.1 | 2025‑07‑09 | Minor fence fixes. |
|
||||
| v1.0 | 2025‑07‑09 | Original draft. |
|
||||
|
||||
@@ -1,190 +1,190 @@
|
||||
# Stella Ops — Installation Guide (Docker & Air‑Gap)
|
||||
|
||||
<!--
|
||||
This file is processed by the Eleventy build.
|
||||
Do **not** hard‑code versions or quota numbers; inherit from
|
||||
docs/_includes/CONSTANTS.md instead.
|
||||
{{ dotnet }} → ".NET 10 LTS"
|
||||
{{ angular }} → "20"
|
||||
-->
|
||||
|
||||
> **Status — public α not yet published.**
|
||||
> The commands below will work as soon as the first image is tagged
|
||||
> `registry.stella-ops.org/stella-ops/stella-ops:0.1.0-alpha`
|
||||
> (target date: **late 2025**). Track progress on the
|
||||
> [road‑map](/roadmap/).
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
| Item | Minimum | Notes |
|
||||
|------|---------|-------|
|
||||
| Linux | Ubuntu 22.04 LTS / Alma 9 | x86‑64 or arm64 |
|
||||
| CPU / RAM | 2 vCPU / 2 GiB | Laptop baseline |
|
||||
| Disk | 10 GiB SSD | SBOM + vuln DB cache |
|
||||
| Docker | **Engine 25 + Compose v2** | `docker -v` |
|
||||
| TLS | OpenSSL 1.1 + | Self‑signed cert generated at first run |
|
||||
|
||||
---
|
||||
|
||||
## 1 · Connected‑host install (Docker Compose)
|
||||
|
||||
```bash
|
||||
# 1. Make a working directory
|
||||
mkdir stella && cd stella
|
||||
|
||||
# 2. Download the signed Compose bundle + example .env
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml.sig
|
||||
|
||||
# 3. Verify provenance (Cosign public key is stable)
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature .env.example.sig \
|
||||
.env.example
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.infrastructure.yml.sig \
|
||||
docker-compose.infrastructure.yml
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.stella-ops.yml.sig \
|
||||
docker-compose.stella-ops.yml
|
||||
|
||||
# 4. Copy .env.example → .env and edit secrets
|
||||
cp .env.example .env
|
||||
$EDITOR .env
|
||||
|
||||
# 5. Launch databases (MongoDB + Redis)
|
||||
docker compose --env-file .env -f docker-compose.infrastructure.yml up -d
|
||||
|
||||
# 6. Launch Stella Ops (first run pulls ~50 MB merged vuln DB)
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d
|
||||
````
|
||||
|
||||
*Default login:* `admin / changeme`
|
||||
UI: [https://\<host\>:8443](https://<host>:8443) (self‑signed certificate)
|
||||
|
||||
> **Pinning best‑practice** – in production environments replace
|
||||
> `stella-ops:latest` with the immutable digest printed by
|
||||
> `docker images --digests`.
|
||||
|
||||
> **Repo bundles** – Development, staging, and air‑gapped Compose profiles live
|
||||
> under `deploy/compose/`, already tied to the release manifests in
|
||||
> `deploy/releases/`. Helm users can pull the same channel overlays from
|
||||
> `deploy/helm/stellaops/values-*.yaml` and validate everything with
|
||||
> `deploy/tools/validate-profiles.sh`.
|
||||
|
||||
### 1.1 · Concelier authority configuration
|
||||
|
||||
The Concelier container reads configuration from `etc/concelier.yaml` plus
|
||||
`CONCELIER_` environment variables. To enable the new Authority integration:
|
||||
|
||||
1. Add the following keys to `.env` (replace values for your environment):
|
||||
|
||||
```bash
|
||||
CONCELIER_AUTHORITY__ENABLED=true
|
||||
CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true # temporary rollout only
|
||||
CONCELIER_AUTHORITY__ISSUER="https://authority.internal"
|
||||
CONCELIER_AUTHORITY__AUDIENCES__0="api://concelier"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__CLIENTSECRETFILE="/run/secrets/concelier_authority_client"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__0="127.0.0.1/32"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__1="::1/128"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0="00:00:01"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1="00:00:02"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__2="00:00:05"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE="00:10:00"
|
||||
```
|
||||
|
||||
Store the client secret outside source control (Docker secrets, mounted file,
|
||||
or Kubernetes Secret). Concelier loads the secret during post-configuration, so
|
||||
the value never needs to appear in the YAML template.
|
||||
|
||||
Connected sites can keep the retry ladder short (1 s, 2 s, 5 s) so job triggers fail fast when Authority is down. For air‑gapped or intermittently connected deployments, extend `RESILIENCE__OFFLINECACHETOLERANCE` (e.g. `00:30:00`) so cached discovery/JWKS data remains valid while the Offline Kit synchronises upstream changes.
|
||||
|
||||
2. Redeploy Concelier:
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d concelier
|
||||
```
|
||||
|
||||
3. Tail the logs: `docker compose logs -f concelier`. Successful `/jobs*` calls now
|
||||
emit `Concelier.Authorization.Audit` entries with `route`, `status`, `subject`,
|
||||
`clientId`, `scopes`, `bypass`, and `remote` fields. 401 denials keep the same
|
||||
shape—watch for `bypass=True`, which indicates a bypass CIDR accepted an anonymous
|
||||
call. See `docs/modules/concelier/operations/authority-audit-runbook.md` for a full audit/alerting checklist.
|
||||
|
||||
> **Enforcement deadline** – keep `CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true`
|
||||
> only while validating the rollout. Set it to `false` (and restart Concelier)
|
||||
> before **2025-12-31 UTC** to require tokens in production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Optional: request a free quota token
|
||||
|
||||
Anonymous installs allow **{{ quota\_anon }} scans per UTC day**.
|
||||
Email `token@stella-ops.org` to receive a signed JWT that raises the limit to
|
||||
**{{ quota\_token }} scans/day**. Insert it into `.env`:
|
||||
|
||||
```bash
|
||||
STELLA_JWT="paste‑token‑here"
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella-ops stella set-jwt "$STELLA_JWT"
|
||||
```
|
||||
|
||||
> The UI shows a reminder at 200 scans and throttles above the limit but will
|
||||
> **never block** your pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Air‑gapped install (Offline Update Kit)
|
||||
|
||||
When running on an isolated network use the **Offline Update Kit (OUK)**:
|
||||
|
||||
```bash
|
||||
# Download & verify on a connected host
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz.sig
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-v0.1a.tgz.sig \
|
||||
stella-ops-offline-kit-v0.1a.tgz
|
||||
|
||||
# Transfer → air‑gap → import
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella admin import-offline-usage-kit stella-ops-offline-kit-v0.1a.tgz
|
||||
```
|
||||
|
||||
*Import is atomic; no service downtime.*
|
||||
|
||||
For details see the dedicated [Offline Kit guide](/offline/).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Next steps
|
||||
|
||||
* **5‑min Quick‑Start:** `/quickstart/`
|
||||
* **CI recipes:** `docs/ci/20_CI_RECIPES.md`
|
||||
* **Plug‑in SDK:** `/plugins/`
|
||||
|
||||
---
|
||||
|
||||
*Generated {{ "now" | date: "%Y‑%m‑%d" }} — build tags inserted at render time.*
|
||||
# Stella Ops — Installation Guide (Docker & Air‑Gap)
|
||||
|
||||
<!--
|
||||
This file is processed by the Eleventy build.
|
||||
Do **not** hard‑code versions or quota numbers; inherit from
|
||||
docs/_includes/CONSTANTS.md instead.
|
||||
{{ dotnet }} → ".NET 10 LTS"
|
||||
{{ angular }} → "20"
|
||||
-->
|
||||
|
||||
> **Status — public α not yet published.**
|
||||
> The commands below will work as soon as the first image is tagged
|
||||
> `registry.stella-ops.org/stella-ops/stella-ops:0.1.0-alpha`
|
||||
> (target date: **late 2025**). Track progress on the
|
||||
> [road‑map](/roadmap/).
|
||||
|
||||
---
|
||||
|
||||
## 0 · Prerequisites
|
||||
|
||||
| Item | Minimum | Notes |
|
||||
|------|---------|-------|
|
||||
| Linux | Ubuntu 22.04 LTS / Alma 9 | x86‑64 or arm64 |
|
||||
| CPU / RAM | 2 vCPU / 2 GiB | Laptop baseline |
|
||||
| Disk | 10 GiB SSD | SBOM + vuln DB cache |
|
||||
| Docker | **Engine 25 + Compose v2** | `docker -v` |
|
||||
| TLS | OpenSSL 1.1 + | Self‑signed cert generated at first run |
|
||||
|
||||
---
|
||||
|
||||
## 1 · Connected‑host install (Docker Compose)
|
||||
|
||||
```bash
|
||||
# 1. Make a working directory
|
||||
mkdir stella && cd stella
|
||||
|
||||
# 2. Download the signed Compose bundle + example .env
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example
|
||||
curl -LO https://get.stella-ops.org/releases/latest/.env.example.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.infrastructure.yml.sig
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml
|
||||
curl -LO https://get.stella-ops.org/releases/latest/docker-compose.stella-ops.yml.sig
|
||||
|
||||
# 3. Verify provenance (Cosign public key is stable)
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature .env.example.sig \
|
||||
.env.example
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.infrastructure.yml.sig \
|
||||
docker-compose.infrastructure.yml
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature docker-compose.stella-ops.yml.sig \
|
||||
docker-compose.stella-ops.yml
|
||||
|
||||
# 4. Copy .env.example → .env and edit secrets
|
||||
cp .env.example .env
|
||||
$EDITOR .env
|
||||
|
||||
# 5. Launch databases (MongoDB + Redis)
|
||||
docker compose --env-file .env -f docker-compose.infrastructure.yml up -d
|
||||
|
||||
# 6. Launch Stella Ops (first run pulls ~50 MB merged vuln DB)
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d
|
||||
````
|
||||
|
||||
*Default login:* `admin / changeme`
|
||||
UI: [https://\<host\>:8443](https://<host>:8443) (self‑signed certificate)
|
||||
|
||||
> **Pinning best‑practice** – in production environments replace
|
||||
> `stella-ops:latest` with the immutable digest printed by
|
||||
> `docker images --digests`.
|
||||
|
||||
> **Repo bundles** – Development, staging, and air‑gapped Compose profiles live
|
||||
> under `deploy/compose/`, already tied to the release manifests in
|
||||
> `deploy/releases/`. Helm users can pull the same channel overlays from
|
||||
> `deploy/helm/stellaops/values-*.yaml` and validate everything with
|
||||
> `deploy/tools/validate-profiles.sh`.
|
||||
|
||||
### 1.1 · Concelier authority configuration
|
||||
|
||||
The Concelier container reads configuration from `etc/concelier.yaml` plus
|
||||
`CONCELIER_` environment variables. To enable the new Authority integration:
|
||||
|
||||
1. Add the following keys to `.env` (replace values for your environment):
|
||||
|
||||
```bash
|
||||
CONCELIER_AUTHORITY__ENABLED=true
|
||||
CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true # temporary rollout only
|
||||
CONCELIER_AUTHORITY__ISSUER="https://authority.internal"
|
||||
CONCELIER_AUTHORITY__AUDIENCES__0="api://concelier"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__REQUIREDSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__REQUIREDTENANTS__0="tenant-default"
|
||||
CONCELIER_AUTHORITY__CLIENTID="concelier-jobs"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__0="concelier.jobs.trigger"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__1="advisory:read"
|
||||
CONCELIER_AUTHORITY__CLIENTSCOPES__2="advisory:ingest"
|
||||
CONCELIER_AUTHORITY__CLIENTSECRETFILE="/run/secrets/concelier_authority_client"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__0="127.0.0.1/32"
|
||||
CONCELIER_AUTHORITY__BYPASSNETWORKS__1="::1/128"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0="00:00:01"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1="00:00:02"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__2="00:00:05"
|
||||
CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK=true
|
||||
CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE="00:10:00"
|
||||
```
|
||||
|
||||
Store the client secret outside source control (Docker secrets, mounted file,
|
||||
or Kubernetes Secret). Concelier loads the secret during post-configuration, so
|
||||
the value never needs to appear in the YAML template.
|
||||
|
||||
Connected sites can keep the retry ladder short (1 s, 2 s, 5 s) so job triggers fail fast when Authority is down. For air‑gapped or intermittently connected deployments, extend `RESILIENCE__OFFLINECACHETOLERANCE` (e.g. `00:30:00`) so cached discovery/JWKS data remains valid while the Offline Kit synchronises upstream changes.
|
||||
|
||||
2. Redeploy Concelier:
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d concelier
|
||||
```
|
||||
|
||||
3. Tail the logs: `docker compose logs -f concelier`. Successful `/jobs*` calls now
|
||||
emit `Concelier.Authorization.Audit` entries with `route`, `status`, `subject`,
|
||||
`clientId`, `scopes`, `bypass`, and `remote` fields. 401 denials keep the same
|
||||
shape—watch for `bypass=True`, which indicates a bypass CIDR accepted an anonymous
|
||||
call. See `docs/modules/concelier/operations/authority-audit-runbook.md` for a full audit/alerting checklist.
|
||||
|
||||
> **Enforcement deadline** – keep `CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK=true`
|
||||
> only while validating the rollout. Set it to `false` (and restart Concelier)
|
||||
> before **2025-12-31 UTC** to require tokens in production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Optional: request a free quota token
|
||||
|
||||
Anonymous installs allow **{{ quota\_anon }} scans per UTC day**.
|
||||
Email `token@stella-ops.org` to receive a signed JWT that raises the limit to
|
||||
**{{ quota\_token }} scans/day**. Insert it into `.env`:
|
||||
|
||||
```bash
|
||||
STELLA_JWT="paste‑token‑here"
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella-ops stella set-jwt "$STELLA_JWT"
|
||||
```
|
||||
|
||||
> The UI shows a reminder at 200 scans and throttles above the limit but will
|
||||
> **never block** your pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Air‑gapped install (Offline Update Kit)
|
||||
|
||||
When running on an isolated network use the **Offline Update Kit (OUK)**:
|
||||
|
||||
```bash
|
||||
# Download & verify on a connected host
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-v0.1a.tgz.sig
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-v0.1a.tgz.sig \
|
||||
stella-ops-offline-kit-v0.1a.tgz
|
||||
|
||||
# Transfer → air‑gap → import
|
||||
docker compose --env-file .env -f docker-compose.stella-ops.yml \
|
||||
exec stella admin import-offline-usage-kit stella-ops-offline-kit-v0.1a.tgz
|
||||
```
|
||||
|
||||
*Import is atomic; no service downtime.*
|
||||
|
||||
For details see the dedicated [Offline Kit guide](/offline/).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Next steps
|
||||
|
||||
* **5‑min Quick‑Start:** `/quickstart/`
|
||||
* **CI recipes:** `docs/ci/20_CI_RECIPES.md`
|
||||
* **Plug‑in SDK:** `/plugins/`
|
||||
|
||||
---
|
||||
|
||||
*Generated {{ "now" | date: "%Y‑%m‑%d" }} — build tags inserted at render time.*
|
||||
|
||||
@@ -1,286 +1,286 @@
|
||||
# Offline Update Kit (OUK) — Air‑Gap Bundle
|
||||
|
||||
<!--
|
||||
Build‑time variable injection:
|
||||
{{ quota_anon }} = 33
|
||||
{{ quota_token }} = 333
|
||||
{{ dotnet }} = "10 LTS"
|
||||
-->
|
||||
|
||||
The **Offline Update Kit** packages everything Stella Ops needs to run on a
|
||||
completely isolated network:
|
||||
|
||||
| Component | Contents |
|
||||
|-----------|----------|
|
||||
| **Merged vulnerability feeds** | OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU |
|
||||
| **Container images** | `stella-ops`, *Zastava* sidecar (x86‑64 & arm64) |
|
||||
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
|
||||
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
|
||||
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
| **Debug store** | `.debug` artefacts laid out under `debug/.build-id/<aa>/<rest>.debug` with `debug/debug-manifest.json` mapping build-ids to originating images for symbol retrieval. |
|
||||
| **Telemetry collector bundle** | `telemetry/telemetry-offline-bundle.tar.gz` plus `.sha256`, containing OTLP collector config, Helm/Compose overlays, and operator instructions. |
|
||||
|
||||
**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.
|
||||
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.
|
||||
|
||||
*Scanner core:* C# 12 on **.NET {{ dotnet }}**.
|
||||
*Imports are idempotent and atomic — no service downtime.*
|
||||
|
||||
## 0 · Prepare the debug store
|
||||
|
||||
Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id `.debug` files and the associated manifest) into the staging directory:
|
||||
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
|
||||
The helper copies `debug/.build-id/**`, validates `debug/debug-manifest.json` against its recorded SHA-256, and writes `out/offline-kit/metadata/debug-store.json` with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.
|
||||
|
||||
---
|
||||
|
||||
## 0.1 · Automated packaging
|
||||
|
||||
The packaging workflow is scripted via `ops/offline-kit/build_offline_kit.py`.
|
||||
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/build_offline_kit.py \
|
||||
--version 2025.10.0 \
|
||||
--channel edge \
|
||||
--release-dir out/release \
|
||||
--staging-dir out/offline-kit/staging \
|
||||
--output-dir out/offline-kit/dist
|
||||
|
||||
# Optional: regenerate the telemetry collector bundle prior to packaging.
|
||||
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz` — bundle (mtime/uid/gid forced to zero for reproducibility)
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256` — bundle digest
|
||||
- `manifest/offline-manifest.json` + `.sha256` — inventories every file in the bundle
|
||||
- `<bundle>.metadata.json` — descriptor consumed by the CLI/Console import tooling
|
||||
- `telemetry/telemetry-offline-bundle.tar.gz` + `.sha256` — packaged OTLP collector assets for environments without upstream access
|
||||
- `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig` (+ `.sha256`) — Cosign signatures for the Python analyzer DLL and manifest
|
||||
|
||||
### Policy Gateway configuration bundle
|
||||
|
||||
- Copy `etc/policy-gateway.yaml` (or the `*.sample` template if you expect operators to override values) into `config/policy-gateway/policy-gateway.yaml` within the staging tree.
|
||||
- Include the gateway DPoP private key under `secrets/policy-gateway/policy-gateway-dpop.pem` and reference the location inside the manifest notes. Set the permissions explicitly (`chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem`) so only the kit importer can read it; the importer will refuse keys that are broader.
|
||||
- Document the gateway base URL and activation verification steps in `docs/policy/gateway.md` (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
|
||||
- Ensure the Prometheus snapshot captured during packaging contains `policy_gateway_activation_requests_total` so auditors can reconcile activation attempts performed via the gateway during the validation window.
|
||||
|
||||
Provide `--cosign-key` / `--cosign-identity-token` (and optional `--cosign-password`) to generate Cosign signatures for both the tarball and manifest.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Download & verify
|
||||
|
||||
```bash
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
stella-ops-offline-kit-<DATE>.tgz
|
||||
````
|
||||
|
||||
**CLI shortcut.** `stellaops-cli offline kit pull --destination ./offline-kit` downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a `.metadata.json` summary for later import.
|
||||
|
||||
Verification prints **OK** and the SHA‑256 digest; cross‑check against the
|
||||
[changelog](https://git.stella-ops.org/stella-ops/offline-kit/-/releases).
|
||||
|
||||
Validate the attested manifest before distribution:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature offline-manifest-<DATE>.json.jws \
|
||||
offline-manifest-<DATE>.json
|
||||
|
||||
jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json
|
||||
```
|
||||
|
||||
The manifest enumerates every artefact (`name`, `sha256`, `size`, `capturedAt`) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.
|
||||
|
||||
Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
|
||||
"sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
|
||||
"size": 33792,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
|
||||
"sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
|
||||
"size": 32152,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
|
||||
"sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
|
||||
"size": 622,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
|
||||
"sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
|
||||
"size": 70144,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
|
||||
"sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
|
||||
"size": 40400,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
|
||||
"sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
|
||||
"size": 647,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
|
||||
"sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
|
||||
"size": 47104,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
|
||||
"sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
|
||||
"size": 31896,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
|
||||
"sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
|
||||
"size": 648,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Import on the air‑gapped host
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env \
|
||||
-f docker-compose.stella-ops.yml \
|
||||
exec stella-ops \
|
||||
stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz
|
||||
```
|
||||
|
||||
Alternatively, run
|
||||
|
||||
```bash
|
||||
stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
|
||||
--manifest offline-manifest-<DATE>.json \
|
||||
--bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
--manifest-signature offline-manifest-<DATE>.json.jws
|
||||
```
|
||||
|
||||
The CLI validates recorded digests (when `.metadata.json` is present) before streaming the multipart payload to `/api/offline-kit/import`.
|
||||
|
||||
* The CLI validates the Cosign signature **before** activation.
|
||||
* Old feeds are kept until the new bundle is fully verified.
|
||||
* Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.
|
||||
|
||||
### 2.1 Validator + idempotency enablement (air-gap)
|
||||
|
||||
The Offline Kit carries the same helper scripts under `scripts/`:
|
||||
|
||||
1. **Duplicate audit:** run
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
to verify no `(vendor, upstream_id, content_hash, tenant)` conflicts remain before enabling the idempotency index.
|
||||
2. **Apply validators:** execute `mongo concelier ops/devops/scripts/apply-aoc-validators.js` (and the Excititor equivalent) with `validationLevel: "moderate"` in maintenance mode.
|
||||
3. **Restart Concelier** so migrations `20251028_advisory_raw_idempotency_index` and `20251028_advisory_supersedes_backfill` run automatically. After the restart:
|
||||
- Confirm `db.advisory` resolves to a view on `advisory_backup_20251028`.
|
||||
- Spot-check a few `advisory_raw` entries to ensure `supersedes` chains are populated deterministically.
|
||||
4. **Smoke test:** run `stella sources ingest --dry-run --fixture advisory` (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.
|
||||
|
||||
### Authority scope sanity check
|
||||
|
||||
Offline installs rely on the bundled `etc/authority.yaml.sample`. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:
|
||||
|
||||
- `aoc-verifier` requests `aoc:verify`, `advisory:read`, and `vex:read`.
|
||||
- `signals-uploader` requests `signals:write`, `signals:read`, and `aoc:verify`.
|
||||
|
||||
Authority now rejects tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope without `aoc:verify`; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with `invalid_scope` when misconfigured.
|
||||
|
||||
**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:
|
||||
|
||||
```bash
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
|
||||
```
|
||||
|
||||
The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.
|
||||
|
||||
> **Release guardrail.** The automated release pipeline now publishes the Python plug-in from source and executes `dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout>` to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run `ops/offline-kit/run-python-analyzer-smoke.sh` locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.
|
||||
|
||||
### Debug store mirror
|
||||
|
||||
Offline symbols (`debug/.build-id/**`) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit `out/release/debug/` containing the build-id tree plus `debug-manifest.json` and its `.sha256` companion. After a release completes:
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-dir out/offline-kit \
|
||||
--summary out/offline-kit/metadata/debug-store.json
|
||||
```
|
||||
|
||||
The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under `metadata/debug-store.json` for audit logs. If the release pipeline does not populate `out/release/debug`, the tooling now logs a warning (`DEVOPS-REL-17-004`)—treat it as a build failure and re-run the release once symbol extraction is enabled.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Delta patch workflow
|
||||
|
||||
1. **Connected site** fetches `stella-ouk-YYYY‑MM‑DD.delta.tgz`.
|
||||
2. Transfer via any medium (USB, portable disk).
|
||||
3. `stella admin import-offline-usage-kit <delta>` applies only changed CVE rows & images.
|
||||
|
||||
Daily deltas are **< 30 MB**; weekly roll‑up produces a fresh full kit.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Quota behaviour offline
|
||||
|
||||
The scanner enforces the same fair‑use limits offline:
|
||||
|
||||
* **Anonymous:** {{ quota\_anon }} scans per UTC day
|
||||
* **Free JWT:** {{ quota\_token }} scans per UTC day
|
||||
|
||||
Soft reminder at 200 scans; throttle above the ceiling but **never block**.
|
||||
See the detailed rules in
|
||||
[`33_333_QUOTA_OVERVIEW.md`](33_333_QUOTA_OVERVIEW.md).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting
|
||||
|
||||
| Symptom | Explanation | Fix |
|
||||
| -------------------------------------- | ---------------------------------------- | ------------------------------------- |
|
||||
| `could not verify SBOM hash` | Bundle corrupted in transit | Re‑download / re‑copy |
|
||||
| Import hangs at `Applying feeds…` | Low disk space in `/var/lib/stella` | Free ≥ 2 GiB before retry |
|
||||
| `quota exceeded` same day after import | Import resets counters at UTC 00:00 only | Wait until next UTC day or load a JWT |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Related documentation
|
||||
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **CERT-Bund snapshots:** `python src/Tools/certbund_offline_snapshot.py --help` (see `docs/modules/concelier/operations/connectors/certbund.md`)
|
||||
# Offline Update Kit (OUK) — Air‑Gap Bundle
|
||||
|
||||
<!--
|
||||
Build‑time variable injection:
|
||||
{{ quota_anon }} = 33
|
||||
{{ quota_token }} = 333
|
||||
{{ dotnet }} = "10 LTS"
|
||||
-->
|
||||
|
||||
The **Offline Update Kit** packages everything Stella Ops needs to run on a
|
||||
completely isolated network:
|
||||
|
||||
| Component | Contents |
|
||||
|-----------|----------|
|
||||
| **Merged vulnerability feeds** | OSV, GHSA plus optional NVD 2.0, CNNVD, CNVD, ENISA, JVN and BDU |
|
||||
| **Container images** | `stella-ops`, *Zastava* sidecar (x86‑64 & arm64) |
|
||||
| **Provenance** | Cosign signature, SPDX 2.3 SBOM, in‑toto SLSA attestation |
|
||||
| **Attested manifest** | `offline-manifest.json` + detached JWS covering bundle metadata, signed during export. |
|
||||
| **Delta patches** | Daily diff bundles keep size \< 350 MB |
|
||||
| **Scanner plug-ins** | OS analyzers plus the Node.js, Go, .NET, and Python language analyzers packaged under `plugins/scanner/analyzers/**` with manifests so Workers load deterministically offline. |
|
||||
| **Debug store** | `.debug` artefacts laid out under `debug/.build-id/<aa>/<rest>.debug` with `debug/debug-manifest.json` mapping build-ids to originating images for symbol retrieval. |
|
||||
| **Telemetry collector bundle** | `telemetry/telemetry-offline-bundle.tar.gz` plus `.sha256`, containing OTLP collector config, Helm/Compose overlays, and operator instructions. |
|
||||
|
||||
**RU BDU note:** ship the official Russian Trusted Root/Sub CA bundle (`certificates/russian_trusted_bundle.pem`) inside the kit so `concelier:httpClients:source.bdu:trustedRootPaths` can resolve it when the service runs in an air‑gapped network. Drop the most recent `vulxml.zip` alongside the kit if operators need a cold-start cache.
|
||||
|
||||
**Language analyzers:** the kit now carries the restart-only Node.js, Go, .NET, and Python analyzer plug-ins (`plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Node/`, `...Lang.Go/`, `...Lang.DotNet/`, `...Lang.Python/`). Drop the directories alongside Worker binaries so the unified plug-in catalog can load them without outbound fetches; Rust remains on the Wave 4 roadmap.
|
||||
|
||||
*Scanner core:* C# 12 on **.NET {{ dotnet }}**.
|
||||
*Imports are idempotent and atomic — no service downtime.*
|
||||
|
||||
## 0 · Prepare the debug store
|
||||
|
||||
Before packaging the Offline Kit, mirror the release debug artefacts (GNU build-id `.debug` files and the associated manifest) into the staging directory:
|
||||
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
|
||||
The helper copies `debug/.build-id/**`, validates `debug/debug-manifest.json` against its recorded SHA-256, and writes `out/offline-kit/metadata/debug-store.json` with a short summary (platforms, artefact counts, sample build-ids). The command exits non-zero if an artefact referenced by the manifest is missing or has the wrong digest, so run it as part of every kit build.
|
||||
|
||||
---
|
||||
|
||||
## 0.1 · Automated packaging
|
||||
|
||||
The packaging workflow is scripted via `ops/offline-kit/build_offline_kit.py`.
|
||||
It verifies the release artefacts, runs the Python analyzer smoke suite, mirrors the debug store, and emits a deterministic tarball + manifest set.
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/build_offline_kit.py \
|
||||
--version 2025.10.0 \
|
||||
--channel edge \
|
||||
--release-dir out/release \
|
||||
--staging-dir out/offline-kit/staging \
|
||||
--output-dir out/offline-kit/dist
|
||||
|
||||
# Optional: regenerate the telemetry collector bundle prior to packaging.
|
||||
python ops/devops/telemetry/package_offline_bundle.py --output out/telemetry/telemetry-offline-bundle.tar.gz
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz` — bundle (mtime/uid/gid forced to zero for reproducibility)
|
||||
- `stella-ops-offline-kit-<version>-<channel>.tar.gz.sha256` — bundle digest
|
||||
- `manifest/offline-manifest.json` + `.sha256` — inventories every file in the bundle
|
||||
- `<bundle>.metadata.json` — descriptor consumed by the CLI/Console import tooling
|
||||
- `telemetry/telemetry-offline-bundle.tar.gz` + `.sha256` — packaged OTLP collector assets for environments without upstream access
|
||||
- `plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*.sig` (+ `.sha256`) — Cosign signatures for the Python analyzer DLL and manifest
|
||||
|
||||
### Policy Gateway configuration bundle
|
||||
|
||||
- Copy `etc/policy-gateway.yaml` (or the `*.sample` template if you expect operators to override values) into `config/policy-gateway/policy-gateway.yaml` within the staging tree.
|
||||
- Include the gateway DPoP private key under `secrets/policy-gateway/policy-gateway-dpop.pem` and reference the location inside the manifest notes. Set the permissions explicitly (`chmod 600 secrets/policy-gateway/policy-gateway-dpop.pem`) so only the kit importer can read it; the importer will refuse keys that are broader.
|
||||
- Document the gateway base URL and activation verification steps in `docs/policy/gateway.md` (bundled alongside the kit). Operators can use those curl snippets to smoke-test pack CRUD once the Offline Kit is imported.
|
||||
- Ensure the Prometheus snapshot captured during packaging contains `policy_gateway_activation_requests_total` so auditors can reconcile activation attempts performed via the gateway during the validation window.
|
||||
|
||||
Provide `--cosign-key` / `--cosign-identity-token` (and optional `--cosign-password`) to generate Cosign signatures for both the tarball and manifest.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Download & verify
|
||||
|
||||
```bash
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz
|
||||
curl -LO https://get.stella-ops.org/ouk/stella-ops-offline-kit-<DATE>.tgz.sig
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json
|
||||
curl -LO https://get.stella-ops.org/ouk/offline-manifest-<DATE>.json.jws
|
||||
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
stella-ops-offline-kit-<DATE>.tgz
|
||||
````
|
||||
|
||||
**CLI shortcut.** `stellaops-cli offline kit pull --destination ./offline-kit` downloads the bundle, manifest, and detached signatures in one step, resumes partial transfers, and writes a `.metadata.json` summary for later import.
|
||||
|
||||
Verification prints **OK** and the SHA‑256 digest; cross‑check against the
|
||||
[changelog](https://git.stella-ops.org/stella-ops/offline-kit/-/releases).
|
||||
|
||||
Validate the attested manifest before distribution:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--key https://stella-ops.org/keys/cosign.pub \
|
||||
--signature offline-manifest-<DATE>.json.jws \
|
||||
offline-manifest-<DATE>.json
|
||||
|
||||
jq '.artifacts[] | {name, sha256, size, capturedAt}' offline-manifest-<DATE>.json
|
||||
```
|
||||
|
||||
The manifest enumerates every artefact (`name`, `sha256`, `size`, `capturedAt`) and is signed with the same key registry as Authority revocation bundles. Operators can ship the manifest alongside the tarball so downstream mirrors can re-verify without unpacking the kit.
|
||||
|
||||
Example excerpt (2025-10-23 kit) showing the Go and .NET analyzer plug-in payloads:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.dll",
|
||||
"sha256": "a6dc850fc51151c8967ef46a3c4730f08b549667e041079431f39a8a72d0b641",
|
||||
"size": 33792,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/StellaOps.Scanner.Analyzers.Lang.Go.pdb",
|
||||
"sha256": "6cbdabf155282f458b89edf267e7f6bb2441a93029aad7aad45c8a9ec58b1b3b",
|
||||
"size": 32152,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json",
|
||||
"sha256": "c19bfca2fcbb7cb18f1082b5d0d5a8f15fc799c648b50e95fce8d8b109ce48c9",
|
||||
"size": 622,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.dll",
|
||||
"sha256": "0734d23e33277ce2ccb596782d2d42cfe394b3d372dc34da9cb28b59df9b9d22",
|
||||
"size": 70144,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/StellaOps.Scanner.Analyzers.Lang.DotNet.pdb",
|
||||
"sha256": "b853c1ff4b196715f5bd1447e1a13edeb4940917527ec9bf153b5048da49abaf",
|
||||
"size": 40400,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/manifest.json",
|
||||
"sha256": "5d483885f825f01bfd9943dcf2889ec2e0beba38ede92ecfe67d4f506cf14e37",
|
||||
"size": 647,
|
||||
"capturedAt": "2025-10-23T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.dll",
|
||||
"sha256": "a4f558f363394096e3dd6263f35b180b93b4112f9cf616c05872da8a8657d518",
|
||||
"size": 47104,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/StellaOps.Scanner.Analyzers.Lang.Python.pdb",
|
||||
"sha256": "ef2ad78bc2cd1d7e99bae000b92357aa9a9c32938501899e9033d001096196d0",
|
||||
"size": 31896,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
{
|
||||
"name": "plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/manifest.json",
|
||||
"sha256": "668ad9a1a35485628677b639db4d996d1e25f62021680a81a22482483800e557",
|
||||
"size": 648,
|
||||
"capturedAt": "2025-10-26T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Import on the air‑gapped host
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env \
|
||||
-f docker-compose.stella-ops.yml \
|
||||
exec stella-ops \
|
||||
stella admin import-offline-usage-kit stella-ops-offline-kit-<DATE>.tgz
|
||||
```
|
||||
|
||||
Alternatively, run
|
||||
|
||||
```bash
|
||||
stellaops-cli offline kit import stella-ops-offline-kit-<DATE>.tgz \
|
||||
--manifest offline-manifest-<DATE>.json \
|
||||
--bundle-signature stella-ops-offline-kit-<DATE>.tgz.sig \
|
||||
--manifest-signature offline-manifest-<DATE>.json.jws
|
||||
```
|
||||
|
||||
The CLI validates recorded digests (when `.metadata.json` is present) before streaming the multipart payload to `/api/offline-kit/import`.
|
||||
|
||||
* The CLI validates the Cosign signature **before** activation.
|
||||
* Old feeds are kept until the new bundle is fully verified.
|
||||
* Import time on a SATA SSD: ≈ 25 s for a 300 MB kit.
|
||||
|
||||
### 2.1 Validator + idempotency enablement (air-gap)
|
||||
|
||||
The Offline Kit carries the same helper scripts under `scripts/`:
|
||||
|
||||
1. **Duplicate audit:** run
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
to verify no `(vendor, upstream_id, content_hash, tenant)` conflicts remain before enabling the idempotency index.
|
||||
2. **Apply validators:** execute `mongo concelier ops/devops/scripts/apply-aoc-validators.js` (and the Excititor equivalent) with `validationLevel: "moderate"` in maintenance mode.
|
||||
3. **Restart Concelier** so migrations `20251028_advisory_raw_idempotency_index` and `20251028_advisory_supersedes_backfill` run automatically. After the restart:
|
||||
- Confirm `db.advisory` resolves to a view on `advisory_backup_20251028`.
|
||||
- Spot-check a few `advisory_raw` entries to ensure `supersedes` chains are populated deterministically.
|
||||
4. **Smoke test:** run `stella sources ingest --dry-run --fixture advisory` (bundled fixtures) to confirm ingestion succeeds post-guard and the CLI reports zero violations.
|
||||
|
||||
### Authority scope sanity check
|
||||
|
||||
Offline installs rely on the bundled `etc/authority.yaml.sample`. Before promoting the kit, confirm the sample clients keep the Aggregation-Only guardrails:
|
||||
|
||||
- `aoc-verifier` requests `aoc:verify`, `advisory:read`, and `vex:read`.
|
||||
- `signals-uploader` requests `signals:write`, `signals:read`, and `aoc:verify`.
|
||||
|
||||
Authority now rejects tokens that request `advisory:read`, `vex:read`, or any `signals:*` scope without `aoc:verify`; the sample has been updated to match. If you maintain tenant-specific overlays, mirror the same pairing so air-gapped automation fails deterministically with `invalid_scope` when misconfigured.
|
||||
|
||||
**Quick smoke test:** before import, verify the tarball carries the Go analyzer plug-in:
|
||||
|
||||
```bash
|
||||
tar -tzf stella-ops-offline-kit-<DATE>.tgz 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.DotNet/*' 'plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Python/*'
|
||||
```
|
||||
|
||||
The manifest lookup above and this `tar` listing should both surface the Go analyzer DLL, PDB, and manifest entries before the kit is promoted.
|
||||
|
||||
> **Release guardrail.** The automated release pipeline now publishes the Python plug-in from source and executes `dotnet run --project src/Tools/LanguageAnalyzerSmoke --configuration Release -- --repo-root <checkout>` to validate manifest integrity and cold/warm determinism within the < 30 s / < 5 s budgets (differences versus repository goldens are logged for triage). Run `ops/offline-kit/run-python-analyzer-smoke.sh` locally before shipping a refreshed kit if you rebuild artefacts outside CI or when preparing the air-gap bundle.
|
||||
|
||||
### Debug store mirror
|
||||
|
||||
Offline symbols (`debug/.build-id/**`) must accompany every Offline Kit to keep symbol lookup deterministic. The release workflow is expected to emit `out/release/debug/` containing the build-id tree plus `debug-manifest.json` and its `.sha256` companion. After a release completes:
|
||||
|
||||
```bash
|
||||
python ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-dir out/offline-kit \
|
||||
--summary out/offline-kit/metadata/debug-store.json
|
||||
```
|
||||
|
||||
The script mirrors the debug tree into the Offline Kit staging directory, verifies SHA-256 values against the manifest, and writes a summary under `metadata/debug-store.json` for audit logs. If the release pipeline does not populate `out/release/debug`, the tooling now logs a warning (`DEVOPS-REL-17-004`)—treat it as a build failure and re-run the release once symbol extraction is enabled.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Delta patch workflow
|
||||
|
||||
1. **Connected site** fetches `stella-ouk-YYYY‑MM‑DD.delta.tgz`.
|
||||
2. Transfer via any medium (USB, portable disk).
|
||||
3. `stella admin import-offline-usage-kit <delta>` applies only changed CVE rows & images.
|
||||
|
||||
Daily deltas are **< 30 MB**; weekly roll‑up produces a fresh full kit.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Quota behaviour offline
|
||||
|
||||
The scanner enforces the same fair‑use limits offline:
|
||||
|
||||
* **Anonymous:** {{ quota\_anon }} scans per UTC day
|
||||
* **Free JWT:** {{ quota\_token }} scans per UTC day
|
||||
|
||||
Soft reminder at 200 scans; throttle above the ceiling but **never block**.
|
||||
See the detailed rules in
|
||||
[`33_333_QUOTA_OVERVIEW.md`](33_333_QUOTA_OVERVIEW.md).
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting
|
||||
|
||||
| Symptom | Explanation | Fix |
|
||||
| -------------------------------------- | ---------------------------------------- | ------------------------------------- |
|
||||
| `could not verify SBOM hash` | Bundle corrupted in transit | Re‑download / re‑copy |
|
||||
| Import hangs at `Applying feeds…` | Low disk space in `/var/lib/stella` | Free ≥ 2 GiB before retry |
|
||||
| `quota exceeded` same day after import | Import resets counters at UTC 00:00 only | Wait until next UTC day or load a JWT |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Related documentation
|
||||
|
||||
* **Install guide:** `/install/#air-gapped`
|
||||
* **Sovereign mode rationale:** `/sovereign/`
|
||||
* **Security policy:** `/security/#reporting-a-vulnerability`
|
||||
* **CERT-Bund snapshots:** `python src/Tools/certbund_offline_snapshot.py --help` (see `docs/modules/concelier/operations/connectors/certbund.md`)
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
# Docs & Enablement Guild
|
||||
|
||||
## Mission
|
||||
Produce and maintain offline-friendly documentation for StellaOps modules, covering architecture, configuration, operator workflows, and developer onboarding.
|
||||
|
||||
## Scope Highlights
|
||||
- Authority docs (`docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, upcoming `docs/11_AUTHORITY.md`).
|
||||
- Concelier quickstarts, CLI guides, Offline Kit manuals.
|
||||
- Release notes and migration playbooks.
|
||||
|
||||
## Operating Principles
|
||||
- Keep guides deterministic and in sync with shipped configuration samples.
|
||||
- Prefer tables/checklists for operator steps; flag security-sensitive actions.
|
||||
- When work involves a specific `StellaOps.<Component>` project, consult both `docs/07_HIGH_LEVEL_ARCHITECTURE.md` and the matching dossier `docs/modules/<component>/architecture.md` before drafting or editing content.
|
||||
- Update `docs/TASKS.md` whenever work items change status (TODO/DOING/REVIEW/DONE/BLOCKED).
|
||||
|
||||
## Coordination
|
||||
- Authority Core & Plugin teams for auth-related changes.
|
||||
- Security Guild for threat-model outputs and mitigations.
|
||||
- DevEx for tooling diagrams and documentation pipeline.
|
||||
# Docs & Enablement Guild
|
||||
|
||||
## Mission
|
||||
Produce and maintain offline-friendly documentation for StellaOps modules, covering architecture, configuration, operator workflows, and developer onboarding.
|
||||
|
||||
## Scope Highlights
|
||||
- Authority docs (`docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`, upcoming `docs/11_AUTHORITY.md`).
|
||||
- Concelier quickstarts, CLI guides, Offline Kit manuals.
|
||||
- Release notes and migration playbooks.
|
||||
|
||||
## Operating Principles
|
||||
- Keep guides deterministic and in sync with shipped configuration samples.
|
||||
- Prefer tables/checklists for operator steps; flag security-sensitive actions.
|
||||
- When work involves a specific `StellaOps.<Component>` project, consult both `docs/07_HIGH_LEVEL_ARCHITECTURE.md` and the matching dossier `docs/modules/<component>/architecture.md` before drafting or editing content.
|
||||
- Update `docs/TASKS.md` whenever work items change status (TODO/DOING/REVIEW/DONE/BLOCKED).
|
||||
|
||||
## Coordination
|
||||
- Authority Core & Plugin teams for auth-related changes.
|
||||
- Security Guild for threat-model outputs and mitigations.
|
||||
- DevEx for tooling diagrams and documentation pipeline.
|
||||
|
||||
@@ -1,131 +1,131 @@
|
||||
# StellaOps Console Accessibility Guide
|
||||
|
||||
> **Audience:** Accessibility Guild, Console Guild, Docs Guild, QA.
|
||||
> **Scope:** Keyboard interaction model, screen-reader behaviour, colour & focus tokens, testing workflows, offline considerations, and compliance checklist for the StellaOps Console (Sprint 23).
|
||||
|
||||
The console targets **WCAG 2.2 AA** across all supported browsers (Chromium, Firefox ESR) and honours StellaOps’ sovereign/offline constraints. Every build must keep keyboard-only users, screen-reader users, and high-contrast operators productive without relying on third-party services.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Accessibility Principles
|
||||
|
||||
1. **Deterministic navigation** – Focus order, shortcuts, and announcements remain stable across releases; URLs encode state for deep links.
|
||||
2. **Keyboard-first design** – Every actionable element is reachable via keyboard; shortcuts provide accelerators, and remapping is available via *Settings → Accessibility → Keyboard shortcuts*.
|
||||
3. **Assistive technology parity** – ARIA roles and live regions mirror visual affordances (status banners, SSE tickers, progress drawers). Screen readers receive polite/atomic updates to avoid chatter.
|
||||
4. **Colour & contrast tokens** – All palettes derive from design tokens that achieve ≥ 4.5:1 contrast (text) and ≥ 3:1 for graphical indicators; tokens pass automated contrast linting.
|
||||
5. **Offline equivalence** – Accessibility features (shortcuts, offline banners, focus restoration) behave the same in sealed environments, with guidance when actions require online authority.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Keyboard Interaction Map
|
||||
|
||||
### 2.1 Global shortcuts
|
||||
|
||||
| Action | Macs | Windows/Linux | Notes |
|
||||
|--------|------|---------------|-------|
|
||||
| Command palette | `⌘ K` | `Ctrl K` | Focuses palette search; respects tenant scope. |
|
||||
| Tenant picker | `⌘ T` | `Ctrl T` | Opens modal; `Enter` confirms, `Esc` cancels. |
|
||||
| Filter tray toggle | `⇧ F` | `Shift F` | Focus lands on first filter; `Tab` cycles filters before returning to page. |
|
||||
| Saved view presets | `⌘ 1-9` | `Ctrl 1-9` | Bound per tenant; missing preset triggers tooltip. |
|
||||
| Keyboard reference | `?` | `?` | Opens overlay listing context-specific shortcuts; `Esc` closes. |
|
||||
| Global search (context) | `/` | `/` | When the filter tray is closed, focuses inline search field. |
|
||||
|
||||
### 2.2 Module-specific shortcuts
|
||||
|
||||
| Module | Action | Macs | Windows/Linux | Notes |
|
||||
|--------|--------|------|---------------|-------|
|
||||
| Findings | Explain search | `⌘ /` | `Ctrl /` | Only when Explain drawer open; announces results via live region. |
|
||||
| SBOM Explorer | Toggle overlays | `⌘ G` | `Ctrl G` | Persists per session (see `/docs/ui/sbom-explorer.md`). |
|
||||
| Advisories & VEX | Provider filter | `⌘ ⌥ F` | `Ctrl Alt F` | Moves focus to provider chip row. |
|
||||
| Runs | Refresh snapshot | `⌘ R` | `Ctrl R` | Soft refresh of SSE state; no full page reload. |
|
||||
| Policies | Save draft | `⌘ S` | `Ctrl S` | Requires edit scope; exposes toast + status live update. |
|
||||
| Downloads | Copy CLI command | `⇧ D` | `Shift D` | Copies manifest or export command; toast announces scope hints. |
|
||||
|
||||
All shortcuts are remappable. Remaps persist in IndexedDB (per tenant) and export as part of profile bundles so operators can restore preferences offline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Screen Reader & Focus Behaviour
|
||||
|
||||
- **Skip navigation** – Each route exposes a “Skip to content” link revealed on keyboard focus. Focus order: global header → page breadcrumb → action shelf → data grid/list → drawers/dialogs.
|
||||
- **Live regions** – Status ticker and SSE progress bars use `aria-live="polite"` with throttling to avoid flooding AT. Error toasts use `aria-live="assertive"` and auto-focus dismiss buttons.
|
||||
- **Drawers & modals** – Dialog components trap focus, support `Esc` to close, and restore focus to the launching control. Screen readers announce title + purpose.
|
||||
- **Tables & grids** – Large tables (Findings, SBOM inventory) switch to virtualised rows but retain ARIA grid semantics (`aria-rowcount`, `aria-colindex`). Column headers include sorting state via `aria-sort`.
|
||||
- **Tenancy context** – Tenant badge exposes `aria-describedby` linking to context summary (environment, offline snapshot). Switching tenant queues a polite announcement summarising new scope.
|
||||
- **Command palette** – Uses `role="dialog"` with search input labelled. Keyboard navigation within results uses `Up/Down`; screen readers announce result category + command.
|
||||
- **Offline banner** – When offline, a dismissible banner announces reason and includes instructions for CLI fallback. The banner has `role="status"` so it announces once without stealing focus.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Colour & Focus Tokens
|
||||
|
||||
Console consumes design tokens published by the Console Guild (tracked via CONSOLE-FEAT-23-102). Tokens live in the design system bundle (`ui/design/tokens/colors.json`, mirrored at build time). Key tokens:
|
||||
|
||||
| Token | Purpose | Contrast target |
|
||||
|-------|---------|-----------------|
|
||||
| `so-color-surface-base` | Primary surface/background | ≥ 4.5:1 against `so-color-text-primary`. |
|
||||
| `so-color-surface-raised` | Cards, drawers, modals | ≥ 3:1 against surrounding surfaces. |
|
||||
| `so-color-text-primary` | Default text colour | ≥ 4.5:1 against base surfaces. |
|
||||
| `so-color-text-inverted` | Text on accent buttons | ≥ 4.5:1 against accent fills. |
|
||||
| `so-color-accent-primary` | Action buttons, focus headings | ≥ 3:1 against surface. |
|
||||
| `so-color-status-critical` | Error toasts, violation chips | ≥ 4.5:1 for text; `critical-bg` provides >3:1 on neutral surface. |
|
||||
| `so-color-status-warning` | Warning banners | Meets 3:1 on surface and 4.5:1 for text overlays. |
|
||||
| `so-color-status-success` | Success toasts, pass badges | ≥ 3:1 for iconography; text uses `text-primary`. |
|
||||
| `so-focus-ring` | 2 px outline used across focusable elements | 3:1 against both light/dark surfaces. |
|
||||
|
||||
Colour tokens undergo automated linting (**axe-core contrast checks** + custom luminance script) during build. Any new token must include dark/light variants and pass the token contract tests.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing Workflow
|
||||
|
||||
| Layer | Tooling | Frequency | Notes |
|
||||
|-------|---------|-----------|-------|
|
||||
| Component a11y | Storybook + axe-core addon | On PR (story CI) | Fails when axe detects violations. |
|
||||
| Route regression | Playwright a11y sweep (`pnpm test:a11y`) | Nightly & release pipeline | Executes keyboard navigation, checks focus trap, runs Axe on key routes (Dashboard, Findings, SBOM, Admin). |
|
||||
| Colour contrast lint | Token validator (`src/Tools/a11y/check-contrast.ts`) | On token change | Guards design token updates. |
|
||||
| CI parity | Pending `scripts/check-console-cli-parity.sh` (CONSOLE-DOC-23-502) | Release CI | Ensures CLI commands documented for parity features. |
|
||||
| Screen-reader spot checks | Manual NVDA + VoiceOver scripts | Pre-release checklist | Scenarios: tenant switch, explain drawer, downloads parity copy. |
|
||||
| Offline smoke | `stella offline kit import` + Playwright sealed-mode run | Prior to Offline Kit cut | Validates offline banners, disabled actions, keyboard flows without Authority. |
|
||||
|
||||
Accessibility QA (CONSOLE-QA-23-402) tracks failing scenarios via Playwright snapshots and publishes reports in the Downloads parity channel (`kind = "parity.report"` placeholder until CLI parity CI lands).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Offline & Internationalisation Considerations
|
||||
|
||||
- Offline mode surfaces staleness badges and disables remote-only palette entries; keyboard focus skips disabled controls.
|
||||
- Saved shortcuts, presets, and remaps serialise into Offline Kit bundles so operators can restore preferences post-import.
|
||||
- Locale switching (future feature flag) will load translations at runtime; ensure ARIA labels use i18n tokens rather than hard-coded strings.
|
||||
- For sealed installs, guidance panels include CLI equivalents (`stella auth fresh-auth`, `stella runs export`) to unblock tasks when Authority is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Compliance Checklist
|
||||
|
||||
- [ ] Keyboard shortcut matrix validated (default + remapped) and documented.
|
||||
- [ ] Screen-reader pass recorded for tenant switch, Explain drawer, Downloads copy-to-clipboard.
|
||||
- [ ] Colour tokens audited; contrast reports stored with release artifacts.
|
||||
- [ ] Automated a11y pipelines (Storybook axe, Playwright a11y) green; failures feed the `#console-qa` channel.
|
||||
- [ ] Offline kit a11y smoke executed before publishing each bundle.
|
||||
- [ ] CLI parity gaps logged in `/docs/cli-vs-ui-parity.md`; UI callouts reference fallback commands until parity closes.
|
||||
- [ ] Accessibility Guild sign-off captured in sprint log and release notes reference this guide.
|
||||
- [ ] References cross-checked (`/docs/ui/navigation.md`, `/docs/ui/downloads.md`, `/docs/security/console-security.md`, `/docs/observability/ui-telemetry.md`).
|
||||
|
||||
---
|
||||
|
||||
## 8 · References
|
||||
|
||||
- `/docs/ui/navigation.md` – shortcut definitions, URL schema.
|
||||
- `/docs/ui/downloads.md` – CLI parity and offline copy workflows.
|
||||
- `/docs/ui/console-overview.md` – tenant model, filter behaviours.
|
||||
- `/docs/security/console-security.md` – security metrics and DPoP/fresh-auth requirements.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics mapped to accessibility features.
|
||||
- `/docs/cli-vs-ui-parity.md` – parity status per console feature.
|
||||
- `CONSOLE-QA-23-402` – Accessibility QA backlog (Playwright + manual checks).
|
||||
- `CONSOLE-FEAT-23-102` – Design tokens & theming delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
# StellaOps Console Accessibility Guide
|
||||
|
||||
> **Audience:** Accessibility Guild, Console Guild, Docs Guild, QA.
|
||||
> **Scope:** Keyboard interaction model, screen-reader behaviour, colour & focus tokens, testing workflows, offline considerations, and compliance checklist for the StellaOps Console (Sprint 23).
|
||||
|
||||
The console targets **WCAG 2.2 AA** across all supported browsers (Chromium, Firefox ESR) and honours StellaOps’ sovereign/offline constraints. Every build must keep keyboard-only users, screen-reader users, and high-contrast operators productive without relying on third-party services.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Accessibility Principles
|
||||
|
||||
1. **Deterministic navigation** – Focus order, shortcuts, and announcements remain stable across releases; URLs encode state for deep links.
|
||||
2. **Keyboard-first design** – Every actionable element is reachable via keyboard; shortcuts provide accelerators, and remapping is available via *Settings → Accessibility → Keyboard shortcuts*.
|
||||
3. **Assistive technology parity** – ARIA roles and live regions mirror visual affordances (status banners, SSE tickers, progress drawers). Screen readers receive polite/atomic updates to avoid chatter.
|
||||
4. **Colour & contrast tokens** – All palettes derive from design tokens that achieve ≥ 4.5:1 contrast (text) and ≥ 3:1 for graphical indicators; tokens pass automated contrast linting.
|
||||
5. **Offline equivalence** – Accessibility features (shortcuts, offline banners, focus restoration) behave the same in sealed environments, with guidance when actions require online authority.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Keyboard Interaction Map
|
||||
|
||||
### 2.1 Global shortcuts
|
||||
|
||||
| Action | Macs | Windows/Linux | Notes |
|
||||
|--------|------|---------------|-------|
|
||||
| Command palette | `⌘ K` | `Ctrl K` | Focuses palette search; respects tenant scope. |
|
||||
| Tenant picker | `⌘ T` | `Ctrl T` | Opens modal; `Enter` confirms, `Esc` cancels. |
|
||||
| Filter tray toggle | `⇧ F` | `Shift F` | Focus lands on first filter; `Tab` cycles filters before returning to page. |
|
||||
| Saved view presets | `⌘ 1-9` | `Ctrl 1-9` | Bound per tenant; missing preset triggers tooltip. |
|
||||
| Keyboard reference | `?` | `?` | Opens overlay listing context-specific shortcuts; `Esc` closes. |
|
||||
| Global search (context) | `/` | `/` | When the filter tray is closed, focuses inline search field. |
|
||||
|
||||
### 2.2 Module-specific shortcuts
|
||||
|
||||
| Module | Action | Macs | Windows/Linux | Notes |
|
||||
|--------|--------|------|---------------|-------|
|
||||
| Findings | Explain search | `⌘ /` | `Ctrl /` | Only when Explain drawer open; announces results via live region. |
|
||||
| SBOM Explorer | Toggle overlays | `⌘ G` | `Ctrl G` | Persists per session (see `/docs/ui/sbom-explorer.md`). |
|
||||
| Advisories & VEX | Provider filter | `⌘ ⌥ F` | `Ctrl Alt F` | Moves focus to provider chip row. |
|
||||
| Runs | Refresh snapshot | `⌘ R` | `Ctrl R` | Soft refresh of SSE state; no full page reload. |
|
||||
| Policies | Save draft | `⌘ S` | `Ctrl S` | Requires edit scope; exposes toast + status live update. |
|
||||
| Downloads | Copy CLI command | `⇧ D` | `Shift D` | Copies manifest or export command; toast announces scope hints. |
|
||||
|
||||
All shortcuts are remappable. Remaps persist in IndexedDB (per tenant) and export as part of profile bundles so operators can restore preferences offline.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Screen Reader & Focus Behaviour
|
||||
|
||||
- **Skip navigation** – Each route exposes a “Skip to content” link revealed on keyboard focus. Focus order: global header → page breadcrumb → action shelf → data grid/list → drawers/dialogs.
|
||||
- **Live regions** – Status ticker and SSE progress bars use `aria-live="polite"` with throttling to avoid flooding AT. Error toasts use `aria-live="assertive"` and auto-focus dismiss buttons.
|
||||
- **Drawers & modals** – Dialog components trap focus, support `Esc` to close, and restore focus to the launching control. Screen readers announce title + purpose.
|
||||
- **Tables & grids** – Large tables (Findings, SBOM inventory) switch to virtualised rows but retain ARIA grid semantics (`aria-rowcount`, `aria-colindex`). Column headers include sorting state via `aria-sort`.
|
||||
- **Tenancy context** – Tenant badge exposes `aria-describedby` linking to context summary (environment, offline snapshot). Switching tenant queues a polite announcement summarising new scope.
|
||||
- **Command palette** – Uses `role="dialog"` with search input labelled. Keyboard navigation within results uses `Up/Down`; screen readers announce result category + command.
|
||||
- **Offline banner** – When offline, a dismissible banner announces reason and includes instructions for CLI fallback. The banner has `role="status"` so it announces once without stealing focus.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Colour & Focus Tokens
|
||||
|
||||
Console consumes design tokens published by the Console Guild (tracked via CONSOLE-FEAT-23-102). Tokens live in the design system bundle (`ui/design/tokens/colors.json`, mirrored at build time). Key tokens:
|
||||
|
||||
| Token | Purpose | Contrast target |
|
||||
|-------|---------|-----------------|
|
||||
| `so-color-surface-base` | Primary surface/background | ≥ 4.5:1 against `so-color-text-primary`. |
|
||||
| `so-color-surface-raised` | Cards, drawers, modals | ≥ 3:1 against surrounding surfaces. |
|
||||
| `so-color-text-primary` | Default text colour | ≥ 4.5:1 against base surfaces. |
|
||||
| `so-color-text-inverted` | Text on accent buttons | ≥ 4.5:1 against accent fills. |
|
||||
| `so-color-accent-primary` | Action buttons, focus headings | ≥ 3:1 against surface. |
|
||||
| `so-color-status-critical` | Error toasts, violation chips | ≥ 4.5:1 for text; `critical-bg` provides >3:1 on neutral surface. |
|
||||
| `so-color-status-warning` | Warning banners | Meets 3:1 on surface and 4.5:1 for text overlays. |
|
||||
| `so-color-status-success` | Success toasts, pass badges | ≥ 3:1 for iconography; text uses `text-primary`. |
|
||||
| `so-focus-ring` | 2 px outline used across focusable elements | 3:1 against both light/dark surfaces. |
|
||||
|
||||
Colour tokens undergo automated linting (**axe-core contrast checks** + custom luminance script) during build. Any new token must include dark/light variants and pass the token contract tests.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Testing Workflow
|
||||
|
||||
| Layer | Tooling | Frequency | Notes |
|
||||
|-------|---------|-----------|-------|
|
||||
| Component a11y | Storybook + axe-core addon | On PR (story CI) | Fails when axe detects violations. |
|
||||
| Route regression | Playwright a11y sweep (`pnpm test:a11y`) | Nightly & release pipeline | Executes keyboard navigation, checks focus trap, runs Axe on key routes (Dashboard, Findings, SBOM, Admin). |
|
||||
| Colour contrast lint | Token validator (`src/Tools/a11y/check-contrast.ts`) | On token change | Guards design token updates. |
|
||||
| CI parity | Pending `scripts/check-console-cli-parity.sh` (CONSOLE-DOC-23-502) | Release CI | Ensures CLI commands documented for parity features. |
|
||||
| Screen-reader spot checks | Manual NVDA + VoiceOver scripts | Pre-release checklist | Scenarios: tenant switch, explain drawer, downloads parity copy. |
|
||||
| Offline smoke | `stella offline kit import` + Playwright sealed-mode run | Prior to Offline Kit cut | Validates offline banners, disabled actions, keyboard flows without Authority. |
|
||||
|
||||
Accessibility QA (CONSOLE-QA-23-402) tracks failing scenarios via Playwright snapshots and publishes reports in the Downloads parity channel (`kind = "parity.report"` placeholder until CLI parity CI lands).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Offline & Internationalisation Considerations
|
||||
|
||||
- Offline mode surfaces staleness badges and disables remote-only palette entries; keyboard focus skips disabled controls.
|
||||
- Saved shortcuts, presets, and remaps serialise into Offline Kit bundles so operators can restore preferences post-import.
|
||||
- Locale switching (future feature flag) will load translations at runtime; ensure ARIA labels use i18n tokens rather than hard-coded strings.
|
||||
- For sealed installs, guidance panels include CLI equivalents (`stella auth fresh-auth`, `stella runs export`) to unblock tasks when Authority is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Compliance Checklist
|
||||
|
||||
- [ ] Keyboard shortcut matrix validated (default + remapped) and documented.
|
||||
- [ ] Screen-reader pass recorded for tenant switch, Explain drawer, Downloads copy-to-clipboard.
|
||||
- [ ] Colour tokens audited; contrast reports stored with release artifacts.
|
||||
- [ ] Automated a11y pipelines (Storybook axe, Playwright a11y) green; failures feed the `#console-qa` channel.
|
||||
- [ ] Offline kit a11y smoke executed before publishing each bundle.
|
||||
- [ ] CLI parity gaps logged in `/docs/cli-vs-ui-parity.md`; UI callouts reference fallback commands until parity closes.
|
||||
- [ ] Accessibility Guild sign-off captured in sprint log and release notes reference this guide.
|
||||
- [ ] References cross-checked (`/docs/ui/navigation.md`, `/docs/ui/downloads.md`, `/docs/security/console-security.md`, `/docs/observability/ui-telemetry.md`).
|
||||
|
||||
---
|
||||
|
||||
## 8 · References
|
||||
|
||||
- `/docs/ui/navigation.md` – shortcut definitions, URL schema.
|
||||
- `/docs/ui/downloads.md` – CLI parity and offline copy workflows.
|
||||
- `/docs/ui/console-overview.md` – tenant model, filter behaviours.
|
||||
- `/docs/security/console-security.md` – security metrics and DPoP/fresh-auth requirements.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics mapped to accessibility features.
|
||||
- `/docs/cli-vs-ui-parity.md` – parity status per console feature.
|
||||
- `CONSOLE-QA-23-402` – Accessibility QA backlog (Playwright + manual checks).
|
||||
- `CONSOLE-FEAT-23-102` – Design tokens & theming delivery.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
|
||||
@@ -1,71 +1,71 @@
|
||||
# Air-Gapped Mode Playbook
|
||||
|
||||
> Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
## Overview
|
||||
|
||||
Air-Gapped Mode is the supported operating profile for deployments with **zero external egress**. All inputs arrive via signed mirror bundles, and every surface (CLI, Console, APIs, schedulers, scanners) operates under sealed-network constraints while preserving Aggregation-Only Contract invariants.
|
||||
|
||||
- **Primary components:** Web Services API, Console, CLI, Orchestrator, Task Runner, Conseiller (Feedser), Excitator (VEXer), Policy Engine, Findings Ledger, Export Center, Authority & Tenancy, Notifications, Observability & Forensics.
|
||||
- **Surfaces:** offline bootstrap, mirror ingestion, deterministic jobs, offline advisories/VEX/policy packs/notifications, evidence exports.
|
||||
- **Dependencies:** Export Center, Containerized Distribution, Authority-backed scopes & tenancy, Observability & Forensics, Policy Studio.
|
||||
|
||||
## Guiding principles
|
||||
|
||||
1. **Zero egress:** all outbound network calls are disabled unless explicitly allowed. Any feature requiring online data must degrade gracefully with clear UX messaging.
|
||||
2. **Deterministic inputs:** the platform accepts only signed Mirror Bundles (advisories, VEX, policy packs, vendor feeds, images, dashboards). Bundles carry provenance attestations and chain-of-custody manifests.
|
||||
3. **Auditable exchange:** every import/export records provenance, signatures, and operator identity. Evidence bundles and reports remain verifiable offline.
|
||||
4. **Aggregation-Only Contract compliance:** Conseiller and Excitator continue to aggregate without mutating source records, even when ingesting mirrored feeds.
|
||||
5. **Operator ergonomics:** offline bootstrap, upgrade, and verification steps are reproducible and scripted.
|
||||
|
||||
## Lifecycle & modes
|
||||
|
||||
| Mode | Description | Tooling |
|
||||
| --- | --- | --- |
|
||||
| Connected | Standard deployment with online feeds. Operators use Export Center to build mirror bundles for offline environments. | `stella export bundle create --profile mirror:full` |
|
||||
| Staging mirror | Sealed host that fetches upstream feeds, runs validation, and signs mirror bundles. | Export Center, cosign, bundle validation scripts |
|
||||
| Air-gapped | Production cluster with egress sealed, consuming validated bundles, issuing provenance for inward/outward transfers. | Mirror import CLI, sealed-mode runtime flags |
|
||||
|
||||
### Installation & bootstrap
|
||||
|
||||
1. Prepare mirror bundles (images, charts, advisories/VEX, policy packs, dashboards, telemetry configs).
|
||||
2. Transfer bundles via approved media and validate signatures (`cosign verify`, bundle manifest hash).
|
||||
3. Deploy platform using offline artefacts (`helm install --set airgap.enabled=true`), referencing local registry/object storage.
|
||||
|
||||
### Updates
|
||||
|
||||
1. Staging host generates incremental bundles (mirror delta) with provenance.
|
||||
2. Offline site imports bundles via the CLI (`stella airgap import --bundle`) and records chain-of-custody.
|
||||
3. Scheduler triggers replay jobs with deterministic timelines; results remain reproducible across imports.
|
||||
|
||||
## Component responsibilities
|
||||
|
||||
| Component | Offline duties |
|
||||
| --- | --- |
|
||||
| Export Center | Produce full/delta mirror bundles, signed manifests, provenance attestations. |
|
||||
| Authority & Tenancy | Provide offline scope enforcement, short-lived tokens, revocation via local CRLs. |
|
||||
| Conseiller / Excitator | Ingest mirrored advisories/VEX, enforce AOC, versioned observations. |
|
||||
| Policy Engine & Findings Ledger | Replay evaluations using offline feeds, emit explain traces, support sealed-mode hints. |
|
||||
| Notifications | Deliver locally via approved channels (email relay, webhook proxies) or queue for manual export. |
|
||||
| Observability | Collect metrics/logs/traces locally, generate forensic bundles for external analysis. |
|
||||
|
||||
## Operational guardrails
|
||||
|
||||
- **Network policy:** enforce allowlists (`airgap.egressAllowlist=[]`). Any unexpected outbound request raises an alert.
|
||||
- **Bundle validation:** double-sign manifests (bundle signer + site-specific cosign key); reject on mismatch.
|
||||
- **Time synchronization:** rely on local NTP or manual clock audits; many signatures require monotonic time.
|
||||
- **Key rotation:** plan for offline key ceremonies; Export Center and Authority document rotation playbooks.
|
||||
- **Incident response:** maintain scripts for replaying imports, regenerating manifests, and exporting forensic data without egress.
|
||||
|
||||
## Testing & verification
|
||||
|
||||
- Integration tests mimic offline installs by running with `AIRGAP_ENABLED=true` in CI.
|
||||
- Mirror bundles include validation scripts to compare hash manifests across staging and production.
|
||||
- Sealed-mode smoke tests ensure services fail closed when attempting egress.
|
||||
|
||||
## References
|
||||
|
||||
- Export workflows: `docs/modules/export-center/overview.md`
|
||||
- Policy sealed-mode hints: `docs/policy/overview.md`
|
||||
- Observability forensic bundles: `docs/modules/telemetry/architecture.md`
|
||||
- Runtime posture enforcement: `docs/modules/zastava/operations/runtime.md`
|
||||
# Air-Gapped Mode Playbook
|
||||
|
||||
> Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
## Overview
|
||||
|
||||
Air-Gapped Mode is the supported operating profile for deployments with **zero external egress**. All inputs arrive via signed mirror bundles, and every surface (CLI, Console, APIs, schedulers, scanners) operates under sealed-network constraints while preserving Aggregation-Only Contract invariants.
|
||||
|
||||
- **Primary components:** Web Services API, Console, CLI, Orchestrator, Task Runner, Conseiller (Feedser), Excitator (VEXer), Policy Engine, Findings Ledger, Export Center, Authority & Tenancy, Notifications, Observability & Forensics.
|
||||
- **Surfaces:** offline bootstrap, mirror ingestion, deterministic jobs, offline advisories/VEX/policy packs/notifications, evidence exports.
|
||||
- **Dependencies:** Export Center, Containerized Distribution, Authority-backed scopes & tenancy, Observability & Forensics, Policy Studio.
|
||||
|
||||
## Guiding principles
|
||||
|
||||
1. **Zero egress:** all outbound network calls are disabled unless explicitly allowed. Any feature requiring online data must degrade gracefully with clear UX messaging.
|
||||
2. **Deterministic inputs:** the platform accepts only signed Mirror Bundles (advisories, VEX, policy packs, vendor feeds, images, dashboards). Bundles carry provenance attestations and chain-of-custody manifests.
|
||||
3. **Auditable exchange:** every import/export records provenance, signatures, and operator identity. Evidence bundles and reports remain verifiable offline.
|
||||
4. **Aggregation-Only Contract compliance:** Conseiller and Excitator continue to aggregate without mutating source records, even when ingesting mirrored feeds.
|
||||
5. **Operator ergonomics:** offline bootstrap, upgrade, and verification steps are reproducible and scripted.
|
||||
|
||||
## Lifecycle & modes
|
||||
|
||||
| Mode | Description | Tooling |
|
||||
| --- | --- | --- |
|
||||
| Connected | Standard deployment with online feeds. Operators use Export Center to build mirror bundles for offline environments. | `stella export bundle create --profile mirror:full` |
|
||||
| Staging mirror | Sealed host that fetches upstream feeds, runs validation, and signs mirror bundles. | Export Center, cosign, bundle validation scripts |
|
||||
| Air-gapped | Production cluster with egress sealed, consuming validated bundles, issuing provenance for inward/outward transfers. | Mirror import CLI, sealed-mode runtime flags |
|
||||
|
||||
### Installation & bootstrap
|
||||
|
||||
1. Prepare mirror bundles (images, charts, advisories/VEX, policy packs, dashboards, telemetry configs).
|
||||
2. Transfer bundles via approved media and validate signatures (`cosign verify`, bundle manifest hash).
|
||||
3. Deploy platform using offline artefacts (`helm install --set airgap.enabled=true`), referencing local registry/object storage.
|
||||
|
||||
### Updates
|
||||
|
||||
1. Staging host generates incremental bundles (mirror delta) with provenance.
|
||||
2. Offline site imports bundles via the CLI (`stella airgap import --bundle`) and records chain-of-custody.
|
||||
3. Scheduler triggers replay jobs with deterministic timelines; results remain reproducible across imports.
|
||||
|
||||
## Component responsibilities
|
||||
|
||||
| Component | Offline duties |
|
||||
| --- | --- |
|
||||
| Export Center | Produce full/delta mirror bundles, signed manifests, provenance attestations. |
|
||||
| Authority & Tenancy | Provide offline scope enforcement, short-lived tokens, revocation via local CRLs. |
|
||||
| Conseiller / Excitator | Ingest mirrored advisories/VEX, enforce AOC, versioned observations. |
|
||||
| Policy Engine & Findings Ledger | Replay evaluations using offline feeds, emit explain traces, support sealed-mode hints. |
|
||||
| Notifications | Deliver locally via approved channels (email relay, webhook proxies) or queue for manual export. |
|
||||
| Observability | Collect metrics/logs/traces locally, generate forensic bundles for external analysis. |
|
||||
|
||||
## Operational guardrails
|
||||
|
||||
- **Network policy:** enforce allowlists (`airgap.egressAllowlist=[]`). Any unexpected outbound request raises an alert.
|
||||
- **Bundle validation:** double-sign manifests (bundle signer + site-specific cosign key); reject on mismatch.
|
||||
- **Time synchronization:** rely on local NTP or manual clock audits; many signatures require monotonic time.
|
||||
- **Key rotation:** plan for offline key ceremonies; Export Center and Authority document rotation playbooks.
|
||||
- **Incident response:** maintain scripts for replaying imports, regenerating manifests, and exporting forensic data without egress.
|
||||
|
||||
## Testing & verification
|
||||
|
||||
- Integration tests mimic offline installs by running with `AIRGAP_ENABLED=true` in CI.
|
||||
- Mirror bundles include validation scripts to compare hash manifests across staging and production.
|
||||
- Sealed-mode smoke tests ensure services fail closed when attempting egress.
|
||||
|
||||
## References
|
||||
|
||||
- Export workflows: `docs/modules/export-center/overview.md`
|
||||
- Policy sealed-mode hints: `docs/policy/overview.md`
|
||||
- Observability forensic bundles: `docs/modules/telemetry/architecture.md`
|
||||
- Runtime posture enforcement: `docs/modules/zastava/operations/runtime.md`
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# Aggregation-Only Contract (AOC) Guardrails
|
||||
|
||||
The Aggregation-Only Contract keeps ingestion services deterministic and policy-neutral. Use these checkpoints whenever you add or modify backlog items:
|
||||
|
||||
1. **Ingestion writes raw facts only.** Concelier and Excititor append immutable observations/linksets. No precedence, severity, suppression, or "safe fix" hints may be computed at ingest time.
|
||||
2. **Derived semantics live elsewhere.** Policy Engine overlays, Vuln Explorer composition, and downstream reporting layers attach severity, precedence, policy verdicts, and UI hints.
|
||||
3. **Provenance is mandatory.** Every ingestion write must include original source metadata, digests, and signing/provenance evidence when available. Reject writes lacking provenance.
|
||||
4. **Deterministic outputs.** Given the same inputs, ingestion must produce identical documents, hashes, and event payloads across reruns.
|
||||
5. **Guardrails everywhere.** Roslyn analyzers, schema validators, and CI smoke tests should fail builds that attempt forbidden writes.
|
||||
|
||||
For detailed roles and ownership boundaries, see `AGENTS.md` at the repo root and the module-specific dossiers under `docs/modules/<module>/architecture.md`.
|
||||
|
||||
Need the full contract? Read the [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md) for schemas, error codes, and migration guidance.
|
||||
# Aggregation-Only Contract (AOC) Guardrails
|
||||
|
||||
The Aggregation-Only Contract keeps ingestion services deterministic and policy-neutral. Use these checkpoints whenever you add or modify backlog items:
|
||||
|
||||
1. **Ingestion writes raw facts only.** Concelier and Excititor append immutable observations/linksets. No precedence, severity, suppression, or "safe fix" hints may be computed at ingest time.
|
||||
2. **Derived semantics live elsewhere.** Policy Engine overlays, Vuln Explorer composition, and downstream reporting layers attach severity, precedence, policy verdicts, and UI hints.
|
||||
3. **Provenance is mandatory.** Every ingestion write must include original source metadata, digests, and signing/provenance evidence when available. Reject writes lacking provenance.
|
||||
4. **Deterministic outputs.** Given the same inputs, ingestion must produce identical documents, hashes, and event payloads across reruns.
|
||||
5. **Guardrails everywhere.** Roslyn analyzers, schema validators, and CI smoke tests should fail builds that attempt forbidden writes.
|
||||
|
||||
For detailed roles and ownership boundaries, see `AGENTS.md` at the repo root and the module-specific dossiers under `docs/modules/<module>/architecture.md`.
|
||||
|
||||
Need the full contract? Read the [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md) for schemas, error codes, and migration guidance.
|
||||
|
||||
@@ -1,51 +1,51 @@
|
||||
# SDK & OpenAPI Program
|
||||
|
||||
> Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
## Overview
|
||||
|
||||
The SDK & OpenAPI program delivers canonical OpenAPI 3.1 contracts for every Stella Ops surface, plus officially supported SDKs (TypeScript/Node, Python, Go, Java, C#). It ensures backwards-compatible evolution, documentation, and offline availability.
|
||||
|
||||
- **Primary components:** API Gateway, Web Services, Policy Engine, Conseiller, Excitator, Orchestrator, Findings Ledger, Export Center, Authority & Tenancy, Console, CLI.
|
||||
- **Surfaces:** OpenAPI specs, language SDKs, developer portal, examples, mock server, conformance tests, changelog feeds, deprecation notices.
|
||||
- **Dependencies:** Authority scopes/tenancy, CLI parity, Export Center, Notifications, Air-Gapped Mode, Observability.
|
||||
|
||||
## Program pillars
|
||||
|
||||
1. **Contract-first:** treat OpenAPI specs as the source of truth. CI validates schemas, compatibility, and documentation generation.
|
||||
2. **SDK parity:** language SDKs cover the same surfaces with deterministic clients, pagination helpers, and typed models mirroring Aggregation-Only Contract semantics.
|
||||
3. **Version discipline:** semantic versioning for specs and SDKs, release notes, deprecation windows, and automated change alerts via Notifications.
|
||||
4. **Offline readiness:** specs and SDK bundles ship in Mirror Bundles for air-gapped environments; examples include smoke tests.
|
||||
5. **Observability:** telemetry around SDK usage, spec download metrics, and error reporting funnels back into product decisions.
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Workstream | Deliverable |
|
||||
| --- | --- |
|
||||
| Spec authoring | Unified OpenAPI 3.1 documents per service plus aggregate spec; lint rules; schema registries. |
|
||||
| SDK generation | Language-specific clients with idiomatic ergonomics, retries, pagination, long-running operation helpers, unit + integration tests. |
|
||||
| Dev portal | Consolidated documentation, guides, changelog, copy/paste examples, quickstart scripts. |
|
||||
| Testing | Contract tests against staging, mock server for integration tests, compatibility verification per release. |
|
||||
| Release ops | Automated CI pipelines, version bump workflows, release notes, deprecation policies. |
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Aggregation-Only Contract compliance:** SDKs expose raw advisory/VEX objects without hidden merges; all derived fields require explicit Policy Engine calls.
|
||||
- **Security:** enforce scopes via SDK configuration; redact secrets; support DPoP/mTLS and offline token provisioning.
|
||||
- **Compatibility:** maintain backwards-compatible paths for at least two minor releases; log warnings on deprecated endpoints.
|
||||
- **Documentation:** publish examples for common workflows (scan, policy evaluate, export, attestation) with language parity.
|
||||
|
||||
## Roadmap checkpoints
|
||||
|
||||
1. Baseline OpenAPI specs extracted from gateway, validated, and published.
|
||||
2. TypeScript/Node SDK as pilot, followed by Python and Go.
|
||||
3. Developer portal launch with SDK docs, quickstarts, and mock server.
|
||||
4. Offline kit integration (mirror bundles include specs + SDK tarballs).
|
||||
5. Runtime alerting for breaking changes and dependency vulnerabilities.
|
||||
|
||||
## References
|
||||
|
||||
- API gateway integration: `docs/modules/platform/architecture-overview.md`
|
||||
- Policy/Findings models: `docs/modules/policy/architecture.md`, `docs/modules/vuln-explorer/architecture.md`
|
||||
- Export bundle distribution: `docs/modules/export-center/overview.md`
|
||||
- Offline workflows: `docs/airgap/airgap-mode.md`
|
||||
# SDK & OpenAPI Program
|
||||
|
||||
> Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
## Overview
|
||||
|
||||
The SDK & OpenAPI program delivers canonical OpenAPI 3.1 contracts for every Stella Ops surface, plus officially supported SDKs (TypeScript/Node, Python, Go, Java, C#). It ensures backwards-compatible evolution, documentation, and offline availability.
|
||||
|
||||
- **Primary components:** API Gateway, Web Services, Policy Engine, Conseiller, Excitator, Orchestrator, Findings Ledger, Export Center, Authority & Tenancy, Console, CLI.
|
||||
- **Surfaces:** OpenAPI specs, language SDKs, developer portal, examples, mock server, conformance tests, changelog feeds, deprecation notices.
|
||||
- **Dependencies:** Authority scopes/tenancy, CLI parity, Export Center, Notifications, Air-Gapped Mode, Observability.
|
||||
|
||||
## Program pillars
|
||||
|
||||
1. **Contract-first:** treat OpenAPI specs as the source of truth. CI validates schemas, compatibility, and documentation generation.
|
||||
2. **SDK parity:** language SDKs cover the same surfaces with deterministic clients, pagination helpers, and typed models mirroring Aggregation-Only Contract semantics.
|
||||
3. **Version discipline:** semantic versioning for specs and SDKs, release notes, deprecation windows, and automated change alerts via Notifications.
|
||||
4. **Offline readiness:** specs and SDK bundles ship in Mirror Bundles for air-gapped environments; examples include smoke tests.
|
||||
5. **Observability:** telemetry around SDK usage, spec download metrics, and error reporting funnels back into product decisions.
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Workstream | Deliverable |
|
||||
| --- | --- |
|
||||
| Spec authoring | Unified OpenAPI 3.1 documents per service plus aggregate spec; lint rules; schema registries. |
|
||||
| SDK generation | Language-specific clients with idiomatic ergonomics, retries, pagination, long-running operation helpers, unit + integration tests. |
|
||||
| Dev portal | Consolidated documentation, guides, changelog, copy/paste examples, quickstart scripts. |
|
||||
| Testing | Contract tests against staging, mock server for integration tests, compatibility verification per release. |
|
||||
| Release ops | Automated CI pipelines, version bump workflows, release notes, deprecation policies. |
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Aggregation-Only Contract compliance:** SDKs expose raw advisory/VEX objects without hidden merges; all derived fields require explicit Policy Engine calls.
|
||||
- **Security:** enforce scopes via SDK configuration; redact secrets; support DPoP/mTLS and offline token provisioning.
|
||||
- **Compatibility:** maintain backwards-compatible paths for at least two minor releases; log warnings on deprecated endpoints.
|
||||
- **Documentation:** publish examples for common workflows (scan, policy evaluate, export, attestation) with language parity.
|
||||
|
||||
## Roadmap checkpoints
|
||||
|
||||
1. Baseline OpenAPI specs extracted from gateway, validated, and published.
|
||||
2. TypeScript/Node SDK as pilot, followed by Python and Go.
|
||||
3. Developer portal launch with SDK docs, quickstarts, and mock server.
|
||||
4. Offline kit integration (mirror bundles include specs + SDK tarballs).
|
||||
5. Runtime alerting for breaking changes and dependency vulnerabilities.
|
||||
|
||||
## References
|
||||
|
||||
- API gateway integration: `docs/modules/platform/architecture-overview.md`
|
||||
- Policy/Findings models: `docs/modules/policy/architecture.md`, `docs/modules/vuln-explorer/architecture.md`
|
||||
- Export bundle distribution: `docs/modules/export-center/overview.md`
|
||||
- Offline workflows: `docs/airgap/airgap-mode.md`
|
||||
|
||||
@@ -1,155 +1,155 @@
|
||||
# Console CLI ↔ UI Parity Matrix
|
||||
|
||||
> **Audience:** Docs Guild, Console Guild, CLI Guild, DevOps automation.
|
||||
> **Scope:** Track feature-level parity between the StellaOps Console and the `stella` CLI, surface pending work, and describe the parity CI check owned by CONSOLE-DOC-23-502.
|
||||
|
||||
Status key:
|
||||
|
||||
- **✅ Available** – command exists in `StellaOps.Cli` and is documented.
|
||||
- **🟡 In progress** – command implemented but still under active delivery (task status `DOING`).
|
||||
- **🟩 Planned** – command spec’d but not yet implemented (task `TODO`).
|
||||
- **⚪ UI-only** – no CLI equivalent required.
|
||||
- **🔴 Gap** – CLI feature missing with no active task; file a task before sprint exit.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Navigation & Tenancy
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Login / token cache status (`/console/profile`) | `stella auth login`, `stella auth status`, `stella auth whoami` | ✅ Available | Command definitions in `CommandFactory.BuildAuthCommand`. |
|
||||
| Fresh-auth challenge for sensitive actions | `stella auth fresh-auth` | ✅ Available | Referenced in `/docs/ui/admin.md`. |
|
||||
| Tenant switcher (UI shell) | `--tenant` flag across CLI commands | ✅ Available | All multi-tenant commands require explicit `--tenant`. |
|
||||
| Tenant creation / suspension | *(pending CLI)* | 🟩 Planned | No `stella auth tenant *` commands yet – track via `CLI-TEN-47-001` (scopes & tenancy). |
|
||||
|
||||
---
|
||||
|
||||
## 2 · Policies & Findings
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Policy simulation diff, explain | `stella policy simulate` | 🟡 In progress | Implementation present; task `CLI-POLICY-20-002` marked DOING. |
|
||||
| Promote / activate policy | `stella policy promote`, `stella policy activate` | 🟩 Planned | Spec tracked under `CLI-POLICY-23-005`. |
|
||||
| History & explain trees | `stella policy history`, `stella policy explain` | 🟩 Planned | `CLI-POLICY-23-006`. |
|
||||
| Findings explorer export | `stella findings get`, `stella findings export` | 🟩 Planned | Part of `CLI-POLICY-20-003`. |
|
||||
| Explain drawer JSON | `stella policy simulate --format json` | 🟡 In progress | Same command; JSON output flagged for CLI tests. |
|
||||
|
||||
---
|
||||
|
||||
## 3 · Runs & Evidence
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Run retry / cancel | `stella runs retry`, `stella runs cancel` | 🟩 Planned | Included in export suite task `CLI-EXPORT-35-001`. |
|
||||
| Manual run submit / preview | `stella runs submit`, `stella runs preview` | 🟩 Planned | `CLI-EXPORT-35-001`. |
|
||||
| Evidence bundle export | `stella runs export --run <id> --bundle` | 🟩 Planned | `CLI-EXPORT-35-001`. |
|
||||
| Run status polling | `stella runs status` | 🟩 Planned | Same task. |
|
||||
|
||||
---
|
||||
|
||||
## 4 · Advisories, VEX, SBOM
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Advisory observations search | `stella vuln observations` | ✅ Available | Implemented via `BuildVulnCommand`. |
|
||||
| Advisory linkset export | `stella advisory linkset show/export` | 🟩 Planned | `CLI-LNM-22-001`. |
|
||||
| VEX observations / linksets | `stella vex obs get/linkset show` | 🟩 Planned | `CLI-LNM-22-002`. |
|
||||
| SBOM overlay export | `stella sbom overlay apply/export` | 🟩 Planned | Scoped to upcoming SBOM CLI sprint (`SBOM-CONSOLE-23-001/002` + CLI backlog). |
|
||||
|
||||
---
|
||||
|
||||
## 5 · Downloads & Offline Kit
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Manifest lookup (Console Downloads) | `stella downloads manifest show --artifact <id>` | 🟩 Planned | Delivered with `CONSOLE-DOC-23-502` + CLI parity commands. |
|
||||
| Mirror digest to OCI archive | `stella downloads mirror --artifact <id> --to <target>` | 🟩 Planned | Same task bundle (`CONSOLE-DOC-23-502`). |
|
||||
| Console health check | `stella console status --endpoint <url>` | 🟩 Planned | Tracked in `CONSOLE-DOC-23-502`; interim use `curl` as documented. |
|
||||
| Offline kit import/export | `stella offline kit import`, `stella offline kit export` | ✅ Available | Implemented (see `CommandHandlers.HandleOfflineKitImportAsync/HandleOfflineKitPullAsync`). |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Admin & Security
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Client creation / rotation | `stella auth client create` *(planned)* | 🟩 Planned | Pending tenancy backlog `CLI-TEN-47-001`. |
|
||||
| Token revoke | `stella auth revoke export/verify` | ✅ Available | Already implemented. |
|
||||
| Audit export | `stella auth audit export` | 🟩 Planned | Needs CLI work item (Authority guild). |
|
||||
| Signing key rotation | `stella auth signing rotate` | 🟩 Planned | To be added with AUTH-CONSOLE-23-003 follow-up. |
|
||||
|
||||
---
|
||||
|
||||
## 7 · Telemetry & Observability
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Telemetry dashboard parity | `stella obs top`, `stella obs trace`, `stella obs logs` | 🟩 Planned | CLI observability epic (`CLI-OBS-51-001`, `CLI-OBS-52-001`). |
|
||||
| Incident mode toggle | `stella obs incident-mode enable|disable|status` | 🟩 Planned | CLI task `CLI-OBS-55-001`. |
|
||||
| Verify console telemetry health | `stella console status --telemetry` | 🟩 Planned | Part of `CONSOLE-DOC-23-502`. |
|
||||
|
||||
---
|
||||
|
||||
## 8 · Parity Gaps & Follow-up
|
||||
|
||||
- **Tenant and client lifecycle CLI**: create/suspend tenants, manage clients. Coordinate with Authority CLI epic (`CLI-TEN-47-001`, `CLI-TEN-49-001`).
|
||||
- **Downloads parity commands**: blocked on `CONSOLE-DOC-23-502` and DevOps pipeline `DOWNLOADS-CONSOLE-23-001`.
|
||||
- **Policy promotion/history**: requires completion of CLI policy epic (`CLI-POLICY-23-005`/`23-006`).
|
||||
- **Runs/evidence exports**: waiting on `CLI-EXPORT-35-001`.
|
||||
- **Observability tooling**: deliver `stella obs` commands before enabling parity CI checks for telemetry.
|
||||
|
||||
Document updates should occur whenever a row changes status. When promoting a command from Planned → Available, ensure:
|
||||
|
||||
1. CLI command merged with help text.
|
||||
2. Relevant UI doc references updated to remove “pending” callouts.
|
||||
3. This matrix row status updated to ✅ and task IDs moved to release notes.
|
||||
|
||||
---
|
||||
|
||||
## 9 · Parity CI Check (CONSOLE-DOC-23-502)
|
||||
|
||||
- **Owner:** Docs Guild + DevEx/CLI Guild.
|
||||
- **Artefact:** Planned `.gitea/workflows/cli-parity-console.yml`.
|
||||
- **What it does:** Runs `scripts/check-console-cli-parity.sh` (to be committed with the workflow) which:
|
||||
1. Parses this matrix (YAML view exported from Markdown) to identify rows marked ✅.
|
||||
2. Executes `stella --help` to confirm listed commands exist.
|
||||
3. Optionally triggers smoke commands in sandbox mode (e.g., `stella policy simulate --help`).
|
||||
- **Failure action:** Workflow fails when a listed command is missing or when a row marked ✅ still contains “pending” notes. Update the matrix or fix CLI implementation before merging.
|
||||
|
||||
Until the workflow lands, run the checker locally:
|
||||
|
||||
```bash
|
||||
# Pending CONSOLE-DOC-23-502 – placeholder command
|
||||
./scripts/check-console-cli-parity.sh
|
||||
```
|
||||
|
||||
The script should emit a parity report that feeds into the Downloads workspace (`kind = "parity.report"`).
|
||||
|
||||
---
|
||||
|
||||
## 10 · Compliance checklist
|
||||
|
||||
- [ ] Matrix reflects latest command availability (statuses accurate, task IDs linked).
|
||||
- [ ] Notes include owning backlog items for every 🟩 / 🟡 row.
|
||||
- [ ] CLI commands marked ✅ have corresponding entries in `/docs/modules/cli/guides/*.md` or module-specific docs.
|
||||
- [ ] CI parity workflow description kept in sync with CONSOLE-DOC-23-502 implementation.
|
||||
- [ ] Downloads workspace links to latest parity report.
|
||||
- [ ] Install / observability guides reference this matrix for pending CLI parity.
|
||||
- [ ] Offline workflows capture CLI fallbacks when commands are pending.
|
||||
- [ ] Docs Guild review recorded in sprint log once parity CI lands.
|
||||
|
||||
---
|
||||
|
||||
## 11 · References
|
||||
|
||||
- `/docs/ui/*.md` – per-surface UI parity callouts.
|
||||
- `/docs/install/docker.md` – CLI parity section for deployments.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics referencing CLI checks.
|
||||
- `/docs/security/console-security.md` – security metrics & CLI parity expectations.
|
||||
- `src/Cli/StellaOps.Cli/TASKS.md` – authoritative status for CLI backlog.
|
||||
- `/docs/updates/2025-10-28-docs-guild.md` – coordination note for Authority/Security follow-up.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
# Console CLI ↔ UI Parity Matrix
|
||||
|
||||
> **Audience:** Docs Guild, Console Guild, CLI Guild, DevOps automation.
|
||||
> **Scope:** Track feature-level parity between the StellaOps Console and the `stella` CLI, surface pending work, and describe the parity CI check owned by CONSOLE-DOC-23-502.
|
||||
|
||||
Status key:
|
||||
|
||||
- **✅ Available** – command exists in `StellaOps.Cli` and is documented.
|
||||
- **🟡 In progress** – command implemented but still under active delivery (task status `DOING`).
|
||||
- **🟩 Planned** – command spec’d but not yet implemented (task `TODO`).
|
||||
- **⚪ UI-only** – no CLI equivalent required.
|
||||
- **🔴 Gap** – CLI feature missing with no active task; file a task before sprint exit.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Navigation & Tenancy
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Login / token cache status (`/console/profile`) | `stella auth login`, `stella auth status`, `stella auth whoami` | ✅ Available | Command definitions in `CommandFactory.BuildAuthCommand`. |
|
||||
| Fresh-auth challenge for sensitive actions | `stella auth fresh-auth` | ✅ Available | Referenced in `/docs/ui/admin.md`. |
|
||||
| Tenant switcher (UI shell) | `--tenant` flag across CLI commands | ✅ Available | All multi-tenant commands require explicit `--tenant`. |
|
||||
| Tenant creation / suspension | *(pending CLI)* | 🟩 Planned | No `stella auth tenant *` commands yet – track via `CLI-TEN-47-001` (scopes & tenancy). |
|
||||
|
||||
---
|
||||
|
||||
## 2 · Policies & Findings
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Policy simulation diff, explain | `stella policy simulate` | 🟡 In progress | Implementation present; task `CLI-POLICY-20-002` marked DOING. |
|
||||
| Promote / activate policy | `stella policy promote`, `stella policy activate` | 🟩 Planned | Spec tracked under `CLI-POLICY-23-005`. |
|
||||
| History & explain trees | `stella policy history`, `stella policy explain` | 🟩 Planned | `CLI-POLICY-23-006`. |
|
||||
| Findings explorer export | `stella findings get`, `stella findings export` | 🟩 Planned | Part of `CLI-POLICY-20-003`. |
|
||||
| Explain drawer JSON | `stella policy simulate --format json` | 🟡 In progress | Same command; JSON output flagged for CLI tests. |
|
||||
|
||||
---
|
||||
|
||||
## 3 · Runs & Evidence
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Run retry / cancel | `stella runs retry`, `stella runs cancel` | 🟩 Planned | Included in export suite task `CLI-EXPORT-35-001`. |
|
||||
| Manual run submit / preview | `stella runs submit`, `stella runs preview` | 🟩 Planned | `CLI-EXPORT-35-001`. |
|
||||
| Evidence bundle export | `stella runs export --run <id> --bundle` | 🟩 Planned | `CLI-EXPORT-35-001`. |
|
||||
| Run status polling | `stella runs status` | 🟩 Planned | Same task. |
|
||||
|
||||
---
|
||||
|
||||
## 4 · Advisories, VEX, SBOM
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Advisory observations search | `stella vuln observations` | ✅ Available | Implemented via `BuildVulnCommand`. |
|
||||
| Advisory linkset export | `stella advisory linkset show/export` | 🟩 Planned | `CLI-LNM-22-001`. |
|
||||
| VEX observations / linksets | `stella vex obs get/linkset show` | 🟩 Planned | `CLI-LNM-22-002`. |
|
||||
| SBOM overlay export | `stella sbom overlay apply/export` | 🟩 Planned | Scoped to upcoming SBOM CLI sprint (`SBOM-CONSOLE-23-001/002` + CLI backlog). |
|
||||
|
||||
---
|
||||
|
||||
## 5 · Downloads & Offline Kit
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Manifest lookup (Console Downloads) | `stella downloads manifest show --artifact <id>` | 🟩 Planned | Delivered with `CONSOLE-DOC-23-502` + CLI parity commands. |
|
||||
| Mirror digest to OCI archive | `stella downloads mirror --artifact <id> --to <target>` | 🟩 Planned | Same task bundle (`CONSOLE-DOC-23-502`). |
|
||||
| Console health check | `stella console status --endpoint <url>` | 🟩 Planned | Tracked in `CONSOLE-DOC-23-502`; interim use `curl` as documented. |
|
||||
| Offline kit import/export | `stella offline kit import`, `stella offline kit export` | ✅ Available | Implemented (see `CommandHandlers.HandleOfflineKitImportAsync/HandleOfflineKitPullAsync`). |
|
||||
|
||||
---
|
||||
|
||||
## 6 · Admin & Security
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Client creation / rotation | `stella auth client create` *(planned)* | 🟩 Planned | Pending tenancy backlog `CLI-TEN-47-001`. |
|
||||
| Token revoke | `stella auth revoke export/verify` | ✅ Available | Already implemented. |
|
||||
| Audit export | `stella auth audit export` | 🟩 Planned | Needs CLI work item (Authority guild). |
|
||||
| Signing key rotation | `stella auth signing rotate` | 🟩 Planned | To be added with AUTH-CONSOLE-23-003 follow-up. |
|
||||
|
||||
---
|
||||
|
||||
## 7 · Telemetry & Observability
|
||||
|
||||
| UI capability | CLI command(s) | Status | Notes / Tasks |
|
||||
|---------------|----------------|--------|---------------|
|
||||
| Telemetry dashboard parity | `stella obs top`, `stella obs trace`, `stella obs logs` | 🟩 Planned | CLI observability epic (`CLI-OBS-51-001`, `CLI-OBS-52-001`). |
|
||||
| Incident mode toggle | `stella obs incident-mode enable|disable|status` | 🟩 Planned | CLI task `CLI-OBS-55-001`. |
|
||||
| Verify console telemetry health | `stella console status --telemetry` | 🟩 Planned | Part of `CONSOLE-DOC-23-502`. |
|
||||
|
||||
---
|
||||
|
||||
## 8 · Parity Gaps & Follow-up
|
||||
|
||||
- **Tenant and client lifecycle CLI**: create/suspend tenants, manage clients. Coordinate with Authority CLI epic (`CLI-TEN-47-001`, `CLI-TEN-49-001`).
|
||||
- **Downloads parity commands**: blocked on `CONSOLE-DOC-23-502` and DevOps pipeline `DOWNLOADS-CONSOLE-23-001`.
|
||||
- **Policy promotion/history**: requires completion of CLI policy epic (`CLI-POLICY-23-005`/`23-006`).
|
||||
- **Runs/evidence exports**: waiting on `CLI-EXPORT-35-001`.
|
||||
- **Observability tooling**: deliver `stella obs` commands before enabling parity CI checks for telemetry.
|
||||
|
||||
Document updates should occur whenever a row changes status. When promoting a command from Planned → Available, ensure:
|
||||
|
||||
1. CLI command merged with help text.
|
||||
2. Relevant UI doc references updated to remove “pending” callouts.
|
||||
3. This matrix row status updated to ✅ and task IDs moved to release notes.
|
||||
|
||||
---
|
||||
|
||||
## 9 · Parity CI Check (CONSOLE-DOC-23-502)
|
||||
|
||||
- **Owner:** Docs Guild + DevEx/CLI Guild.
|
||||
- **Artefact:** Planned `.gitea/workflows/cli-parity-console.yml`.
|
||||
- **What it does:** Runs `scripts/check-console-cli-parity.sh` (to be committed with the workflow) which:
|
||||
1. Parses this matrix (YAML view exported from Markdown) to identify rows marked ✅.
|
||||
2. Executes `stella --help` to confirm listed commands exist.
|
||||
3. Optionally triggers smoke commands in sandbox mode (e.g., `stella policy simulate --help`).
|
||||
- **Failure action:** Workflow fails when a listed command is missing or when a row marked ✅ still contains “pending” notes. Update the matrix or fix CLI implementation before merging.
|
||||
|
||||
Until the workflow lands, run the checker locally:
|
||||
|
||||
```bash
|
||||
# Pending CONSOLE-DOC-23-502 – placeholder command
|
||||
./scripts/check-console-cli-parity.sh
|
||||
```
|
||||
|
||||
The script should emit a parity report that feeds into the Downloads workspace (`kind = "parity.report"`).
|
||||
|
||||
---
|
||||
|
||||
## 10 · Compliance checklist
|
||||
|
||||
- [ ] Matrix reflects latest command availability (statuses accurate, task IDs linked).
|
||||
- [ ] Notes include owning backlog items for every 🟩 / 🟡 row.
|
||||
- [ ] CLI commands marked ✅ have corresponding entries in `/docs/modules/cli/guides/*.md` or module-specific docs.
|
||||
- [ ] CI parity workflow description kept in sync with CONSOLE-DOC-23-502 implementation.
|
||||
- [ ] Downloads workspace links to latest parity report.
|
||||
- [ ] Install / observability guides reference this matrix for pending CLI parity.
|
||||
- [ ] Offline workflows capture CLI fallbacks when commands are pending.
|
||||
- [ ] Docs Guild review recorded in sprint log once parity CI lands.
|
||||
|
||||
---
|
||||
|
||||
## 11 · References
|
||||
|
||||
- `/docs/ui/*.md` – per-surface UI parity callouts.
|
||||
- `/docs/install/docker.md` – CLI parity section for deployments.
|
||||
- `/docs/observability/ui-telemetry.md` – telemetry metrics referencing CLI checks.
|
||||
- `/docs/security/console-security.md` – security metrics & CLI parity expectations.
|
||||
- `src/Cli/StellaOps.Cli/TASKS.md` – authoritative status for CLI backlog.
|
||||
- `/docs/updates/2025-10-28-docs-guild.md` – coordination note for Authority/Security follow-up.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-28 (Sprint 23).*
|
||||
|
||||
|
||||
@@ -1,43 +1,43 @@
|
||||
# Concelier Connector Research – 2025-10-11
|
||||
|
||||
Snapshot of direct network checks performed on 2025-10-11 (UTC) for the national/vendor connectors in scope. Use alongside each module’s `TASKS.md` notes.
|
||||
|
||||
## ACSC (Australia)
|
||||
- Enumerated feed slugs `/acsc/view-all-content/{alerts,advisories,news,publications,threats}/rss`; every endpoint negotiates HTTP/2 then aborts with `INTERNAL_ERROR` (curl exit 92). Forcing HTTP/1.1 hangs >600 s and sitemap/HTML fetches fail the same way.
|
||||
- Next actions: prototype `SocketsHttpHandler` settings (`RequestVersionOrLower`, allow fallback to relay), capture successful headers from partner vantage (need retention + cache semantics), and keep `FEEDCONN-SHARED-HTTP2-001` open for downgrade work.
|
||||
|
||||
## CCCS (Canada)
|
||||
- JSON endpoint (`https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=<lang>&content_type=cccs_threat`) returns ~5 100 records per language; `page=<n>` still works for segmented pulls and the earliest `date_created` seen is 2018‑06‑08 (EN) / 2018‑06‑08 (FR). Use an explicit `User-Agent` to avoid 403 responses.
|
||||
- Follow-up: telemetry, sanitiser coverage, and backfill procedures are documented in `docs/modules/concelier/operations/connectors/cccs.md` (2025‑10‑15). Adjust `maxEntriesPerFetch` when performing historical sweeps so cursor state remains responsive.
|
||||
|
||||
## CERT-Bund (Germany)
|
||||
- `https://wid.cert-bund.de/content/public/securityAdvisory/rss` responds 200 without cookies (≈250-item window, German taxonomy). Detail links load an Angular SPA that fetches JSON behind the bootstrap session.
|
||||
- Confirmed `GET https://wid.cert-bund.de/portal/api/securityadvisory?name=<WID-SEC-…>` returns JSON once the portal cookie container is primed; payload includes severity, CVEs, products, and references used by the connector fixtures.
|
||||
- Historical advisories accessible through the SPA search/export endpoints once the `XSRF-TOKEN` cookie (exposed via `GET /portal/api/security/csrf`) is supplied with the `X-XSRF-TOKEN` header:
|
||||
- `POST /portal/api/securityadvisory/search` (`{"page":N,"size":100,"sort":["published,desc"]}`) pages data back to 2014.
|
||||
- `GET /portal/api/securityadvisory/export?format=json&from=YYYY-MM-DD` emits JSON bundles suitable for Offline Kit mirrors.
|
||||
- Locale note: content is German-only; Concelier preserves `language=de` and Docs will publish a CERT-Bund glossary so operators can bridge terminology without machine translation.
|
||||
|
||||
## KISA / KNVD (Korea)
|
||||
- `https://knvd.krcert.or.kr/rss/securityInfo.do` and `/rss/securityNotice.do` return UTF-8 RSS (10-item window) with `detailDos.do?IDX=` links. No cookies required for feed fetch.
|
||||
- Detail SPA calls resolve to `rssDetailData.do?IDX=` JSON payloads; connector fetches those directly, sanitises HTML, and records Hangul metadata (NFC). See `docs/dev/kisa_connector_notes.md` for telemetry + localisation guidance.
|
||||
|
||||
## BDU (Russia / FSTEC)
|
||||
- Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml/json`) return 403/404; TLS chain requires Russian Trusted Sub CA and WAF expects additional headers.
|
||||
- Next actions: acquire official PEM chain, point `concelier:httpClients:source.bdu:trustedRootPaths` (or `concelier:sources:bdu:http:trustedRootPaths`) at the Offline Kit PEM, keep `allowInvalidCertificates=false`, script session bootstrap, then capture RSS/HTML schema for parser work.
|
||||
|
||||
## NKTsKI / cert.gov.ru (Russia)
|
||||
- `https://cert.gov.ru/rss/advisories.xml` served via Bitrix returns 403/404 even with `Accept-Language: ru-RU`; TLS chain also requires Russian trust anchors.
|
||||
- Next actions: source trust store, configure `concelier:httpClients:source.nkcki:trustedRootPaths` (Offline Kit root via `concelier:offline:root`), prepare proxy fallback, and once accessible document taxonomy/retention plus attachment handling.
|
||||
|
||||
## CISA ICS (United States)
|
||||
- `curl -I https://www.cisa.gov/cybersecurity-advisories/ics-advisories.xml` returns HTTP 403 + `x-reference-error` (Akamai). Same for legacy feed paths.
|
||||
- Next actions: secure GovDelivery access, document token rotation, and build HTML/email fallback with throttling.
|
||||
|
||||
## Cisco PSIRT
|
||||
- `https://api.cisco.com/security/advisories/latest` returns `ERR_596_SERVICE_NOT_FOUND` when unauthenticated. openVuln REST requires Mashery OAuth (client credentials) with quotas ~5 req/s, 30/min, 5 000/day; supports `pageIndex/pageSize` pagination.
|
||||
- Next actions: register OAuth app, capture pagination/delta parameters, and compare API vs RSS coverage.
|
||||
|
||||
## Microsoft MSRC
|
||||
- REST endpoint (`https://api.msrc.microsoft.com/sug/v2.0/en-US/vulnerabilities`) requires Azure AD token + `api-version` (current `2024-08-01`) and supports delta filters (`lastModifiedStartDateTime`). CVRF ZIP remains available for offline use.
|
||||
- Next actions: finalise AAD app registration, implement token cache, and design combined REST+CVRF ingestion path for determinism.
|
||||
# Concelier Connector Research – 2025-10-11
|
||||
|
||||
Snapshot of direct network checks performed on 2025-10-11 (UTC) for the national/vendor connectors in scope. Use alongside each module’s `TASKS.md` notes.
|
||||
|
||||
## ACSC (Australia)
|
||||
- Enumerated feed slugs `/acsc/view-all-content/{alerts,advisories,news,publications,threats}/rss`; every endpoint negotiates HTTP/2 then aborts with `INTERNAL_ERROR` (curl exit 92). Forcing HTTP/1.1 hangs >600 s and sitemap/HTML fetches fail the same way.
|
||||
- Next actions: prototype `SocketsHttpHandler` settings (`RequestVersionOrLower`, allow fallback to relay), capture successful headers from partner vantage (need retention + cache semantics), and keep `FEEDCONN-SHARED-HTTP2-001` open for downgrade work.
|
||||
|
||||
## CCCS (Canada)
|
||||
- JSON endpoint (`https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=<lang>&content_type=cccs_threat`) returns ~5 100 records per language; `page=<n>` still works for segmented pulls and the earliest `date_created` seen is 2018‑06‑08 (EN) / 2018‑06‑08 (FR). Use an explicit `User-Agent` to avoid 403 responses.
|
||||
- Follow-up: telemetry, sanitiser coverage, and backfill procedures are documented in `docs/modules/concelier/operations/connectors/cccs.md` (2025‑10‑15). Adjust `maxEntriesPerFetch` when performing historical sweeps so cursor state remains responsive.
|
||||
|
||||
## CERT-Bund (Germany)
|
||||
- `https://wid.cert-bund.de/content/public/securityAdvisory/rss` responds 200 without cookies (≈250-item window, German taxonomy). Detail links load an Angular SPA that fetches JSON behind the bootstrap session.
|
||||
- Confirmed `GET https://wid.cert-bund.de/portal/api/securityadvisory?name=<WID-SEC-…>` returns JSON once the portal cookie container is primed; payload includes severity, CVEs, products, and references used by the connector fixtures.
|
||||
- Historical advisories accessible through the SPA search/export endpoints once the `XSRF-TOKEN` cookie (exposed via `GET /portal/api/security/csrf`) is supplied with the `X-XSRF-TOKEN` header:
|
||||
- `POST /portal/api/securityadvisory/search` (`{"page":N,"size":100,"sort":["published,desc"]}`) pages data back to 2014.
|
||||
- `GET /portal/api/securityadvisory/export?format=json&from=YYYY-MM-DD` emits JSON bundles suitable for Offline Kit mirrors.
|
||||
- Locale note: content is German-only; Concelier preserves `language=de` and Docs will publish a CERT-Bund glossary so operators can bridge terminology without machine translation.
|
||||
|
||||
## KISA / KNVD (Korea)
|
||||
- `https://knvd.krcert.or.kr/rss/securityInfo.do` and `/rss/securityNotice.do` return UTF-8 RSS (10-item window) with `detailDos.do?IDX=` links. No cookies required for feed fetch.
|
||||
- Detail SPA calls resolve to `rssDetailData.do?IDX=` JSON payloads; connector fetches those directly, sanitises HTML, and records Hangul metadata (NFC). See `docs/dev/kisa_connector_notes.md` for telemetry + localisation guidance.
|
||||
|
||||
## BDU (Russia / FSTEC)
|
||||
- Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml/json`) return 403/404; TLS chain requires Russian Trusted Sub CA and WAF expects additional headers.
|
||||
- Next actions: acquire official PEM chain, point `concelier:httpClients:source.bdu:trustedRootPaths` (or `concelier:sources:bdu:http:trustedRootPaths`) at the Offline Kit PEM, keep `allowInvalidCertificates=false`, script session bootstrap, then capture RSS/HTML schema for parser work.
|
||||
|
||||
## NKTsKI / cert.gov.ru (Russia)
|
||||
- `https://cert.gov.ru/rss/advisories.xml` served via Bitrix returns 403/404 even with `Accept-Language: ru-RU`; TLS chain also requires Russian trust anchors.
|
||||
- Next actions: source trust store, configure `concelier:httpClients:source.nkcki:trustedRootPaths` (Offline Kit root via `concelier:offline:root`), prepare proxy fallback, and once accessible document taxonomy/retention plus attachment handling.
|
||||
|
||||
## CISA ICS (United States)
|
||||
- `curl -I https://www.cisa.gov/cybersecurity-advisories/ics-advisories.xml` returns HTTP 403 + `x-reference-error` (Akamai). Same for legacy feed paths.
|
||||
- Next actions: secure GovDelivery access, document token rotation, and build HTML/email fallback with throttling.
|
||||
|
||||
## Cisco PSIRT
|
||||
- `https://api.cisco.com/security/advisories/latest` returns `ERR_596_SERVICE_NOT_FOUND` when unauthenticated. openVuln REST requires Mashery OAuth (client credentials) with quotas ~5 req/s, 30/min, 5 000/day; supports `pageIndex/pageSize` pagination.
|
||||
- Next actions: register OAuth app, capture pagination/delta parameters, and compare API vs RSS coverage.
|
||||
|
||||
## Microsoft MSRC
|
||||
- REST endpoint (`https://api.msrc.microsoft.com/sug/v2.0/en-US/vulnerabilities`) requires Azure AD token + `api-version` (current `2024-08-01`) and supports delta filters (`lastModifiedStartDateTime`). CVRF ZIP remains available for offline use.
|
||||
- Next actions: finalise AAD app registration, implement token cache, and design combined REST+CVRF ingestion path for determinism.
|
||||
|
||||
@@ -1,228 +1,228 @@
|
||||
# Deploying the StellaOps Console
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, operators rolling out the web console.
|
||||
> **Scope:** Helm and Docker Compose deployment steps, ingress/TLS configuration, required environment variables, health checks, offline/air-gap operation, and compliance checklist (Sprint 23).
|
||||
|
||||
The StellaOps Console ships as part of the `stellaops` stack Helm chart and Compose bundles maintained under `deploy/`. This guide describes the supported deployment paths, the configuration surface, and operational checks needed to run the console in connected or air-gapped environments.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.28+) with ingress controller (NGINX, Traefik, or equivalent) and Cert-Manager for automated TLS, or Docker host for Compose deployments.
|
||||
- Container registry access to `registry.stella-ops.org` (or mirrored registry) for all images listed in `deploy/releases/*.yaml`.
|
||||
- Authority service configured with console client (`aud=ui`, scopes `ui.read`, `ui.admin`).
|
||||
- DNS entry pointing to the console hostname (for example, `console.acme.internal`).
|
||||
- Cosign public key for manifest verification (`deploy/releases/manifest.json.sig`).
|
||||
- Optional: Offline Kit bundle for air-gapped sites (`stella-ops-offline-kit-<ver>.tar.gz`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Helm deployment (recommended)
|
||||
|
||||
### 2.1 Install chart repository
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://downloads.stella-ops.org/helm
|
||||
helm repo update stellaops
|
||||
```
|
||||
|
||||
If operating offline, copy the chart archive from the Offline Kit (`deploy/helm/stellaops-<ver>.tgz`) and run:
|
||||
|
||||
```bash
|
||||
helm install stellaops ./stellaops-<ver>.tgz --namespace stellaops --create-namespace
|
||||
```
|
||||
|
||||
### 2.2 Base installation
|
||||
|
||||
```bash
|
||||
helm install stellaops stellaops/stellaops \
|
||||
--namespace stellaops \
|
||||
--create-namespace \
|
||||
--values deploy/helm/stellaops/values-prod.yaml
|
||||
```
|
||||
|
||||
The chart deploys Authority, Console web/API gateway, Scanner API, Scheduler, and supporting services. The console frontend pod is labelled `app=stellaops-web-ui`.
|
||||
|
||||
### 2.3 Helm values highlights
|
||||
|
||||
Key sections in `deploy/helm/stellaops/values-prod.yaml`:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `console.ingress.host` | Hostname served by the console (`console.example.com`). |
|
||||
| `console.ingress.tls.secretName` | Kubernetes secret containing TLS certificate (generated by Cert-Manager or uploaded manually). |
|
||||
| `console.config.apiGateway.baseUrl` | Internal base URL the UI uses to reach the gateway (defaults to `https://stellaops-web`). |
|
||||
| `console.env.AUTHORITY_ISSUER` | Authority issuer URL (for example, `https://authority.example.com`). |
|
||||
| `console.env.AUTHORITY_CLIENT_ID` | Authority client ID for the console UI. |
|
||||
| `console.env.AUTHORITY_SCOPES` | Space-separated scopes required by UI (`ui.read ui.admin`). |
|
||||
| `console.resources` | CPU/memory requests and limits (default 250m CPU / 512Mi memory). |
|
||||
| `console.podAnnotations` | Optional annotations for service mesh or monitoring. |
|
||||
|
||||
Use `values-stage.yaml`, `values-dev.yaml`, or `values-airgap.yaml` as templates for other environments.
|
||||
|
||||
### 2.4 TLS and ingress
|
||||
|
||||
Example ingress override:
|
||||
|
||||
```yaml
|
||||
console:
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
host: console.acme.internal
|
||||
tls:
|
||||
enabled: true
|
||||
secretName: console-tls
|
||||
```
|
||||
|
||||
Generate certificates using Cert-Manager or provide an existing secret. For air-gapped deployments, pre-create the secret with the mirrored CA chain.
|
||||
|
||||
### 2.5 Health checks
|
||||
|
||||
Console pods expose:
|
||||
|
||||
| Path | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `/health/live` | Liveness probe | Confirms process responsive. |
|
||||
| `/health/ready` | Readiness probe | Verifies configuration bootstrap and Authority reachability. |
|
||||
| `/metrics` | Prometheus metrics | Enabled when `console.metrics.enabled=true`. |
|
||||
|
||||
Helm chart sets default probes (`initialDelaySeconds: 10`, `periodSeconds: 15`). Adjust via `console.livenessProbe` and `console.readinessProbe`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Docker Compose deployment
|
||||
|
||||
Located in `deploy/compose/docker-compose.console.yaml`. Quick start:
|
||||
|
||||
```bash
|
||||
cd deploy/compose
|
||||
docker compose -f docker-compose.console.yaml --env-file console.env up -d
|
||||
```
|
||||
|
||||
`console.env` should define:
|
||||
|
||||
```
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.acme.internal
|
||||
AUTHORITY_ISSUER=https://authority.acme.internal
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_CLIENT_SECRET=<if using confidential client>
|
||||
AUTHORITY_SCOPES=ui.read ui.admin
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.acme.internal
|
||||
```
|
||||
|
||||
The compose bundle includes Traefik as reverse proxy with TLS termination. Update `traefik/dynamic/console.yml` for custom certificates or additional middlewares (CSP headers, rate limits).
|
||||
|
||||
---
|
||||
|
||||
## 4. Environment variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `CONSOLE_PUBLIC_BASE_URL` | External URL used for redirects, deep links, and telemetry. | None (required). |
|
||||
| `CONSOLE_GATEWAY_BASE_URL` | URL of the web gateway that proxies API calls (`/console/*`). | Chart service name. |
|
||||
| `AUTHORITY_ISSUER` | Authority issuer (`https://authority.example.com`). | None (required). |
|
||||
| `AUTHORITY_CLIENT_ID` | OIDC client configured in Authority. | None (required). |
|
||||
| `AUTHORITY_SCOPES` | Space-separated scopes assigned to the console client. | `ui.read ui.admin`. |
|
||||
| `AUTHORITY_DPOP_ENABLED` | Enables DPoP challenge/response (recommended true). | `true`. |
|
||||
| `CONSOLE_FEATURE_FLAGS` | Comma-separated feature flags (`runs`, `downloads.offline`, etc.). | `runs,downloads,policies`. |
|
||||
| `CONSOLE_LOG_LEVEL` | Minimum log level (`Information`, `Debug`, etc.). | `Information`. |
|
||||
| `CONSOLE_METRICS_ENABLED` | Expose `/metrics` endpoint. | `true`. |
|
||||
| `CONSOLE_SENTRY_DSN` | Optional error reporting DSN. | Blank. |
|
||||
|
||||
When running behind additional proxies, set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true` to honour `X-Forwarded-*` headers.
|
||||
|
||||
---
|
||||
|
||||
## 5. Security headers and CSP
|
||||
|
||||
The console serves a strict Content Security Policy (CSP) by default:
|
||||
|
||||
```
|
||||
default-src 'self';
|
||||
connect-src 'self' https://*.stella-ops.local;
|
||||
script-src 'self';
|
||||
style-src 'self' 'unsafe-inline';
|
||||
img-src 'self' data:;
|
||||
font-src 'self';
|
||||
frame-ancestors 'none';
|
||||
```
|
||||
|
||||
Adjust via `console.config.cspOverrides` if additional domains are required. For integrations embedding the console, update OIDC redirect URIs and Authority scopes accordingly.
|
||||
|
||||
TLS recommendations:
|
||||
|
||||
- Use TLS 1.2+ with modern cipher suite policy.
|
||||
- Enable HSTS (`Strict-Transport-Security: max-age=31536000; includeSubDomains`).
|
||||
- Provide custom trust bundles via `console.config.trustBundleSecret` when using private CAs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logging and metrics
|
||||
|
||||
- Structured logs emitted to stdout with correlation IDs. Configure log shipping via Fluent Bit or similar.
|
||||
- Metrics available at `/metrics` in Prometheus format. Key metrics include `ui_request_duration_seconds`, `ui_tenant_switch_total`, and `ui_download_manifest_refresh_seconds`.
|
||||
- Enable OpenTelemetry exporter by setting `OTEL_EXPORTER_OTLP_ENDPOINT` and associated headers in environment variables.
|
||||
|
||||
---
|
||||
|
||||
## 7. Offline and air-gap deployment
|
||||
|
||||
- Mirror container images using the Downloads workspace or Offline Kit manifest. Example:
|
||||
|
||||
```bash
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> \
|
||||
registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
- Import Offline Kit using `stella ouk import` before starting the console so manifest parity checks succeed.
|
||||
- Use `values-airgap.yaml` to disable external telemetry endpoints and configure internal certificate chains.
|
||||
- Run `helm upgrade --install` using the mirrored chart (`stellaops-<ver>.tgz`) and set `console.offlineMode=true` to surface offline banners.
|
||||
|
||||
---
|
||||
|
||||
## 8. Health checks and remediation
|
||||
|
||||
| Check | Command | Expected result |
|
||||
|-------|---------|-----------------|
|
||||
| Pod status | `kubectl get pods -n stellaops` | `Running` state with restarts = 0. |
|
||||
| Liveness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/live` | Returns `{"status":"Healthy"}`. |
|
||||
| Readiness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/ready` | Returns `{"status":"Ready"}`. |
|
||||
| Gateway reachability | `curl -I https://console.example.com/api/console/status` | `200 OK` with CSP headers. |
|
||||
| Static assets | `curl -I https://console.example.com/static/assets/app.js` | `200 OK` with long cache headers. |
|
||||
|
||||
Troubleshooting steps:
|
||||
|
||||
- **Authority unreachable:** readiness fails with `AUTHORITY_UNREACHABLE`. Check DNS, trust bundles, and Authority service health.
|
||||
- **Manifest mismatch:** console logs `DOWNLOAD_MANIFEST_SIGNATURE_INVALID`. Verify cosign key and re-sync manifest.
|
||||
- **Ingress 404:** ensure ingress controller routes host to `stellaops-web-ui` service; check TLS secret name.
|
||||
- **SSE blocked:** confirm proxy allows HTTP/1.1 and disables buffering on `/console/runs/*`.
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
- `deploy/helm/stellaops/values-*.yaml` - environment-specific overrides.
|
||||
- `deploy/compose/docker-compose.console.yaml` - Compose bundle.
|
||||
- `/docs/ui/downloads.md` - manifest and offline bundle guidance.
|
||||
- `/docs/security/console-security.md` - CSP and Authority scopes.
|
||||
- `/docs/24_OFFLINE_KIT.md` - Offline kit packaging and verification.
|
||||
- `/docs/modules/devops/runbooks/deployment-runbook.md` (pending) - wider platform deployment steps.
|
||||
|
||||
---
|
||||
|
||||
## 10. Compliance checklist
|
||||
|
||||
- [ ] Helm and Compose instructions verified against `deploy/` assets.
|
||||
- [ ] Ingress/TLS guidance aligns with Security Guild recommendations.
|
||||
- [ ] Environment variables documented with defaults and required values.
|
||||
- [ ] Health/liveness/readiness endpoints tested and listed.
|
||||
- [ ] Offline workflow (mirrors, manifest parity) captured.
|
||||
- [ ] Logging and metrics surface documented metrics.
|
||||
- [ ] CSP and security header defaults stated alongside override guidance.
|
||||
- [ ] Troubleshooting section linked to relevant runbooks.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
# Deploying the StellaOps Console
|
||||
|
||||
> **Audience:** Deployment Guild, Console Guild, operators rolling out the web console.
|
||||
> **Scope:** Helm and Docker Compose deployment steps, ingress/TLS configuration, required environment variables, health checks, offline/air-gap operation, and compliance checklist (Sprint 23).
|
||||
|
||||
The StellaOps Console ships as part of the `stellaops` stack Helm chart and Compose bundles maintained under `deploy/`. This guide describes the supported deployment paths, the configuration surface, and operational checks needed to run the console in connected or air-gapped environments.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.28+) with ingress controller (NGINX, Traefik, or equivalent) and Cert-Manager for automated TLS, or Docker host for Compose deployments.
|
||||
- Container registry access to `registry.stella-ops.org` (or mirrored registry) for all images listed in `deploy/releases/*.yaml`.
|
||||
- Authority service configured with console client (`aud=ui`, scopes `ui.read`, `ui.admin`).
|
||||
- DNS entry pointing to the console hostname (for example, `console.acme.internal`).
|
||||
- Cosign public key for manifest verification (`deploy/releases/manifest.json.sig`).
|
||||
- Optional: Offline Kit bundle for air-gapped sites (`stella-ops-offline-kit-<ver>.tar.gz`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Helm deployment (recommended)
|
||||
|
||||
### 2.1 Install chart repository
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://downloads.stella-ops.org/helm
|
||||
helm repo update stellaops
|
||||
```
|
||||
|
||||
If operating offline, copy the chart archive from the Offline Kit (`deploy/helm/stellaops-<ver>.tgz`) and run:
|
||||
|
||||
```bash
|
||||
helm install stellaops ./stellaops-<ver>.tgz --namespace stellaops --create-namespace
|
||||
```
|
||||
|
||||
### 2.2 Base installation
|
||||
|
||||
```bash
|
||||
helm install stellaops stellaops/stellaops \
|
||||
--namespace stellaops \
|
||||
--create-namespace \
|
||||
--values deploy/helm/stellaops/values-prod.yaml
|
||||
```
|
||||
|
||||
The chart deploys Authority, Console web/API gateway, Scanner API, Scheduler, and supporting services. The console frontend pod is labelled `app=stellaops-web-ui`.
|
||||
|
||||
### 2.3 Helm values highlights
|
||||
|
||||
Key sections in `deploy/helm/stellaops/values-prod.yaml`:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `console.ingress.host` | Hostname served by the console (`console.example.com`). |
|
||||
| `console.ingress.tls.secretName` | Kubernetes secret containing TLS certificate (generated by Cert-Manager or uploaded manually). |
|
||||
| `console.config.apiGateway.baseUrl` | Internal base URL the UI uses to reach the gateway (defaults to `https://stellaops-web`). |
|
||||
| `console.env.AUTHORITY_ISSUER` | Authority issuer URL (for example, `https://authority.example.com`). |
|
||||
| `console.env.AUTHORITY_CLIENT_ID` | Authority client ID for the console UI. |
|
||||
| `console.env.AUTHORITY_SCOPES` | Space-separated scopes required by UI (`ui.read ui.admin`). |
|
||||
| `console.resources` | CPU/memory requests and limits (default 250m CPU / 512Mi memory). |
|
||||
| `console.podAnnotations` | Optional annotations for service mesh or monitoring. |
|
||||
|
||||
Use `values-stage.yaml`, `values-dev.yaml`, or `values-airgap.yaml` as templates for other environments.
|
||||
|
||||
### 2.4 TLS and ingress
|
||||
|
||||
Example ingress override:
|
||||
|
||||
```yaml
|
||||
console:
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
host: console.acme.internal
|
||||
tls:
|
||||
enabled: true
|
||||
secretName: console-tls
|
||||
```
|
||||
|
||||
Generate certificates using Cert-Manager or provide an existing secret. For air-gapped deployments, pre-create the secret with the mirrored CA chain.
|
||||
|
||||
### 2.5 Health checks
|
||||
|
||||
Console pods expose:
|
||||
|
||||
| Path | Purpose | Notes |
|
||||
|------|---------|-------|
|
||||
| `/health/live` | Liveness probe | Confirms process responsive. |
|
||||
| `/health/ready` | Readiness probe | Verifies configuration bootstrap and Authority reachability. |
|
||||
| `/metrics` | Prometheus metrics | Enabled when `console.metrics.enabled=true`. |
|
||||
|
||||
Helm chart sets default probes (`initialDelaySeconds: 10`, `periodSeconds: 15`). Adjust via `console.livenessProbe` and `console.readinessProbe`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Docker Compose deployment
|
||||
|
||||
Located in `deploy/compose/docker-compose.console.yaml`. Quick start:
|
||||
|
||||
```bash
|
||||
cd deploy/compose
|
||||
docker compose -f docker-compose.console.yaml --env-file console.env up -d
|
||||
```
|
||||
|
||||
`console.env` should define:
|
||||
|
||||
```
|
||||
CONSOLE_PUBLIC_BASE_URL=https://console.acme.internal
|
||||
AUTHORITY_ISSUER=https://authority.acme.internal
|
||||
AUTHORITY_CLIENT_ID=console-ui
|
||||
AUTHORITY_CLIENT_SECRET=<if using confidential client>
|
||||
AUTHORITY_SCOPES=ui.read ui.admin
|
||||
CONSOLE_GATEWAY_BASE_URL=https://api.acme.internal
|
||||
```
|
||||
|
||||
The compose bundle includes Traefik as reverse proxy with TLS termination. Update `traefik/dynamic/console.yml` for custom certificates or additional middlewares (CSP headers, rate limits).
|
||||
|
||||
---
|
||||
|
||||
## 4. Environment variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `CONSOLE_PUBLIC_BASE_URL` | External URL used for redirects, deep links, and telemetry. | None (required). |
|
||||
| `CONSOLE_GATEWAY_BASE_URL` | URL of the web gateway that proxies API calls (`/console/*`). | Chart service name. |
|
||||
| `AUTHORITY_ISSUER` | Authority issuer (`https://authority.example.com`). | None (required). |
|
||||
| `AUTHORITY_CLIENT_ID` | OIDC client configured in Authority. | None (required). |
|
||||
| `AUTHORITY_SCOPES` | Space-separated scopes assigned to the console client. | `ui.read ui.admin`. |
|
||||
| `AUTHORITY_DPOP_ENABLED` | Enables DPoP challenge/response (recommended true). | `true`. |
|
||||
| `CONSOLE_FEATURE_FLAGS` | Comma-separated feature flags (`runs`, `downloads.offline`, etc.). | `runs,downloads,policies`. |
|
||||
| `CONSOLE_LOG_LEVEL` | Minimum log level (`Information`, `Debug`, etc.). | `Information`. |
|
||||
| `CONSOLE_METRICS_ENABLED` | Expose `/metrics` endpoint. | `true`. |
|
||||
| `CONSOLE_SENTRY_DSN` | Optional error reporting DSN. | Blank. |
|
||||
|
||||
When running behind additional proxies, set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true` to honour `X-Forwarded-*` headers.
|
||||
|
||||
---
|
||||
|
||||
## 5. Security headers and CSP
|
||||
|
||||
The console serves a strict Content Security Policy (CSP) by default:
|
||||
|
||||
```
|
||||
default-src 'self';
|
||||
connect-src 'self' https://*.stella-ops.local;
|
||||
script-src 'self';
|
||||
style-src 'self' 'unsafe-inline';
|
||||
img-src 'self' data:;
|
||||
font-src 'self';
|
||||
frame-ancestors 'none';
|
||||
```
|
||||
|
||||
Adjust via `console.config.cspOverrides` if additional domains are required. For integrations embedding the console, update OIDC redirect URIs and Authority scopes accordingly.
|
||||
|
||||
TLS recommendations:
|
||||
|
||||
- Use TLS 1.2+ with modern cipher suite policy.
|
||||
- Enable HSTS (`Strict-Transport-Security: max-age=31536000; includeSubDomains`).
|
||||
- Provide custom trust bundles via `console.config.trustBundleSecret` when using private CAs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logging and metrics
|
||||
|
||||
- Structured logs emitted to stdout with correlation IDs. Configure log shipping via Fluent Bit or similar.
|
||||
- Metrics available at `/metrics` in Prometheus format. Key metrics include `ui_request_duration_seconds`, `ui_tenant_switch_total`, and `ui_download_manifest_refresh_seconds`.
|
||||
- Enable OpenTelemetry exporter by setting `OTEL_EXPORTER_OTLP_ENDPOINT` and associated headers in environment variables.
|
||||
|
||||
---
|
||||
|
||||
## 7. Offline and air-gap deployment
|
||||
|
||||
- Mirror container images using the Downloads workspace or Offline Kit manifest. Example:
|
||||
|
||||
```bash
|
||||
oras copy registry.stella-ops.org/stellaops/web-ui@sha256:<digest> \
|
||||
registry.airgap.local/stellaops/web-ui:2025.10.0
|
||||
```
|
||||
|
||||
- Import Offline Kit using `stella ouk import` before starting the console so manifest parity checks succeed.
|
||||
- Use `values-airgap.yaml` to disable external telemetry endpoints and configure internal certificate chains.
|
||||
- Run `helm upgrade --install` using the mirrored chart (`stellaops-<ver>.tgz`) and set `console.offlineMode=true` to surface offline banners.
|
||||
|
||||
---
|
||||
|
||||
## 8. Health checks and remediation
|
||||
|
||||
| Check | Command | Expected result |
|
||||
|-------|---------|-----------------|
|
||||
| Pod status | `kubectl get pods -n stellaops` | `Running` state with restarts = 0. |
|
||||
| Liveness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/live` | Returns `{"status":"Healthy"}`. |
|
||||
| Readiness | `kubectl exec deploy/stellaops-web-ui -- curl -fsS http://localhost:8080/health/ready` | Returns `{"status":"Ready"}`. |
|
||||
| Gateway reachability | `curl -I https://console.example.com/api/console/status` | `200 OK` with CSP headers. |
|
||||
| Static assets | `curl -I https://console.example.com/static/assets/app.js` | `200 OK` with long cache headers. |
|
||||
|
||||
Troubleshooting steps:
|
||||
|
||||
- **Authority unreachable:** readiness fails with `AUTHORITY_UNREACHABLE`. Check DNS, trust bundles, and Authority service health.
|
||||
- **Manifest mismatch:** console logs `DOWNLOAD_MANIFEST_SIGNATURE_INVALID`. Verify cosign key and re-sync manifest.
|
||||
- **Ingress 404:** ensure ingress controller routes host to `stellaops-web-ui` service; check TLS secret name.
|
||||
- **SSE blocked:** confirm proxy allows HTTP/1.1 and disables buffering on `/console/runs/*`.
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
- `deploy/helm/stellaops/values-*.yaml` - environment-specific overrides.
|
||||
- `deploy/compose/docker-compose.console.yaml` - Compose bundle.
|
||||
- `/docs/ui/downloads.md` - manifest and offline bundle guidance.
|
||||
- `/docs/security/console-security.md` - CSP and Authority scopes.
|
||||
- `/docs/24_OFFLINE_KIT.md` - Offline kit packaging and verification.
|
||||
- `/docs/modules/devops/runbooks/deployment-runbook.md` (pending) - wider platform deployment steps.
|
||||
|
||||
---
|
||||
|
||||
## 10. Compliance checklist
|
||||
|
||||
- [ ] Helm and Compose instructions verified against `deploy/` assets.
|
||||
- [ ] Ingress/TLS guidance aligns with Security Guild recommendations.
|
||||
- [ ] Environment variables documented with defaults and required values.
|
||||
- [ ] Health/liveness/readiness endpoints tested and listed.
|
||||
- [ ] Offline workflow (mirrors, manifest parity) captured.
|
||||
- [ ] Logging and metrics surface documented metrics.
|
||||
- [ ] CSP and security header defaults stated alongside override guidance.
|
||||
- [ ] Troubleshooting section linked to relevant runbooks.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 23).*
|
||||
|
||||
@@ -1,160 +1,160 @@
|
||||
# Container Deployment Guide — AOC Update
|
||||
|
||||
> **Audience:** DevOps Guild, platform operators deploying StellaOps services.
|
||||
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.
|
||||
|
||||
This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/modules/devops/architecture.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Schema validator enablement
|
||||
|
||||
### 1.1 MongoDB validators
|
||||
|
||||
- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
|
||||
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
Resolve any reported rows prior to rollout.
|
||||
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:
|
||||
|
||||
```bash
|
||||
kubectl exec -n concelier deploy/concelier-mongo -- \
|
||||
mongo concelier ops/devops/scripts/apply-aoc-validators.js
|
||||
|
||||
kubectl exec -n excititor deploy/excititor-mongo -- \
|
||||
mongo excititor ops/devops/scripts/apply-aoc-validators.js
|
||||
```
|
||||
|
||||
- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
|
||||
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.
|
||||
|
||||
### 1.2 Migration order
|
||||
|
||||
1. Deploy validators in maintenance window.
|
||||
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
|
||||
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.
|
||||
|
||||
### 1.3 Supersedes backfill verification
|
||||
|
||||
1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
|
||||
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
|
||||
```bash
|
||||
mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
|
||||
```
|
||||
The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
|
||||
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
|
||||
```bash
|
||||
mongo concelier --quiet --eval '
|
||||
db.advisory_raw.aggregate([
|
||||
{ $match: { "upstream.upstream_id": { $exists: true } } },
|
||||
{ $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
|
||||
{ $limit: 5 },
|
||||
{ $project: { _id: 1, supersedes: 1 } }
|
||||
]).forEach(printjson)'
|
||||
```
|
||||
Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Container environment flags
|
||||
|
||||
Add the following environment variables to Concelier/Excititor deployments:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
|
||||
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
|
||||
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
|
||||
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
|
||||
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |
|
||||
|
||||
Compose snippet:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- AOC_GUARD_ENABLED=true
|
||||
- AOC_ALLOW_SUPERSEDES_RETROFIT=false
|
||||
- AOC_METRICS_ENABLED=true
|
||||
- AOC_TENANT_HEADER=X-Stella-Tenant
|
||||
- AOC_VERIFIER_USER=stella-aoc-verify
|
||||
```
|
||||
|
||||
Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Verifier identity
|
||||
|
||||
- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:
|
||||
|
||||
```yaml
|
||||
clients:
|
||||
- clientId: stella-aoc-verify
|
||||
grantTypes: [client_credentials]
|
||||
scopes: [aoc:verify, advisory:read, vex:read]
|
||||
tenants: [default]
|
||||
```
|
||||
|
||||
- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
|
||||
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
|
||||
- Rotate quarterly; document in `ops/authority-key-rotation.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Deployment steps
|
||||
|
||||
1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
|
||||
2. **Apply validators:** Run scripts per § 1.1.
|
||||
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
|
||||
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
|
||||
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
|
||||
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
|
||||
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
|
||||
8. **Record handoff:** Capture console screenshots and verification logs for release notes.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Offline Kit updates
|
||||
|
||||
- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
|
||||
- Include pre-generated verification reports for air-gapped deployments.
|
||||
- Document offline CLI workflow in bundle README referencing `docs/modules/cli/guides/cli-reference.md`.
|
||||
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Rollback plan
|
||||
|
||||
1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
|
||||
2. Remove validators with the migration script (`--remove`).
|
||||
3. Pause verification jobs to prevent noise.
|
||||
4. Investigate and remediate upstream issues before re-enabling guards.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Authority scopes & tenancy](../security/authority-scopes.md)
|
||||
- [Observability guide](../observability/observability.md)
|
||||
- [CLI AOC commands](../modules/cli/guides/cli-reference.md)
|
||||
- [Concelier architecture](../modules/concelier/architecture.md)
|
||||
- [Excititor architecture](../modules/excititor/architecture.md)
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance checklist
|
||||
|
||||
- [ ] Validators documented and scripts referenced for online/offline deployments.
|
||||
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
|
||||
- [ ] Read-only verifier user installation steps included.
|
||||
- [ ] Offline kit instructions align with validator/verification workflow.
|
||||
- [ ] Rollback procedure captured.
|
||||
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
|
||||
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
# Container Deployment Guide — AOC Update
|
||||
|
||||
> **Audience:** DevOps Guild, platform operators deploying StellaOps services.
|
||||
> **Scope:** Deployment configuration changes required by the Aggregation-Only Contract (AOC), including schema validators, guard environment flags, and verifier identities.
|
||||
|
||||
This guide supplements existing deployment manuals with AOC-specific configuration. It assumes familiarity with the base Compose/Helm manifests described in `ops/deployment/` and `docs/modules/devops/architecture.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Schema validator enablement
|
||||
|
||||
### 1.1 MongoDB validators
|
||||
|
||||
- Apply JSON schema validators to `advisory_raw` and `vex_raw` collections before enabling AOC guards.
|
||||
- Before enabling validators or the idempotency index, run the duplicate audit helper to confirm no conflicting raw advisories remain:
|
||||
```bash
|
||||
mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'
|
||||
```
|
||||
Resolve any reported rows prior to rollout.
|
||||
- Use the migration script provided in `ops/devops/scripts/apply-aoc-validators.js`:
|
||||
|
||||
```bash
|
||||
kubectl exec -n concelier deploy/concelier-mongo -- \
|
||||
mongo concelier ops/devops/scripts/apply-aoc-validators.js
|
||||
|
||||
kubectl exec -n excititor deploy/excititor-mongo -- \
|
||||
mongo excititor ops/devops/scripts/apply-aoc-validators.js
|
||||
```
|
||||
|
||||
- Validators enforce required fields (`tenant`, `source`, `upstream`, `linkset`) and reject forbidden keys at DB level.
|
||||
- Rollback plan: validators are applied with `validationLevel: moderate`—downgrade via the same script with `--remove` if required.
|
||||
|
||||
### 1.2 Migration order
|
||||
|
||||
1. Deploy validators in maintenance window.
|
||||
2. Roll out Concelier/Excititor images with guard middleware enabled (`AOC_GUARD_ENABLED=true`).
|
||||
3. Run smoke tests (`stella sources ingest --dry-run` fixtures) before resuming production ingestion.
|
||||
|
||||
### 1.3 Supersedes backfill verification
|
||||
|
||||
1. **Duplicate audit:** Confirm `mongo concelier ops/devops/scripts/check-advisory-raw-duplicates.js --eval 'var LIMIT=200;'` reports no conflicts before restarting Concelier with the new migrations.
|
||||
2. **Post-migration check:** After the service restarts, validate that `db.advisory` is a view pointing to `advisory_backup_20251028`:
|
||||
```bash
|
||||
mongo concelier --quiet --eval 'db.getCollectionInfos({ name: "advisory" })[0]'
|
||||
```
|
||||
The `type` should be `"view"` and `options.viewOn` should equal `"advisory_backup_20251028"`.
|
||||
3. **Supersedes chain spot-check:** Inspect a sample set to ensure deterministic chaining:
|
||||
```bash
|
||||
mongo concelier --quiet --eval '
|
||||
db.advisory_raw.aggregate([
|
||||
{ $match: { "upstream.upstream_id": { $exists: true } } },
|
||||
{ $sort: { "tenant": 1, "source.vendor": 1, "upstream.upstream_id": 1, "upstream.retrieved_at": 1 } },
|
||||
{ $limit: 5 },
|
||||
{ $project: { _id: 1, supersedes: 1 } }
|
||||
]).forEach(printjson)'
|
||||
```
|
||||
Each revision should reference the previous `_id` (or `null` for the first revision). Record findings in the change ticket before proceeding to production.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Container environment flags
|
||||
|
||||
Add the following environment variables to Concelier/Excititor deployments:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `AOC_GUARD_ENABLED` | `true` | Enables `AOCWriteGuard` interception. Set `false` only for controlled rollback. |
|
||||
| `AOC_ALLOW_SUPERSEDES_RETROFIT` | `false` | Allows temporary supersedes backfill during migration. Remove after cutover. |
|
||||
| `AOC_METRICS_ENABLED` | `true` | Emits `ingestion_write_total`, `aoc_violation_total`, etc. |
|
||||
| `AOC_TENANT_HEADER` | `X-Stella-Tenant` | Header name expected from Gateway. |
|
||||
| `AOC_VERIFIER_USER` | `stella-aoc-verify` | Read-only service user used by UI/CLI verification. |
|
||||
|
||||
Compose snippet:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- AOC_GUARD_ENABLED=true
|
||||
- AOC_ALLOW_SUPERSEDES_RETROFIT=false
|
||||
- AOC_METRICS_ENABLED=true
|
||||
- AOC_TENANT_HEADER=X-Stella-Tenant
|
||||
- AOC_VERIFIER_USER=stella-aoc-verify
|
||||
```
|
||||
|
||||
Ensure `AOC_VERIFIER_USER` exists in Authority with `aoc:verify` scope and no write permissions.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Verifier identity
|
||||
|
||||
- Create a dedicated client (`stella-aoc-verify`) via Authority bootstrap:
|
||||
|
||||
```yaml
|
||||
clients:
|
||||
- clientId: stella-aoc-verify
|
||||
grantTypes: [client_credentials]
|
||||
scopes: [aoc:verify, advisory:read, vex:read]
|
||||
tenants: [default]
|
||||
```
|
||||
|
||||
- Store credentials in secret store (`Kubernetes Secret`, `Docker swarm secret`).
|
||||
- Bind credentials to `stella aoc verify` CI jobs and Console verification service.
|
||||
- Rotate quarterly; document in `ops/authority-key-rotation.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Deployment steps
|
||||
|
||||
1. **Pre-checks:** Confirm database backups, alerting in maintenance mode, and staging environment validated.
|
||||
2. **Apply validators:** Run scripts per § 1.1.
|
||||
3. **Update manifests:** Inject environment variables (§ 2) and mount guard configuration configmaps.
|
||||
4. **Redeploy services:** Rolling restart Concelier/Excititor pods. Monitor `ingestion_write_total` for steady throughput.
|
||||
5. **Seed verifier:** Deploy read-only verifier user and store credentials.
|
||||
6. **Run verification:** Execute `stella aoc verify --since 24h` and ensure exit code `0`.
|
||||
7. **Update dashboards:** Point Grafana panels to new metrics (`aoc_violation_total`).
|
||||
8. **Record handoff:** Capture console screenshots and verification logs for release notes.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Offline Kit updates
|
||||
|
||||
- Ship validator scripts with Offline Kit (`offline-kit/scripts/apply-aoc-validators.js`).
|
||||
- Include pre-generated verification reports for air-gapped deployments.
|
||||
- Document offline CLI workflow in bundle README referencing `docs/modules/cli/guides/cli-reference.md`.
|
||||
- Ensure `stella-aoc-verify` credentials are scoped to offline tenant and rotated during bundle refresh.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Rollback plan
|
||||
|
||||
1. Disable guard via `AOC_GUARD_ENABLED=false` on Concelier/Excititor and rollout.
|
||||
2. Remove validators with the migration script (`--remove`).
|
||||
3. Pause verification jobs to prevent noise.
|
||||
4. Investigate and remediate upstream issues before re-enabling guards.
|
||||
|
||||
---
|
||||
|
||||
## 7 · References
|
||||
|
||||
- [Aggregation-Only Contract reference](../ingestion/aggregation-only-contract.md)
|
||||
- [Authority scopes & tenancy](../security/authority-scopes.md)
|
||||
- [Observability guide](../observability/observability.md)
|
||||
- [CLI AOC commands](../modules/cli/guides/cli-reference.md)
|
||||
- [Concelier architecture](../modules/concelier/architecture.md)
|
||||
- [Excititor architecture](../modules/excititor/architecture.md)
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance checklist
|
||||
|
||||
- [ ] Validators documented and scripts referenced for online/offline deployments.
|
||||
- [ ] Environment variables cover guard enablement, metrics, and tenant header.
|
||||
- [ ] Read-only verifier user installation steps included.
|
||||
- [ ] Offline kit instructions align with validator/verification workflow.
|
||||
- [ ] Rollback procedure captured.
|
||||
- [ ] Cross-links to AOC docs, Authority scopes, and observability guides present.
|
||||
- [ ] DevOps Guild sign-off tracked (owner: @devops-guild, due 2025-10-29).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-26 (Sprint 19).*
|
||||
|
||||
@@ -1,220 +1,220 @@
|
||||
# Excititor Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Excititor provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/modules/excititor/architecture.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Excititor connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Excititor Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/excititor-connector/`. It contains:
|
||||
|
||||
```
|
||||
Excititor.MyConnector/
|
||||
├── src/
|
||||
│ ├── Excititor.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Excititor.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Excititor.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Excititor.Connectors.Abstractions\StellaOps.Excititor.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("excititor:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Excititor Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Excititor plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: excititor-my-provider
|
||||
assembly: StellaOps.Excititor.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Excititor.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- excititor
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/excititor/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/excititor/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Excititor Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Excititor.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/excititor-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
# Excititor Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Excititor provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/modules/excititor/architecture.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Excititor connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Excititor Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/excititor-connector/`. It contains:
|
||||
|
||||
```
|
||||
Excititor.MyConnector/
|
||||
├── src/
|
||||
│ ├── Excititor.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Excititor.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Excititor.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Excititor.Connectors.Abstractions\StellaOps.Excititor.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("excititor:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Excititor Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Excititor plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: excititor-my-provider
|
||||
assembly: StellaOps.Excititor.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Excititor.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- excititor
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/excititor/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/excititor/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Excititor Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Excititor.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/excititor-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
|
||||
@@ -1,220 +1,220 @@
|
||||
# Vexer Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Vexer provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/modules/vexer/architecture.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Vexer connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Vexer Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/vexer-connector/`. It contains:
|
||||
|
||||
```
|
||||
Vexer.MyConnector/
|
||||
├── src/
|
||||
│ ├── Vexer.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Vexer.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Vexer.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("vexer:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Vexer Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Vexer plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: vexer-my-provider
|
||||
assembly: StellaOps.Vexer.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- vexer
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/vexer/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/vexer/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Vexer Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Vexer.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/vexer-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
# Vexer Connector Packaging Guide
|
||||
|
||||
> **Audience:** teams implementing new Vexer provider plug‑ins (CSAF feeds,
|
||||
> OpenVEX attestations, etc.)
|
||||
> **Prerequisites:** read `docs/modules/vexer/architecture.md` and the module
|
||||
> `AGENTS.md` in `src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/`.
|
||||
|
||||
The Vexer connector SDK gives you:
|
||||
|
||||
- `VexConnectorBase` – deterministic logging, SHA‑256 helpers, time provider.
|
||||
- `VexConnectorOptionsBinder` – strongly typed YAML/JSON configuration binding.
|
||||
- `IVexConnectorOptionsValidator<T>` – custom validation hooks (offline defaults, auth invariants).
|
||||
- `VexConnectorDescriptor` & metadata helpers for consistent telemetry.
|
||||
|
||||
This guide explains how to package a connector so the Vexer Worker/WebService
|
||||
can load it via the plugin host.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project layout
|
||||
|
||||
Start from the template under
|
||||
`docs/dev/templates/vexer-connector/`. It contains:
|
||||
|
||||
```
|
||||
Vexer.MyConnector/
|
||||
├── src/
|
||||
│ ├── Vexer.MyConnector.csproj
|
||||
│ ├── MyConnectorOptions.cs
|
||||
│ ├── MyConnector.cs
|
||||
│ └── MyConnectorPlugin.cs
|
||||
└── manifest/
|
||||
└── connector.manifest.yaml
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Target `net10.0`, enable `TreatWarningsAsErrors`, reference the
|
||||
`StellaOps.Vexer.Connectors.Abstractions` project (or NuGet once published).
|
||||
- Keep project ID prefix `StellaOps.Vexer.Connectors.<Provider>` so the
|
||||
plugin loader can discover it with the default search pattern.
|
||||
|
||||
### 1.1 csproj snippet
|
||||
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
|
||||
Adjust the `ProjectReference` for your checkout (or switch to a NuGet package
|
||||
once published).
|
||||
|
||||
---
|
||||
|
||||
## 2. Implement the connector
|
||||
|
||||
1. **Options model** – create an options POCO with data-annotation attributes.
|
||||
Bind it via `VexConnectorOptionsBinder.Bind<TOptions>` in your connector
|
||||
constructor or `ValidateAsync`.
|
||||
2. **Validator** – implement `IVexConnectorOptionsValidator<TOptions>` to add
|
||||
complex checks (e.g., ensure both `clientId` and `clientSecret` are present).
|
||||
3. **Connector** – inherit from `VexConnectorBase`. Implement:
|
||||
- `ValidateAsync` – run binder/validators, log configuration summary.
|
||||
- `FetchAsync` – stream raw documents to `context.RawSink`.
|
||||
- `NormalizeAsync` – convert raw documents into `VexClaimBatch` via
|
||||
format-specific normalizers (`context.Normalizers`).
|
||||
4. **Plugin adapter** – expose the connector via a plugin entry point so the
|
||||
host can instantiate it.
|
||||
|
||||
### 2.1 Options binding example
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorOptions
|
||||
{
|
||||
[Required]
|
||||
[Url]
|
||||
public string CatalogUri { get; set; } = default!;
|
||||
|
||||
[Required]
|
||||
public string ApiKey { get; set; } = default!;
|
||||
|
||||
[Range(1, 64)]
|
||||
public int MaxParallelRequests { get; set; } = 4;
|
||||
}
|
||||
|
||||
public sealed class MyConnectorOptionsValidator : IVexConnectorOptionsValidator<MyConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, MyConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
if (!options.CatalogUri.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errors.Add("CatalogUri must use HTTPS.");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Bind inside the connector:
|
||||
|
||||
```csharp
|
||||
private readonly MyConnectorOptions _options;
|
||||
|
||||
public MyConnector(VexConnectorDescriptor descriptor, ILogger<MyConnector> logger, TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
// `settings` comes from the orchestrator; validators registered via DI.
|
||||
_options = VexConnectorOptionsBinder.Bind<MyConnectorOptions>(
|
||||
descriptor,
|
||||
VexConnectorSettings.Empty,
|
||||
validators: new[] { new MyConnectorOptionsValidator() });
|
||||
}
|
||||
```
|
||||
|
||||
Replace `VexConnectorSettings.Empty` with the actual settings from context
|
||||
inside `ValidateAsync`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Plugin adapter & manifest
|
||||
|
||||
Create a simple plugin class that implements
|
||||
`StellaOps.Plugin.IConnectorPlugin`. The Worker/WebService plugin host uses
|
||||
this contract today.
|
||||
|
||||
```csharp
|
||||
public sealed class MyConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor =
|
||||
new("vexer:my-provider", VexProviderKind.Vendor, "My Provider VEX");
|
||||
|
||||
public string Name => Descriptor.DisplayName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => true; // inject feature flags if needed
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
var logger = services.GetRequiredService<ILogger<MyConnector>>();
|
||||
var timeProvider = services.GetRequiredService<TimeProvider>();
|
||||
return new MyConnector(Descriptor, logger, timeProvider);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** the Vexer Worker currently instantiates connectors through the
|
||||
> shared `IConnectorPlugin` contract. Once a dedicated Vexer plugin interface
|
||||
> lands you simply swap the base interface; the descriptor/connector code
|
||||
> remains unchanged.
|
||||
|
||||
Provide a manifest describing the assembly for operational tooling:
|
||||
|
||||
```yaml
|
||||
# manifest/connector.manifest.yaml
|
||||
id: vexer-my-provider
|
||||
assembly: StellaOps.Vexer.Connectors.MyProvider.dll
|
||||
entryPoint: StellaOps.Vexer.Connectors.MyProvider.MyConnectorPlugin
|
||||
description: >
|
||||
Official VEX feed for ExampleCorp products (CSAF JSON, daily updates).
|
||||
tags:
|
||||
- vexer
|
||||
- csaf
|
||||
- vendor
|
||||
```
|
||||
|
||||
Store manifests under `/opt/stella/vexer/plugins/<connector>/manifest/` in
|
||||
production so the deployment tooling can inventory and verify plug‑ins.
|
||||
|
||||
---
|
||||
|
||||
## 4. Packaging workflow
|
||||
|
||||
1. `dotnet publish -c Release` → copy the published DLLs to
|
||||
`/opt/stella/vexer/plugins/<Provider>/`.
|
||||
2. Place `connector.manifest.yaml` next to the binaries.
|
||||
3. Restart the Vexer Worker or WebService (hot reload not supported yet).
|
||||
4. Verify logs: `VEX-ConnectorLoader` should list the connector descriptor.
|
||||
|
||||
### 4.1 Offline kits
|
||||
|
||||
- Add the connector folder (binaries + manifest) to the Offline Kit bundle.
|
||||
- Include a `settings.sample.yaml` demonstrating offline-friendly defaults.
|
||||
- Document any external dependencies (e.g., SHA mirrors) in the manifest `notes`
|
||||
field.
|
||||
|
||||
---
|
||||
|
||||
## 5. Testing checklist
|
||||
|
||||
- Unit tests around options binding & validators.
|
||||
- Integration tests (future `StellaOps.Vexer.Connectors.Abstractions.Tests`)
|
||||
verifying deterministic logging scopes:
|
||||
`logger.BeginScope` should produce `vex.connector.id`, `vex.connector.kind`,
|
||||
and `vex.connector.operation`.
|
||||
- Deterministic SHA tests: repeated `CreateRawDocument` calls with identical
|
||||
content must return the same digest.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference template
|
||||
|
||||
See `docs/dev/templates/vexer-connector/` for the full quick‑start including:
|
||||
|
||||
- Sample options class + validator.
|
||||
- Connector implementation inheriting from `VexConnectorBase`.
|
||||
- Plugin adapter + manifest.
|
||||
|
||||
Copy the directory, rename namespaces/IDs, then iterate on provider-specific
|
||||
logic.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-17*
|
||||
|
||||
@@ -1,212 +1,212 @@
|
||||
# Authority Plug-in Developer Guide
|
||||
|
||||
> **Status:** Updated 2025-10-11 (AUTHPLUG-DOCS-01-001) with lifecycle + limiter diagrams and refreshed rate-limit guidance aligned to PLG6 acceptance criteria.
|
||||
|
||||
## 1. Overview
|
||||
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Concelier plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
|
||||
|
||||
- **Use cases:** integrate corporate directories (LDAP/AD)[^ldap-rfc], delegate to external IDPs, enforce bespoke password/lockout policies, or add client provisioning automation.
|
||||
- **Constraints:** plug-ins load only during service start (no hot-reload), must function without outbound internet access, and must emit deterministic results for identical configuration input.
|
||||
- **Ship targets:** build against the host’s .NET 10 preview SDK, honour offline-first requirements, and surface actionable diagnostics so operators can triage issues from `/ready`.
|
||||
|
||||
## 2. Architecture Snapshot
|
||||
Authority hosts follow a deterministic plug-in lifecycle. The exported diagram (`docs/assets/authority/authority-plugin-lifecycle.svg`) mirrors the steps below; regenerate it from the Mermaid source if you update the flow.
|
||||
|
||||
1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`.
|
||||
2. **Assembly discovery** – the shared `PluginHost` scans `StellaOps.Authority.PluginBinaries` for `StellaOps.Authority.Plugin.*.dll` assemblies.
|
||||
3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
|
||||
4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-plugin-lifecycle.mmd`
|
||||
|
||||
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
|
||||
|
||||
## 3. Capability Metadata
|
||||
Capability flags let the host reason about what your plug-in supports:
|
||||
|
||||
- Declare capabilities in your descriptor using the string constants from `AuthorityPluginCapabilities` (`password`, `mfa`, `clientProvisioning`, `bootstrap`). The configuration loader now validates these tokens and rejects unknown values at startup.
|
||||
- `AuthorityIdentityProviderCapabilities.FromCapabilities` projects those strings into strongly typed booleans (`SupportsPassword`, etc.). Authority Core will use these flags when wiring flows such as the password grant. Built-in plugins (e.g., Standard) will fail fast or force-enable required capabilities if the descriptor is misconfigured, so keep manifests accurate.
|
||||
- Typical configuration (`etc/authority.plugins/standard.yaml`):
|
||||
```yaml
|
||||
plugins:
|
||||
descriptors:
|
||||
standard:
|
||||
assemblyName: "StellaOps.Authority.Plugin.Standard"
|
||||
capabilities:
|
||||
- password
|
||||
- bootstrap
|
||||
```
|
||||
- Only declare a capability if the plug-in genuinely implements it. For example, if `SupportsClientProvisioning` is `true`, the plug-in must supply a working `IClientProvisioningStore`.
|
||||
|
||||
**Operational reminder:** the Authority host surfaces capability summaries during startup (see `AuthorityIdentityProviderRegistry` log lines). Use those logs during smoke tests to ensure manifests align with expectations.
|
||||
|
||||
**Configuration path normalisation:** Manifest-relative paths (e.g., `tokenSigning.keyDirectory: "../keys"`) are resolved against the YAML file location and environment variables are expanded before validation. Plug-ins should expect to receive an absolute, canonical path when options are injected.
|
||||
|
||||
**Password policy guardrails:** The Standard registrar logs a warning when a plug-in weakens the default password policy (minimum length or required character classes). Keep overrides at least as strong as the compiled defaults—operators treat the warning as an actionable security deviation.
|
||||
|
||||
## 4. Project Scaffold
|
||||
- Target **.NET 10 preview**, enable nullable, treat warnings as errors, and mark Authority plug-ins with `<IsAuthorityPlugin>true</IsAuthorityPlugin>`.
|
||||
- Minimum references:
|
||||
- `StellaOps.Authority.Plugins.Abstractions` (contracts & capability helpers)
|
||||
- `StellaOps.Plugin` (hosting/DI helpers)
|
||||
- `StellaOps.Auth.*` libraries as needed for shared token utilities (optional today).
|
||||
- Example `.csproj` (trimmed from `StellaOps.Authority.Plugin.Standard`):
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<IsAuthorityPlugin>true</IsAuthorityPlugin>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Authority.Plugins.Abstractions\StellaOps.Authority.Plugins.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\..\StellaOps.Plugin\StellaOps.Plugin.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
(Add other references—e.g., MongoDB driver, shared auth libraries—according to your implementation.)
|
||||
|
||||
## 5. Implementing `IAuthorityPluginRegistrar`
|
||||
- Create a parameterless registrar class that returns your plug-in type name via `PluginType`.
|
||||
- Use `AuthorityPluginRegistrationContext` to:
|
||||
- Bind options (`AddOptions<T>(pluginName).Bind(...)`).
|
||||
- Register singletons for stores/enrichers using manifest metadata.
|
||||
- Register any hosted bootstrap tasks (e.g., seed admin users).
|
||||
- Always validate configuration inside `PostConfigure` and throw meaningful `InvalidOperationException` to fail fast during startup.
|
||||
- Use the provided `ILoggerFactory` from DI; avoid static loggers or console writes.
|
||||
- Example skeleton:
|
||||
```csharp
|
||||
internal sealed class MyPluginRegistrar : IAuthorityPluginRegistrar
|
||||
{
|
||||
public string PluginType => "my-custom";
|
||||
|
||||
public void Register(AuthorityPluginRegistrationContext context)
|
||||
{
|
||||
var name = context.Plugin.Manifest.Name;
|
||||
|
||||
context.Services.AddOptions<MyPluginOptions>(name)
|
||||
.Bind(context.Plugin.Configuration)
|
||||
.PostConfigure(opts => opts.Validate(name));
|
||||
|
||||
context.Services.AddSingleton<IIdentityProviderPlugin>(sp =>
|
||||
new MyIdentityProvider(context.Plugin, sp.GetRequiredService<MyCredentialStore>(),
|
||||
sp.GetRequiredService<MyClaimsEnricher>(),
|
||||
sp.GetRequiredService<ILogger<MyIdentityProvider>>()));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Identity Provider Surface
|
||||
- Implement `IIdentityProviderPlugin` to expose:
|
||||
- `IUserCredentialStore` for password validation and user CRUD.
|
||||
- `IClaimsEnricher` to append roles/attributes onto issued principals.
|
||||
- Optional `IClientProvisioningStore` for machine-to-machine clients.
|
||||
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
|
||||
- Password guidance:
|
||||
- Standard plug-in hashes via `ICryptoProvider` using Argon2id by default and emits PHC-compliant strings. Successful PBKDF2 logins trigger automatic rehashes so migrations complete gradually. See `docs/security/password-hashing.md` for tuning advice.
|
||||
- Enforce password policies before hashing to avoid storing weak credentials.
|
||||
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
|
||||
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
|
||||
|
||||
## 7. Configuration & Secrets
|
||||
- Authority looks for manifests under `etc/authority.plugins/`. Each YAML file maps directly to a plug-in name.
|
||||
- Support environment overrides using `STELLAOPS_AUTHORITY_PLUGINS__DESCRIPTORS__<NAME>__...`.
|
||||
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
|
||||
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
|
||||
- Cross-reference bootstrap workflows with `docs/modules/authority/operations/bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
|
||||
- `passwordHashing` inherits defaults from `authority.security.passwordHashing`. Override only when hardware constraints differ per plug-in:
|
||||
```yaml
|
||||
passwordHashing:
|
||||
algorithm: Argon2id
|
||||
memorySizeInKib: 19456
|
||||
iterations: 2
|
||||
parallelism: 1
|
||||
```
|
||||
Invalid values (≤0) fail fast during startup, and legacy PBKDF2 hashes rehash automatically once the new algorithm succeeds.
|
||||
|
||||
### 7.1 Token Persistence Contract
|
||||
- The host automatically persists every issued principal (access, refresh, device, authorization code) in `authority_tokens`. Plug-in code **must not** bypass this store; use the provided `IAuthorityTokenStore` helpers when implementing custom flows.
|
||||
- When a plug-in disables a subject or client outside the standard handlers, call `IAuthorityTokenStore.UpdateStatusAsync(...)` for each affected token so revocation bundles stay consistent.
|
||||
- Supply machine-friendly `revokedReason` codes (`compromised`, `rotation`, `policy`, `lifecycle`, etc.) and optional `revokedMetadata` entries when invalidating credentials. These flow straight into `revocation-bundle.json` and should remain deterministic.
|
||||
- Token scopes should be normalised (trimmed, unique, ordinal sort) before returning from plug-in verification paths. `TokenPersistenceHandlers` will keep that ordering for downstream consumers.
|
||||
|
||||
### 7.2 Claims & Enrichment Checklist
|
||||
- Authority always sets the OpenID Connect basics: `sub`, `client_id`, `preferred_username`, optional `name`, and `role` (for password flows). Plug-ins must use `IClaimsEnricher` to append additional claims in a **deterministic** order (sort arrays, normalise casing) so resource servers can rely on stable shapes.
|
||||
- Recommended enrichment keys:
|
||||
- `stellaops.realm` – plug-in/tenant identifier so services can scope policies.
|
||||
- `stellaops.subject.type` – values such as `human`, `service`, `bootstrap`.
|
||||
- `groups` / `projects` – sorted arrays describing operator entitlements.
|
||||
- Claims visible in tokens should mirror what `/token` and `/userinfo` emit. Avoid injecting sensitive PII directly; mark values with `ClassifiedString.Personal` inside the plug-in so audit sinks can tag them appropriately.
|
||||
- For client-credential flows, remember to enrich both the client principal and the validation path (`TokenValidationHandlers`) so refresh flows keep the same metadata.
|
||||
|
||||
### 7.3 Revocation Bundles & Reasons
|
||||
- Use `IAuthorityRevocationStore` to record subject/client/token revocations when credentials are deleted or rotated. Stick to the standard categories (`token`, `subject`, `client`, `key`).
|
||||
- Include a deterministic `reason` string and optional `reasonDescription` so operators understand *why* a subject was revoked when inspecting bundles offline.
|
||||
- Plug-ins should populate `metadata` with stable keys (e.g., `revokedBy`, `sourcePlugin`, `ticketId`) to simplify SOC correlation. The keys must be lowercase, ASCII, and free of secrets—bundles are mirrored to air-gapped agents.
|
||||
|
||||
## 8. Rate Limiting & Lockout Interplay
|
||||
Rate limiting and account lockouts are complementary controls. Plug-ins must surface both deterministically so operators can correlate limiter hits with credential rejections.
|
||||
|
||||
**Baseline quotas** (from `docs/dev/authority-rate-limit-tuning-outline.md`):
|
||||
|
||||
| Endpoint | Default policy | Notes |
|
||||
|----------|----------------|-------|
|
||||
| `/token` | 30 requests / 60s, queue 0 | Drop to 10/60s for untrusted ranges; raise only with WAF + monitoring. |
|
||||
| `/authorize` | 60 requests / 60s, queue 10 | Reduce carefully; interactive UX depends on headroom. |
|
||||
| `/internal/*` | Disabled by default; recommended 5/60s when enabled | Keep queue 0 for bootstrap APIs. |
|
||||
|
||||
**Retry metadata:** The middleware stamps `Retry-After` plus tags `authority.client_id`, `authority.remote_ip`, and `authority.endpoint`. Plug-ins should keep these tags intact when crafting responses or telemetry so dashboards remain consistent.
|
||||
|
||||
**Lockout counters:** Treat lockouts as **subject-scoped** decisions. When multiple instances update counters, reuse the deterministic tie-breakers documented in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` (freshness overrides, precedence, and stable hashes) to avoid divergent lockout states across replicas.
|
||||
|
||||
**Alerting hooks:** Emit structured logs/metrics when either the limiter or credential store rejects access. Suggested gauges include `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and any custom `auth.plugins.<pluginName>.lockouts_total` counter.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd`
|
||||
|
||||
## 9. Logging, Metrics, and Diagnostics
|
||||
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
|
||||
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
|
||||
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
|
||||
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
|
||||
|
||||
## 10. Testing & Tooling
|
||||
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
|
||||
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
|
||||
- Smoke tests: launch `dotnet run --project src/Authority/StellaOps.Authority/StellaOps.Authority` with your plug-in under `StellaOps.Authority.PluginBinaries` and verify `/ready`.
|
||||
- Example verification snippet:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task VerifyPasswordAsync_ReturnsSuccess()
|
||||
{
|
||||
var store = CreateCredentialStore();
|
||||
await store.UpsertUserAsync(new AuthorityUserRegistration("alice", "Pa55!", null, null, false,
|
||||
Array.Empty<string>(), new Dictionary<string, string?>()), CancellationToken.None);
|
||||
|
||||
var result = await store.VerifyPasswordAsync("alice", "Pa55!", CancellationToken.None);
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.True(result.User?.Roles.Count == 0);
|
||||
}
|
||||
```
|
||||
|
||||
## 11. Packaging & Delivery
|
||||
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the host’s search pattern picks it up.
|
||||
- Place the compiled DLL plus dependencies under `StellaOps.Authority.PluginBinaries` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
|
||||
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
|
||||
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
|
||||
|
||||
[^ldap-rfc]: Lightweight Directory Access Protocol (LDAPv3) specification — [RFC 4511](https://datatracker.ietf.org/doc/html/rfc4511).
|
||||
|
||||
## 12. Checklist & Handoff
|
||||
- ✅ Capabilities declared and validated in automated tests.
|
||||
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
|
||||
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
|
||||
- ✅ Operational docs updated: configuration keys, secrets guidance, troubleshooting.
|
||||
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
|
||||
|
||||
---
|
||||
Mermaid sources for the embedded diagrams live under `docs/assets/authority/`. Regenerate the SVG assets with your preferred renderer before committing future updates so the visuals stay in sync with the `.mmd` definitions.
|
||||
# Authority Plug-in Developer Guide
|
||||
|
||||
> **Status:** Updated 2025-10-11 (AUTHPLUG-DOCS-01-001) with lifecycle + limiter diagrams and refreshed rate-limit guidance aligned to PLG6 acceptance criteria.
|
||||
|
||||
## 1. Overview
|
||||
Authority plug-ins extend the **StellaOps Authority** service with custom identity providers, credential stores, and client-management logic. Unlike Concelier plug-ins (which ingest or export advisories), Authority plug-ins participate directly in authentication flows:
|
||||
|
||||
- **Use cases:** integrate corporate directories (LDAP/AD)[^ldap-rfc], delegate to external IDPs, enforce bespoke password/lockout policies, or add client provisioning automation.
|
||||
- **Constraints:** plug-ins load only during service start (no hot-reload), must function without outbound internet access, and must emit deterministic results for identical configuration input.
|
||||
- **Ship targets:** build against the host’s .NET 10 preview SDK, honour offline-first requirements, and surface actionable diagnostics so operators can triage issues from `/ready`.
|
||||
|
||||
## 2. Architecture Snapshot
|
||||
Authority hosts follow a deterministic plug-in lifecycle. The exported diagram (`docs/assets/authority/authority-plugin-lifecycle.svg`) mirrors the steps below; regenerate it from the Mermaid source if you update the flow.
|
||||
|
||||
1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`.
|
||||
2. **Assembly discovery** – the shared `PluginHost` scans `StellaOps.Authority.PluginBinaries` for `StellaOps.Authority.Plugin.*.dll` assemblies.
|
||||
3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks.
|
||||
4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-plugin-lifecycle.mmd`
|
||||
|
||||
**Data persistence primer:** the standard Mongo-backed plugin stores users in collections named `authority_users_<pluginName>` and lockout metadata in embedded documents. Additional plugins must document their storage layout and provide deterministic collection naming to honour the Offline Kit replication process.
|
||||
|
||||
## 3. Capability Metadata
|
||||
Capability flags let the host reason about what your plug-in supports:
|
||||
|
||||
- Declare capabilities in your descriptor using the string constants from `AuthorityPluginCapabilities` (`password`, `mfa`, `clientProvisioning`, `bootstrap`). The configuration loader now validates these tokens and rejects unknown values at startup.
|
||||
- `AuthorityIdentityProviderCapabilities.FromCapabilities` projects those strings into strongly typed booleans (`SupportsPassword`, etc.). Authority Core will use these flags when wiring flows such as the password grant. Built-in plugins (e.g., Standard) will fail fast or force-enable required capabilities if the descriptor is misconfigured, so keep manifests accurate.
|
||||
- Typical configuration (`etc/authority.plugins/standard.yaml`):
|
||||
```yaml
|
||||
plugins:
|
||||
descriptors:
|
||||
standard:
|
||||
assemblyName: "StellaOps.Authority.Plugin.Standard"
|
||||
capabilities:
|
||||
- password
|
||||
- bootstrap
|
||||
```
|
||||
- Only declare a capability if the plug-in genuinely implements it. For example, if `SupportsClientProvisioning` is `true`, the plug-in must supply a working `IClientProvisioningStore`.
|
||||
|
||||
**Operational reminder:** the Authority host surfaces capability summaries during startup (see `AuthorityIdentityProviderRegistry` log lines). Use those logs during smoke tests to ensure manifests align with expectations.
|
||||
|
||||
**Configuration path normalisation:** Manifest-relative paths (e.g., `tokenSigning.keyDirectory: "../keys"`) are resolved against the YAML file location and environment variables are expanded before validation. Plug-ins should expect to receive an absolute, canonical path when options are injected.
|
||||
|
||||
**Password policy guardrails:** The Standard registrar logs a warning when a plug-in weakens the default password policy (minimum length or required character classes). Keep overrides at least as strong as the compiled defaults—operators treat the warning as an actionable security deviation.
|
||||
|
||||
## 4. Project Scaffold
|
||||
- Target **.NET 10 preview**, enable nullable, treat warnings as errors, and mark Authority plug-ins with `<IsAuthorityPlugin>true</IsAuthorityPlugin>`.
|
||||
- Minimum references:
|
||||
- `StellaOps.Authority.Plugins.Abstractions` (contracts & capability helpers)
|
||||
- `StellaOps.Plugin` (hosting/DI helpers)
|
||||
- `StellaOps.Auth.*` libraries as needed for shared token utilities (optional today).
|
||||
- Example `.csproj` (trimmed from `StellaOps.Authority.Plugin.Standard`):
|
||||
```xml
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<IsAuthorityPlugin>true</IsAuthorityPlugin>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Authority.Plugins.Abstractions\StellaOps.Authority.Plugins.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\..\StellaOps.Plugin\StellaOps.Plugin.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
```
|
||||
(Add other references—e.g., MongoDB driver, shared auth libraries—according to your implementation.)
|
||||
|
||||
## 5. Implementing `IAuthorityPluginRegistrar`
|
||||
- Create a parameterless registrar class that returns your plug-in type name via `PluginType`.
|
||||
- Use `AuthorityPluginRegistrationContext` to:
|
||||
- Bind options (`AddOptions<T>(pluginName).Bind(...)`).
|
||||
- Register singletons for stores/enrichers using manifest metadata.
|
||||
- Register any hosted bootstrap tasks (e.g., seed admin users).
|
||||
- Always validate configuration inside `PostConfigure` and throw meaningful `InvalidOperationException` to fail fast during startup.
|
||||
- Use the provided `ILoggerFactory` from DI; avoid static loggers or console writes.
|
||||
- Example skeleton:
|
||||
```csharp
|
||||
internal sealed class MyPluginRegistrar : IAuthorityPluginRegistrar
|
||||
{
|
||||
public string PluginType => "my-custom";
|
||||
|
||||
public void Register(AuthorityPluginRegistrationContext context)
|
||||
{
|
||||
var name = context.Plugin.Manifest.Name;
|
||||
|
||||
context.Services.AddOptions<MyPluginOptions>(name)
|
||||
.Bind(context.Plugin.Configuration)
|
||||
.PostConfigure(opts => opts.Validate(name));
|
||||
|
||||
context.Services.AddSingleton<IIdentityProviderPlugin>(sp =>
|
||||
new MyIdentityProvider(context.Plugin, sp.GetRequiredService<MyCredentialStore>(),
|
||||
sp.GetRequiredService<MyClaimsEnricher>(),
|
||||
sp.GetRequiredService<ILogger<MyIdentityProvider>>()));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Identity Provider Surface
|
||||
- Implement `IIdentityProviderPlugin` to expose:
|
||||
- `IUserCredentialStore` for password validation and user CRUD.
|
||||
- `IClaimsEnricher` to append roles/attributes onto issued principals.
|
||||
- Optional `IClientProvisioningStore` for machine-to-machine clients.
|
||||
- `AuthorityIdentityProviderCapabilities` to advertise supported flows.
|
||||
- Password guidance:
|
||||
- Standard plug-in hashes via `ICryptoProvider` using Argon2id by default and emits PHC-compliant strings. Successful PBKDF2 logins trigger automatic rehashes so migrations complete gradually. See `docs/security/password-hashing.md` for tuning advice.
|
||||
- Enforce password policies before hashing to avoid storing weak credentials.
|
||||
- Health checks should probe backing stores (e.g., Mongo `ping`) and return `AuthorityPluginHealthResult` so `/ready` can surface issues.
|
||||
- When supporting additional factors (e.g., TOTP), implement `SupportsMfa` and document the enrolment flow for resource servers.
|
||||
|
||||
## 7. Configuration & Secrets
|
||||
- Authority looks for manifests under `etc/authority.plugins/`. Each YAML file maps directly to a plug-in name.
|
||||
- Support environment overrides using `STELLAOPS_AUTHORITY_PLUGINS__DESCRIPTORS__<NAME>__...`.
|
||||
- Never store raw secrets in git: allow operators to supply them via `.local.yaml`, environment variables, or injected secret files. Document which keys are mandatory.
|
||||
- Validate configuration as soon as the registrar runs; use explicit error messages to guide operators. The Standard plug-in now enforces complete bootstrap credentials (username + password) and positive lockout windows via `StandardPluginOptions.Validate`.
|
||||
- Cross-reference bootstrap workflows with `docs/modules/authority/operations/bootstrap.md` (to be published alongside CORE6) so operators can reuse the same payload formats for manual provisioning.
|
||||
- `passwordHashing` inherits defaults from `authority.security.passwordHashing`. Override only when hardware constraints differ per plug-in:
|
||||
```yaml
|
||||
passwordHashing:
|
||||
algorithm: Argon2id
|
||||
memorySizeInKib: 19456
|
||||
iterations: 2
|
||||
parallelism: 1
|
||||
```
|
||||
Invalid values (≤0) fail fast during startup, and legacy PBKDF2 hashes rehash automatically once the new algorithm succeeds.
|
||||
|
||||
### 7.1 Token Persistence Contract
|
||||
- The host automatically persists every issued principal (access, refresh, device, authorization code) in `authority_tokens`. Plug-in code **must not** bypass this store; use the provided `IAuthorityTokenStore` helpers when implementing custom flows.
|
||||
- When a plug-in disables a subject or client outside the standard handlers, call `IAuthorityTokenStore.UpdateStatusAsync(...)` for each affected token so revocation bundles stay consistent.
|
||||
- Supply machine-friendly `revokedReason` codes (`compromised`, `rotation`, `policy`, `lifecycle`, etc.) and optional `revokedMetadata` entries when invalidating credentials. These flow straight into `revocation-bundle.json` and should remain deterministic.
|
||||
- Token scopes should be normalised (trimmed, unique, ordinal sort) before returning from plug-in verification paths. `TokenPersistenceHandlers` will keep that ordering for downstream consumers.
|
||||
|
||||
### 7.2 Claims & Enrichment Checklist
|
||||
- Authority always sets the OpenID Connect basics: `sub`, `client_id`, `preferred_username`, optional `name`, and `role` (for password flows). Plug-ins must use `IClaimsEnricher` to append additional claims in a **deterministic** order (sort arrays, normalise casing) so resource servers can rely on stable shapes.
|
||||
- Recommended enrichment keys:
|
||||
- `stellaops.realm` – plug-in/tenant identifier so services can scope policies.
|
||||
- `stellaops.subject.type` – values such as `human`, `service`, `bootstrap`.
|
||||
- `groups` / `projects` – sorted arrays describing operator entitlements.
|
||||
- Claims visible in tokens should mirror what `/token` and `/userinfo` emit. Avoid injecting sensitive PII directly; mark values with `ClassifiedString.Personal` inside the plug-in so audit sinks can tag them appropriately.
|
||||
- For client-credential flows, remember to enrich both the client principal and the validation path (`TokenValidationHandlers`) so refresh flows keep the same metadata.
|
||||
|
||||
### 7.3 Revocation Bundles & Reasons
|
||||
- Use `IAuthorityRevocationStore` to record subject/client/token revocations when credentials are deleted or rotated. Stick to the standard categories (`token`, `subject`, `client`, `key`).
|
||||
- Include a deterministic `reason` string and optional `reasonDescription` so operators understand *why* a subject was revoked when inspecting bundles offline.
|
||||
- Plug-ins should populate `metadata` with stable keys (e.g., `revokedBy`, `sourcePlugin`, `ticketId`) to simplify SOC correlation. The keys must be lowercase, ASCII, and free of secrets—bundles are mirrored to air-gapped agents.
|
||||
|
||||
## 8. Rate Limiting & Lockout Interplay
|
||||
Rate limiting and account lockouts are complementary controls. Plug-ins must surface both deterministically so operators can correlate limiter hits with credential rejections.
|
||||
|
||||
**Baseline quotas** (from `docs/dev/authority-rate-limit-tuning-outline.md`):
|
||||
|
||||
| Endpoint | Default policy | Notes |
|
||||
|----------|----------------|-------|
|
||||
| `/token` | 30 requests / 60s, queue 0 | Drop to 10/60s for untrusted ranges; raise only with WAF + monitoring. |
|
||||
| `/authorize` | 60 requests / 60s, queue 10 | Reduce carefully; interactive UX depends on headroom. |
|
||||
| `/internal/*` | Disabled by default; recommended 5/60s when enabled | Keep queue 0 for bootstrap APIs. |
|
||||
|
||||
**Retry metadata:** The middleware stamps `Retry-After` plus tags `authority.client_id`, `authority.remote_ip`, and `authority.endpoint`. Plug-ins should keep these tags intact when crafting responses or telemetry so dashboards remain consistent.
|
||||
|
||||
**Lockout counters:** Treat lockouts as **subject-scoped** decisions. When multiple instances update counters, reuse the deterministic tie-breakers documented in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` (freshness overrides, precedence, and stable hashes) to avoid divergent lockout states across replicas.
|
||||
|
||||
**Alerting hooks:** Emit structured logs/metrics when either the limiter or credential store rejects access. Suggested gauges include `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and any custom `auth.plugins.<pluginName>.lockouts_total` counter.
|
||||
|
||||
|
||||
|
||||
_Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd`
|
||||
|
||||
## 9. Logging, Metrics, and Diagnostics
|
||||
- Always log via the injected `ILogger<T>`; include `pluginName` and correlation IDs where available.
|
||||
- Activity/metric names should align with `AuthorityTelemetry` constants (`service.name=stellaops-authority`).
|
||||
- Expose additional diagnostics via structured logging rather than writing custom HTTP endpoints; the host will integrate these into `/health` and `/ready`.
|
||||
- Emit metrics with stable names (`auth.plugins.<pluginName>.*`) when introducing custom instrumentation; coordinate with the Observability guild to reserve prefixes.
|
||||
|
||||
## 10. Testing & Tooling
|
||||
- Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template).
|
||||
- Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable.
|
||||
- Smoke tests: launch `dotnet run --project src/Authority/StellaOps.Authority/StellaOps.Authority` with your plug-in under `StellaOps.Authority.PluginBinaries` and verify `/ready`.
|
||||
- Example verification snippet:
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task VerifyPasswordAsync_ReturnsSuccess()
|
||||
{
|
||||
var store = CreateCredentialStore();
|
||||
await store.UpsertUserAsync(new AuthorityUserRegistration("alice", "Pa55!", null, null, false,
|
||||
Array.Empty<string>(), new Dictionary<string, string?>()), CancellationToken.None);
|
||||
|
||||
var result = await store.VerifyPasswordAsync("alice", "Pa55!", CancellationToken.None);
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.True(result.User?.Roles.Count == 0);
|
||||
}
|
||||
```
|
||||
|
||||
## 11. Packaging & Delivery
|
||||
- Output assembly should follow `StellaOps.Authority.Plugin.<Name>.dll` so the host’s search pattern picks it up.
|
||||
- Place the compiled DLL plus dependencies under `StellaOps.Authority.PluginBinaries` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming).
|
||||
- Document any external prerequisites (e.g., CA cert bundle) in your plug-in README.
|
||||
- Update `etc/authority.plugins/<plugin>.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts.
|
||||
|
||||
[^ldap-rfc]: Lightweight Directory Access Protocol (LDAPv3) specification — [RFC 4511](https://datatracker.ietf.org/doc/html/rfc4511).
|
||||
|
||||
## 12. Checklist & Handoff
|
||||
- ✅ Capabilities declared and validated in automated tests.
|
||||
- ✅ Bootstrap workflows documented (if `bootstrap` capability used) and repeatable.
|
||||
- ✅ Local smoke test + unit/integration suites green (`dotnet test`).
|
||||
- ✅ Operational docs updated: configuration keys, secrets guidance, troubleshooting.
|
||||
- Submit the developer guide update referencing PLG6/DOC4 and tag DevEx + Docs reviewers for sign-off.
|
||||
|
||||
---
|
||||
Mermaid sources for the embedded diagrams live under `docs/assets/authority/`. Regenerate the SVG assets with your preferred renderer before committing future updates so the visuals stay in sync with the `.mmd` definitions.
|
||||
|
||||
@@ -1,119 +1,119 @@
|
||||
# BuildX Generator Quickstart
|
||||
|
||||
This quickstart explains how to run the StellaOps **BuildX SBOM generator** offline, verify the CAS handshake, and emit OCI descriptors that downstream services can attest.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Docker 25+ with BuildKit enabled (`docker buildx` available).
|
||||
- .NET 10 (preview) SDK matching the repository `global.json`.
|
||||
- Optional: network access to a StellaOps Attestor endpoint (the quickstart uses a mock service).
|
||||
|
||||
## 2. Publish the plug-in binaries
|
||||
|
||||
The BuildX generator publishes as a .NET self-contained executable with its manifest under `plugins/scanner/buildx/`.
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
DOTNET_CLI_HOME="${PWD}/.dotnet" \
|
||||
dotnet publish src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj \
|
||||
-c Release \
|
||||
-o out/buildx
|
||||
```
|
||||
|
||||
- `out/buildx/` now contains `StellaOps.Scanner.Sbomer.BuildXPlugin.dll` and the manifest `stellaops.sbom-indexer.manifest.json`.
|
||||
- `plugins/scanner/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin/` receives the same artefacts for release packaging.
|
||||
- The CI pipeline also tars and signs (SHA-256 manifest) the OS analyzer plug-ins located under
|
||||
`plugins/scanner/analyzers/os/` so they ship alongside the BuildX generator artefacts.
|
||||
|
||||
## 3. Verify the CAS handshake
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll handshake \
|
||||
--manifest out/buildx \
|
||||
--cas out/cas
|
||||
```
|
||||
|
||||
The command performs a deterministic probe write (`sha256`) into the provided CAS directory and prints the resolved path.
|
||||
|
||||
## 4. Emit a descriptor + provenance placeholder
|
||||
|
||||
1. Build or identify the image you want to describe and capture its digest:
|
||||
|
||||
```bash
|
||||
docker buildx build --load -t stellaops/buildx-demo:ci samples/ci/buildx-demo
|
||||
DIGEST=$(docker image inspect stellaops/buildx-demo:ci --format '{{index .RepoDigests 0}}')
|
||||
```
|
||||
|
||||
2. Generate a CycloneDX SBOM for the built image (any tool works; here we use `docker sbom`):
|
||||
|
||||
```bash
|
||||
docker sbom stellaops/buildx-demo:ci --format cyclonedx-json > out/buildx-sbom.cdx.json
|
||||
```
|
||||
|
||||
3. Invoke the `descriptor` command, pointing at the SBOM file and optional metadata:
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--sbom-name buildx-sbom.cdx.json \
|
||||
--artifact-type application/vnd.stellaops.sbom.layer+json \
|
||||
--sbom-format cyclonedx-json \
|
||||
--sbom-kind inventory \
|
||||
--repository git.stella-ops.org/stellaops/buildx-demo \
|
||||
--build-ref $(git rev-parse HEAD) \
|
||||
> out/buildx-descriptor.json
|
||||
```
|
||||
|
||||
The output JSON captures:
|
||||
|
||||
- OCI artifact descriptor including size, digest, and annotations (`org.stellaops.*`).
|
||||
- Provenance placeholder (`expectedDsseSha256`, `nonce`, `attestorUri` when provided). `nonce` is derived deterministically from the image + SBOM metadata so repeated runs produce identical placeholders for identical inputs.
|
||||
- Generator metadata and deterministic timestamps.
|
||||
|
||||
## 5. (Optional) Send the placeholder to an Attestor
|
||||
|
||||
The plug-in can POST the descriptor metadata to an Attestor endpoint, returning once it receives an HTTP 202.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY' &
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
_ = self.rfile.read(int(self.headers.get('Content-Length', 0)))
|
||||
self.send_response(202); self.end_headers(); self.wfile.write(b'accepted')
|
||||
def log_message(self, fmt, *args):
|
||||
return
|
||||
server = HTTPServer(('127.0.0.1', 8085), Handler)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
PY
|
||||
MOCK_PID=$!
|
||||
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--attestor http://127.0.0.1:8085/provenance \
|
||||
--attestor-token "$STELLAOPS_ATTESTOR_TOKEN" \
|
||||
> out/buildx-descriptor.json
|
||||
|
||||
kill $MOCK_PID
|
||||
```
|
||||
|
||||
Set `STELLAOPS_ATTESTOR_TOKEN` (or pass `--attestor-token`) when the Attestor requires bearer authentication. Use `--attestor-insecure` for lab environments with self-signed certificates.
|
||||
|
||||
## 6. CI workflow example
|
||||
|
||||
A reusable GitHub Actions workflow is provided under `samples/ci/buildx-demo/github-actions-buildx-demo.yml`. It publishes the plug-in, runs the handshake, builds the demo image, emits a descriptor, and uploads both the descriptor and the mock-Attestor request as artefacts.
|
||||
|
||||
Add the workflow to your repository (or call it via `workflow_call`) and adjust the SBOM path + Attestor URL as needed. The workflow also re-runs the `descriptor` command and diffs the results (ignoring the `generatedAt` timestamp) so you catch regressions that would break deterministic CI use.
|
||||
|
||||
---
|
||||
|
||||
For deeper integration guidance (custom SBOM builders, exporting DSSE bundles), track ADRs in `docs/modules/scanner/architecture.md` §7 and follow upcoming Attestor API releases.
|
||||
# BuildX Generator Quickstart
|
||||
|
||||
This quickstart explains how to run the StellaOps **BuildX SBOM generator** offline, verify the CAS handshake, and emit OCI descriptors that downstream services can attest.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Docker 25+ with BuildKit enabled (`docker buildx` available).
|
||||
- .NET 10 (preview) SDK matching the repository `global.json`.
|
||||
- Optional: network access to a StellaOps Attestor endpoint (the quickstart uses a mock service).
|
||||
|
||||
## 2. Publish the plug-in binaries
|
||||
|
||||
The BuildX generator publishes as a .NET self-contained executable with its manifest under `plugins/scanner/buildx/`.
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
DOTNET_CLI_HOME="${PWD}/.dotnet" \
|
||||
dotnet publish src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/StellaOps.Scanner.Sbomer.BuildXPlugin.csproj \
|
||||
-c Release \
|
||||
-o out/buildx
|
||||
```
|
||||
|
||||
- `out/buildx/` now contains `StellaOps.Scanner.Sbomer.BuildXPlugin.dll` and the manifest `stellaops.sbom-indexer.manifest.json`.
|
||||
- `plugins/scanner/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin/` receives the same artefacts for release packaging.
|
||||
- The CI pipeline also tars and signs (SHA-256 manifest) the OS analyzer plug-ins located under
|
||||
`plugins/scanner/analyzers/os/` so they ship alongside the BuildX generator artefacts.
|
||||
|
||||
## 3. Verify the CAS handshake
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll handshake \
|
||||
--manifest out/buildx \
|
||||
--cas out/cas
|
||||
```
|
||||
|
||||
The command performs a deterministic probe write (`sha256`) into the provided CAS directory and prints the resolved path.
|
||||
|
||||
## 4. Emit a descriptor + provenance placeholder
|
||||
|
||||
1. Build or identify the image you want to describe and capture its digest:
|
||||
|
||||
```bash
|
||||
docker buildx build --load -t stellaops/buildx-demo:ci samples/ci/buildx-demo
|
||||
DIGEST=$(docker image inspect stellaops/buildx-demo:ci --format '{{index .RepoDigests 0}}')
|
||||
```
|
||||
|
||||
2. Generate a CycloneDX SBOM for the built image (any tool works; here we use `docker sbom`):
|
||||
|
||||
```bash
|
||||
docker sbom stellaops/buildx-demo:ci --format cyclonedx-json > out/buildx-sbom.cdx.json
|
||||
```
|
||||
|
||||
3. Invoke the `descriptor` command, pointing at the SBOM file and optional metadata:
|
||||
|
||||
```bash
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--sbom-name buildx-sbom.cdx.json \
|
||||
--artifact-type application/vnd.stellaops.sbom.layer+json \
|
||||
--sbom-format cyclonedx-json \
|
||||
--sbom-kind inventory \
|
||||
--repository git.stella-ops.org/stellaops/buildx-demo \
|
||||
--build-ref $(git rev-parse HEAD) \
|
||||
> out/buildx-descriptor.json
|
||||
```
|
||||
|
||||
The output JSON captures:
|
||||
|
||||
- OCI artifact descriptor including size, digest, and annotations (`org.stellaops.*`).
|
||||
- Provenance placeholder (`expectedDsseSha256`, `nonce`, `attestorUri` when provided). `nonce` is derived deterministically from the image + SBOM metadata so repeated runs produce identical placeholders for identical inputs.
|
||||
- Generator metadata and deterministic timestamps.
|
||||
|
||||
## 5. (Optional) Send the placeholder to an Attestor
|
||||
|
||||
The plug-in can POST the descriptor metadata to an Attestor endpoint, returning once it receives an HTTP 202.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY' &
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
_ = self.rfile.read(int(self.headers.get('Content-Length', 0)))
|
||||
self.send_response(202); self.end_headers(); self.wfile.write(b'accepted')
|
||||
def log_message(self, fmt, *args):
|
||||
return
|
||||
server = HTTPServer(('127.0.0.1', 8085), Handler)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
PY
|
||||
MOCK_PID=$!
|
||||
|
||||
dotnet out/buildx/StellaOps.Scanner.Sbomer.BuildXPlugin.dll descriptor \
|
||||
--manifest out/buildx \
|
||||
--image "$DIGEST" \
|
||||
--sbom out/buildx-sbom.cdx.json \
|
||||
--attestor http://127.0.0.1:8085/provenance \
|
||||
--attestor-token "$STELLAOPS_ATTESTOR_TOKEN" \
|
||||
> out/buildx-descriptor.json
|
||||
|
||||
kill $MOCK_PID
|
||||
```
|
||||
|
||||
Set `STELLAOPS_ATTESTOR_TOKEN` (or pass `--attestor-token`) when the Attestor requires bearer authentication. Use `--attestor-insecure` for lab environments with self-signed certificates.
|
||||
|
||||
## 6. CI workflow example
|
||||
|
||||
A reusable GitHub Actions workflow is provided under `samples/ci/buildx-demo/github-actions-buildx-demo.yml`. It publishes the plug-in, runs the handshake, builds the demo image, emits a descriptor, and uploads both the descriptor and the mock-Attestor request as artefacts.
|
||||
|
||||
Add the workflow to your repository (or call it via `workflow_call`) and adjust the SBOM path + Attestor URL as needed. The workflow also re-runs the `descriptor` command and diffs the results (ignoring the `generatedAt` timestamp) so you catch regressions that would break deterministic CI use.
|
||||
|
||||
---
|
||||
|
||||
For deeper integration guidance (custom SBOM builders, exporting DSSE bundles), track ADRs in `docs/modules/scanner/architecture.md` §7 and follow upcoming Attestor API releases.
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
# AOC Normalization Removal Notes
|
||||
|
||||
_Last updated: 2025-10-29_
|
||||
|
||||
## Goal
|
||||
|
||||
Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final pieces of normalization from the ingestion/runtime path.
|
||||
|
||||
## Current Findings
|
||||
|
||||
- `AdvisoryRawService` and `MongoAdvisoryRawRepository` already preserve upstream ordering and duplicate aliases (trim-only). No additional code changes required there.
|
||||
- Observation layers (`AdvisoryObservationFactory`, `AdvisoryObservationQueryService`) still canonicalise aliases, PURLs, CPEs, and references. These need to be relaxed so Policy/overlays receive raw linksets and can own dedupe logic.
|
||||
- Linkset mapper continues to emit deterministic hints. We will keep the mapper but ensure observation output can surface both raw and canonical views for downstream services.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume.
|
||||
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection.
|
||||
# AOC Normalization Removal Notes
|
||||
|
||||
_Last updated: 2025-10-29_
|
||||
|
||||
## Goal
|
||||
|
||||
Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final pieces of normalization from the ingestion/runtime path.
|
||||
|
||||
## Current Findings
|
||||
|
||||
- `AdvisoryRawService` and `MongoAdvisoryRawRepository` already preserve upstream ordering and duplicate aliases (trim-only). No additional code changes required there.
|
||||
- Observation layers (`AdvisoryObservationFactory`, `AdvisoryObservationQueryService`) still canonicalise aliases, PURLs, CPEs, and references. These need to be relaxed so Policy/overlays receive raw linksets and can own dedupe logic.
|
||||
- Linkset mapper continues to emit deterministic hints. We will keep the mapper but ensure observation output can surface both raw and canonical views for downstream services.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. ✅ 2025-10-31: `AdvisoryObservation` now surfaces `RawLinkset`; Mongo documents store both canonical & raw shapes; tests/goldens updated.
|
||||
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. ✅ 2025-10-31.
|
||||
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split.
|
||||
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults.
|
||||
|
||||
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. ↺ Ongoing — see action items in `docs/dev/raw-linkset-backfill-plan.md` (2025-10-31 handshake with POLICY-ENGINE-20-003 owners).
|
||||
|
||||
|
||||
@@ -1,146 +1,146 @@
|
||||
# Authority DPoP & mTLS Implementation Plan (2025-10-19)
|
||||
|
||||
## Purpose
|
||||
- Provide the implementation blueprint for AUTH-DPOP-11-001 and AUTH-MTLS-11-002.
|
||||
- Unify sender-constraint validation across Authority, downstream services, and clients.
|
||||
- Capture deterministic, testable steps that unblock UI/Signer guilds depending on DPoP/mTLS hardening.
|
||||
|
||||
## Scope
|
||||
- Token endpoint validation, issuance, and storage changes inside `StellaOps.Authority`.
|
||||
- Shared security primitives consumed by Authority, Scanner, Signer, CLI, and UI.
|
||||
- Operator-facing configuration, auditing, and observability.
|
||||
- Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities.
|
||||
|
||||
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/Authority/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
|
||||
>
|
||||
> **Status update (2025-10-20):** Redis-backed nonce configuration is exposed through `security.senderConstraints.dpop.nonce` with sample YAML (`etc/authority.yaml.sample`) and architecture docs refreshed. Operator guide now includes concrete Redis/required audiences snippet; nonce challenge regression remains covered by `ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing`.
|
||||
>
|
||||
> **Status update (2025-10-23):** mTLS enforcement now honours `security.senderConstraints.mtls.enforceForAudiences`, automatically rejecting non-mTLS clients targeting audiences such as `signer`. Certificate bindings validate thumbprint, issuer, subject, serial number, and SAN values, producing deterministic error codes for operators. Introspection responses include `cnf.x5t#S256`, and new unit tests cover audience enforcement, binding mismatches, and bootstrap storage. Docs/sample config updated accordingly.
|
||||
|
||||
## Design Summary
|
||||
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
|
||||
- Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc).
|
||||
- Require DPoP proofs on `/token` when the registered client policy is `senderConstraint=dpop`; bind issued access tokens via `cnf.jkt`.
|
||||
- Introduce Authority-managed nonce issuance for “high value” audiences (default: `signer`, `attestor`) with Redis-backed persistence and deterministic auditing.
|
||||
- Enable OAuth 2.0 mTLS (RFC 8705) by storing certificate bindings per client, requesting client certificates at TLS termination, and stamping `cnf.x5t#S256` into issued tokens plus introspection output.
|
||||
- Surface structured logs and counters for both DPoP and mTLS flows; provide integration tests that cover success, replay, invalid proof, and certificate mismatch cases.
|
||||
|
||||
## AUTH-DPOP-11-001 — Proof Validation & Nonce Handling
|
||||
|
||||
**Shared validator**
|
||||
- Move `DpopProofValidator`, option types, and replay cache interfaces from `StellaOps.Scanner.Core` into a new assembly `StellaOps.Auth.Security`.
|
||||
- Provide pluggable caches: `InMemoryDpopReplayCache` (existing) and new `RedisDpopReplayCache` (leveraging the Authority Redis connection).
|
||||
- Ensure the validator exposes the validated `SecurityKey`, `jti`, and `iat` so Authority can construct the `cnf` claim and compute nonce expiry.
|
||||
|
||||
**Configuration model**
|
||||
- Extend `StellaOpsAuthorityOptions.Security` with a `SenderConstraints` property containing:
|
||||
- `Dpop` (`enabled`, `allowedAlgorithms`, `maxAgeSeconds`, `clockSkewSeconds`, `replayWindowSeconds`, `nonce` settings with `enabled`, `ttlSeconds`, `requiredAudiences`, `maxIssuancePerMinute`).
|
||||
- `Mtls` (`enabled`, `requireChainValidation`, `clientCaBundle`, `allowedSubjectPatterns`, `allowedSanTypes`).
|
||||
- Bind from YAML (`authority.security.senderConstraints.*`) while preserving backwards compatibility (defaults keep both disabled).
|
||||
|
||||
**Token endpoint pipeline**
|
||||
- Introduce a scoped OpenIddict handler `ValidateDpopProofHandler` inserted before `ValidateClientCredentialsHandler`.
|
||||
- Determine the required sender constraint from client metadata:
|
||||
- Add `AuthorityClientMetadataKeys.SenderConstraint` storing `dpop` or `mtls`.
|
||||
- Optionally allow per-client overrides for nonce requirement.
|
||||
- When `dpop` is required:
|
||||
- Read the `DPoP` header from the ASP.NET request, reject with `invalid_token` + `WWW-Authenticate: DPoP error="invalid_dpop_proof"` if absent.
|
||||
- Call the shared validator with method/URI. Enforce algorithm allowlist and `iat` window from options.
|
||||
- Persist the `jkt` thumbprint plus replay cache state in the OpenIddict transaction (`AuthorityOpenIddictConstants.DpopKeyThumbprintProperty`, `DpopIssuedAtProperty`).
|
||||
- When the requested audience intersects `SenderConstraints.Dpop.Nonce.RequiredAudiences`, require `nonce` in the proof; on first failure respond with HTTP 401, `error="use_dpop_nonce"`, and include `DPoP-Nonce` header (see nonce note below). Cache the rejection reason for audit logging.
|
||||
|
||||
**Nonce service**
|
||||
- Add `IDpopNonceStore` with methods `IssueAsync(audience, clientId, jkt)` and `TryConsumeAsync(nonce, audience, clientId, jkt)`.
|
||||
- Default implementation `RedisDpopNonceStore` storing SHA-256 hashes of nonces keyed by `audience:clientId:jkt`. TTL comes from `SenderConstraints.Dpop.Nonce.Ttl`.
|
||||
- Create helper `DpopNonceIssuer` used by `ValidateDpopProofHandler` to issue nonces when missing/expired, enforcing issuance rate limits (per options) and tagging audit/log records.
|
||||
- On successful validation (nonce supplied and consumed) stamp metadata into the transaction for auditing.
|
||||
- Update `ClientCredentialsHandlers` to observe nonce enforcement: when a nonce challenge was sent, emit structured audit with `nonce_issued`, `audiences`, and `retry`.
|
||||
|
||||
**Token issuance**
|
||||
- In `HandleClientCredentialsHandler`, if the transaction contains a validated DPoP key:
|
||||
- Build `cnf.jkt` using thumbprint from validator.
|
||||
- Include `auth_time`/`dpop_jti` as needed for diagnostics.
|
||||
- Persist the thumbprint alongside token metadata in Mongo (extend `AuthorityTokenDocument` with `SenderConstraint`, `KeyThumbprint`, `Nonce` fields).
|
||||
|
||||
**Auditing & observability**
|
||||
- Emit new audit events:
|
||||
- `authority.dpop.proof.validated` (success/failure, clientId, audience, thumbprint, nonce status, jti).
|
||||
- `authority.dpop.nonce.issued` and `authority.dpop.nonce.consumed`.
|
||||
- Metrics (Prometheus style):
|
||||
- `authority_dpop_validations_total{result,reason}`.
|
||||
- `authority_dpop_nonce_issued_total{audience}` and `authority_dpop_nonce_fails_total{reason}`.
|
||||
- Structured logs include `authority.sender_constraint=dpop`, `authority.dpop_thumbprint`, `authority.dpop_nonce`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for the handler pipeline using fake OpenIddict transactions.
|
||||
- Replay/nonce tests with in-memory and Redis stores.
|
||||
- Integration tests in `StellaOps.Authority.Tests` covering:
|
||||
- Valid DPoP proof issuing `cnf.jkt`.
|
||||
- Missing header → challenge with nonce.
|
||||
- Replayed `jti` rejected.
|
||||
- Invalid nonce rejected even after issuance.
|
||||
- Contract tests to ensure `/.well-known/openid-configuration` advertises `dpop_signing_alg_values_supported` and `dpop_nonce_supported` when enabled.
|
||||
|
||||
## AUTH-MTLS-11-002 — Certificate-Bound Tokens
|
||||
|
||||
**Configuration model**
|
||||
- Reuse `SenderConstraints.Mtls` described above; include:
|
||||
- `enforceForAudiences` list (defaults `signer`, `attestor`, `scheduler`).
|
||||
- `certificateRotationGraceSeconds` for overlap.
|
||||
- `allowedClientCertificateAuthorities` absolute paths.
|
||||
|
||||
**Kestrel/TLS pipeline**
|
||||
- Configure Kestrel with `ClientCertificateMode.AllowCertificate` globally and implement middleware that enforces certificate presence only when the resolved client requires mTLS.
|
||||
- Add `IAuthorityClientCertificateValidator` that validates presented certificate chain, SANs (`dns`, `uri`, optional SPIFFE), and thumbprint matches one of the stored bindings.
|
||||
- Cache validation results per connection id to avoid rehashing on every request.
|
||||
|
||||
**Client registration & storage**
|
||||
- Extend `AuthorityClientDocument` with `List<AuthorityClientCertificateBinding>` containing:
|
||||
- `Thumbprint`, `SerialNumber`, `Subject`, `NotBefore`, `NotAfter`, `Sans`, `CreatedAt`, `UpdatedAt`, `Label`.
|
||||
- Provide admin API mutations (`/admin/clients/{id}/certificates`) for ops tooling (deferred implementation but schema ready).
|
||||
- Update plugin provisioning store (`StandardClientProvisioningStore`) to map descriptors with certificate bindings and `senderConstraint`.
|
||||
- Persist binding state in Mongo migrations (index on `{clientId, thumbprint}`).
|
||||
|
||||
**Token issuance & introspection**
|
||||
- Add a transaction property capturing the validated client certificate thumbprint.
|
||||
- `HandleClientCredentialsHandler`:
|
||||
- When mTLS required, ensure certificate info present; reject otherwise.
|
||||
- Stamp `cnf` claim: `principal.SetClaim("cnf", JsonSerializer.Serialize(new { x5t#S256 = thumbprint }))`.
|
||||
- Store binding metadata in issued token document for audit.
|
||||
- Update `ValidateAccessTokenHandler` and introspection responses to surface `cnf.x5t#S256`.
|
||||
- Ensure refresh tokens (if ever enabled) copy the binding data.
|
||||
|
||||
**Auditing & observability**
|
||||
- Audit events:
|
||||
- `authority.mtls.handshake` (success/failure, clientId, thumbprint, issuer, subject).
|
||||
- `authority.mtls.binding.missing` when a required client posts without a cert.
|
||||
- Metrics:
|
||||
- `authority_mtls_handshakes_total{result}`.
|
||||
- `authority_mtls_certificate_rotations_total`.
|
||||
- Logs include `authority.sender_constraint=mtls`, `authority.mtls_thumbprint`, `authority.mtls_subject`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for certificate validation rules (SAN mismatches, expiry, CA trust).
|
||||
- Integration tests running Kestrel with test certificates:
|
||||
- Successful token issuance with bound certificate.
|
||||
- Request without certificate → `invalid_client`.
|
||||
- Token introspection reveals `cnf.x5t#S256`.
|
||||
- Rotation scenario (old + new cert allowed during grace window).
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
**DPoP work-stream**
|
||||
1. Extract shared validator into `StellaOps.Auth.Security`; update Scanner references.
|
||||
2. Introduce configuration classes and bind from YAML/environment.
|
||||
3. Implement nonce store (Redis + in-memory), handler integration, and OpenIddict transaction plumbing.
|
||||
4. Stamp `cnf.jkt`, audit events, and metrics; update Mongo documents and migrations.
|
||||
5. Extend docs: `docs/modules/authority/architecture.md`, `docs/security/audit-events.md`, `docs/security/rate-limits.md`, CLI/UI references.
|
||||
|
||||
**mTLS work-stream**
|
||||
1. Extend client document/schema and provisioning stores with certificate bindings + sender constraint flag.
|
||||
2. Configure Kestrel/middleware for optional client certificates and validation service.
|
||||
3. Update token issuance/introspection to honour certificate bindings and emit `cnf.x5t#S256`.
|
||||
4. Add auditing/metrics and integration tests (happy path + failure).
|
||||
5. Refresh operator documentation (`docs/modules/authority/operations/backup-restore.md`, `docs/modules/authority/operations/monitoring.md`, sample `authority.yaml`) to cover certificate lifecycle.
|
||||
|
||||
Both streams should conclude with `dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln` and documentation cross-links so dependent guilds can unblock UI/Signer work.
|
||||
# Authority DPoP & mTLS Implementation Plan (2025-10-19)
|
||||
|
||||
## Purpose
|
||||
- Provide the implementation blueprint for AUTH-DPOP-11-001 and AUTH-MTLS-11-002.
|
||||
- Unify sender-constraint validation across Authority, downstream services, and clients.
|
||||
- Capture deterministic, testable steps that unblock UI/Signer guilds depending on DPoP/mTLS hardening.
|
||||
|
||||
## Scope
|
||||
- Token endpoint validation, issuance, and storage changes inside `StellaOps.Authority`.
|
||||
- Shared security primitives consumed by Authority, Scanner, Signer, CLI, and UI.
|
||||
- Operator-facing configuration, auditing, and observability.
|
||||
- Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities.
|
||||
|
||||
> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/Authority/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors.
|
||||
>
|
||||
> **Status update (2025-10-20):** Redis-backed nonce configuration is exposed through `security.senderConstraints.dpop.nonce` with sample YAML (`etc/authority.yaml.sample`) and architecture docs refreshed. Operator guide now includes concrete Redis/required audiences snippet; nonce challenge regression remains covered by `ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing`.
|
||||
>
|
||||
> **Status update (2025-10-23):** mTLS enforcement now honours `security.senderConstraints.mtls.enforceForAudiences`, automatically rejecting non-mTLS clients targeting audiences such as `signer`. Certificate bindings validate thumbprint, issuer, subject, serial number, and SAN values, producing deterministic error codes for operators. Introspection responses include `cnf.x5t#S256`, and new unit tests cover audience enforcement, binding mismatches, and bootstrap storage. Docs/sample config updated accordingly.
|
||||
|
||||
## Design Summary
|
||||
- Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers.
|
||||
- Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc).
|
||||
- Require DPoP proofs on `/token` when the registered client policy is `senderConstraint=dpop`; bind issued access tokens via `cnf.jkt`.
|
||||
- Introduce Authority-managed nonce issuance for “high value” audiences (default: `signer`, `attestor`) with Redis-backed persistence and deterministic auditing.
|
||||
- Enable OAuth 2.0 mTLS (RFC 8705) by storing certificate bindings per client, requesting client certificates at TLS termination, and stamping `cnf.x5t#S256` into issued tokens plus introspection output.
|
||||
- Surface structured logs and counters for both DPoP and mTLS flows; provide integration tests that cover success, replay, invalid proof, and certificate mismatch cases.
|
||||
|
||||
## AUTH-DPOP-11-001 — Proof Validation & Nonce Handling
|
||||
|
||||
**Shared validator**
|
||||
- Move `DpopProofValidator`, option types, and replay cache interfaces from `StellaOps.Scanner.Core` into a new assembly `StellaOps.Auth.Security`.
|
||||
- Provide pluggable caches: `InMemoryDpopReplayCache` (existing) and new `RedisDpopReplayCache` (leveraging the Authority Redis connection).
|
||||
- Ensure the validator exposes the validated `SecurityKey`, `jti`, and `iat` so Authority can construct the `cnf` claim and compute nonce expiry.
|
||||
|
||||
**Configuration model**
|
||||
- Extend `StellaOpsAuthorityOptions.Security` with a `SenderConstraints` property containing:
|
||||
- `Dpop` (`enabled`, `allowedAlgorithms`, `maxAgeSeconds`, `clockSkewSeconds`, `replayWindowSeconds`, `nonce` settings with `enabled`, `ttlSeconds`, `requiredAudiences`, `maxIssuancePerMinute`).
|
||||
- `Mtls` (`enabled`, `requireChainValidation`, `clientCaBundle`, `allowedSubjectPatterns`, `allowedSanTypes`).
|
||||
- Bind from YAML (`authority.security.senderConstraints.*`) while preserving backwards compatibility (defaults keep both disabled).
|
||||
|
||||
**Token endpoint pipeline**
|
||||
- Introduce a scoped OpenIddict handler `ValidateDpopProofHandler` inserted before `ValidateClientCredentialsHandler`.
|
||||
- Determine the required sender constraint from client metadata:
|
||||
- Add `AuthorityClientMetadataKeys.SenderConstraint` storing `dpop` or `mtls`.
|
||||
- Optionally allow per-client overrides for nonce requirement.
|
||||
- When `dpop` is required:
|
||||
- Read the `DPoP` header from the ASP.NET request, reject with `invalid_token` + `WWW-Authenticate: DPoP error="invalid_dpop_proof"` if absent.
|
||||
- Call the shared validator with method/URI. Enforce algorithm allowlist and `iat` window from options.
|
||||
- Persist the `jkt` thumbprint plus replay cache state in the OpenIddict transaction (`AuthorityOpenIddictConstants.DpopKeyThumbprintProperty`, `DpopIssuedAtProperty`).
|
||||
- When the requested audience intersects `SenderConstraints.Dpop.Nonce.RequiredAudiences`, require `nonce` in the proof; on first failure respond with HTTP 401, `error="use_dpop_nonce"`, and include `DPoP-Nonce` header (see nonce note below). Cache the rejection reason for audit logging.
|
||||
|
||||
**Nonce service**
|
||||
- Add `IDpopNonceStore` with methods `IssueAsync(audience, clientId, jkt)` and `TryConsumeAsync(nonce, audience, clientId, jkt)`.
|
||||
- Default implementation `RedisDpopNonceStore` storing SHA-256 hashes of nonces keyed by `audience:clientId:jkt`. TTL comes from `SenderConstraints.Dpop.Nonce.Ttl`.
|
||||
- Create helper `DpopNonceIssuer` used by `ValidateDpopProofHandler` to issue nonces when missing/expired, enforcing issuance rate limits (per options) and tagging audit/log records.
|
||||
- On successful validation (nonce supplied and consumed) stamp metadata into the transaction for auditing.
|
||||
- Update `ClientCredentialsHandlers` to observe nonce enforcement: when a nonce challenge was sent, emit structured audit with `nonce_issued`, `audiences`, and `retry`.
|
||||
|
||||
**Token issuance**
|
||||
- In `HandleClientCredentialsHandler`, if the transaction contains a validated DPoP key:
|
||||
- Build `cnf.jkt` using thumbprint from validator.
|
||||
- Include `auth_time`/`dpop_jti` as needed for diagnostics.
|
||||
- Persist the thumbprint alongside token metadata in Mongo (extend `AuthorityTokenDocument` with `SenderConstraint`, `KeyThumbprint`, `Nonce` fields).
|
||||
|
||||
**Auditing & observability**
|
||||
- Emit new audit events:
|
||||
- `authority.dpop.proof.validated` (success/failure, clientId, audience, thumbprint, nonce status, jti).
|
||||
- `authority.dpop.nonce.issued` and `authority.dpop.nonce.consumed`.
|
||||
- Metrics (Prometheus style):
|
||||
- `authority_dpop_validations_total{result,reason}`.
|
||||
- `authority_dpop_nonce_issued_total{audience}` and `authority_dpop_nonce_fails_total{reason}`.
|
||||
- Structured logs include `authority.sender_constraint=dpop`, `authority.dpop_thumbprint`, `authority.dpop_nonce`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for the handler pipeline using fake OpenIddict transactions.
|
||||
- Replay/nonce tests with in-memory and Redis stores.
|
||||
- Integration tests in `StellaOps.Authority.Tests` covering:
|
||||
- Valid DPoP proof issuing `cnf.jkt`.
|
||||
- Missing header → challenge with nonce.
|
||||
- Replayed `jti` rejected.
|
||||
- Invalid nonce rejected even after issuance.
|
||||
- Contract tests to ensure `/.well-known/openid-configuration` advertises `dpop_signing_alg_values_supported` and `dpop_nonce_supported` when enabled.
|
||||
|
||||
## AUTH-MTLS-11-002 — Certificate-Bound Tokens
|
||||
|
||||
**Configuration model**
|
||||
- Reuse `SenderConstraints.Mtls` described above; include:
|
||||
- `enforceForAudiences` list (defaults `signer`, `attestor`, `scheduler`).
|
||||
- `certificateRotationGraceSeconds` for overlap.
|
||||
- `allowedClientCertificateAuthorities` absolute paths.
|
||||
|
||||
**Kestrel/TLS pipeline**
|
||||
- Configure Kestrel with `ClientCertificateMode.AllowCertificate` globally and implement middleware that enforces certificate presence only when the resolved client requires mTLS.
|
||||
- Add `IAuthorityClientCertificateValidator` that validates presented certificate chain, SANs (`dns`, `uri`, optional SPIFFE), and thumbprint matches one of the stored bindings.
|
||||
- Cache validation results per connection id to avoid rehashing on every request.
|
||||
|
||||
**Client registration & storage**
|
||||
- Extend `AuthorityClientDocument` with `List<AuthorityClientCertificateBinding>` containing:
|
||||
- `Thumbprint`, `SerialNumber`, `Subject`, `NotBefore`, `NotAfter`, `Sans`, `CreatedAt`, `UpdatedAt`, `Label`.
|
||||
- Provide admin API mutations (`/admin/clients/{id}/certificates`) for ops tooling (deferred implementation but schema ready).
|
||||
- Update plugin provisioning store (`StandardClientProvisioningStore`) to map descriptors with certificate bindings and `senderConstraint`.
|
||||
- Persist binding state in Mongo migrations (index on `{clientId, thumbprint}`).
|
||||
|
||||
**Token issuance & introspection**
|
||||
- Add a transaction property capturing the validated client certificate thumbprint.
|
||||
- `HandleClientCredentialsHandler`:
|
||||
- When mTLS required, ensure certificate info present; reject otherwise.
|
||||
- Stamp `cnf` claim: `principal.SetClaim("cnf", JsonSerializer.Serialize(new { x5t#S256 = thumbprint }))`.
|
||||
- Store binding metadata in issued token document for audit.
|
||||
- Update `ValidateAccessTokenHandler` and introspection responses to surface `cnf.x5t#S256`.
|
||||
- Ensure refresh tokens (if ever enabled) copy the binding data.
|
||||
|
||||
**Auditing & observability**
|
||||
- Audit events:
|
||||
- `authority.mtls.handshake` (success/failure, clientId, thumbprint, issuer, subject).
|
||||
- `authority.mtls.binding.missing` when a required client posts without a cert.
|
||||
- Metrics:
|
||||
- `authority_mtls_handshakes_total{result}`.
|
||||
- `authority_mtls_certificate_rotations_total`.
|
||||
- Logs include `authority.sender_constraint=mtls`, `authority.mtls_thumbprint`, `authority.mtls_subject`.
|
||||
|
||||
**Testing**
|
||||
- Unit tests for certificate validation rules (SAN mismatches, expiry, CA trust).
|
||||
- Integration tests running Kestrel with test certificates:
|
||||
- Successful token issuance with bound certificate.
|
||||
- Request without certificate → `invalid_client`.
|
||||
- Token introspection reveals `cnf.x5t#S256`.
|
||||
- Rotation scenario (old + new cert allowed during grace window).
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
**DPoP work-stream**
|
||||
1. Extract shared validator into `StellaOps.Auth.Security`; update Scanner references.
|
||||
2. Introduce configuration classes and bind from YAML/environment.
|
||||
3. Implement nonce store (Redis + in-memory), handler integration, and OpenIddict transaction plumbing.
|
||||
4. Stamp `cnf.jkt`, audit events, and metrics; update Mongo documents and migrations.
|
||||
5. Extend docs: `docs/modules/authority/architecture.md`, `docs/security/audit-events.md`, `docs/security/rate-limits.md`, CLI/UI references.
|
||||
|
||||
**mTLS work-stream**
|
||||
1. Extend client document/schema and provisioning stores with certificate bindings + sender constraint flag.
|
||||
2. Configure Kestrel/middleware for optional client certificates and validation service.
|
||||
3. Update token issuance/introspection to honour certificate bindings and emit `cnf.x5t#S256`.
|
||||
4. Add auditing/metrics and integration tests (happy path + failure).
|
||||
5. Refresh operator documentation (`docs/modules/authority/operations/backup-restore.md`, `docs/modules/authority/operations/monitoring.md`, sample `authority.yaml`) to cover certificate lifecycle.
|
||||
|
||||
Both streams should conclude with `dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln` and documentation cross-links so dependent guilds can unblock UI/Signer work.
|
||||
|
||||
@@ -1,45 +1,45 @@
|
||||
# Concelier Fixture Maintenance
|
||||
|
||||
Concelier uses a handful of deterministic fixtures to keep connector regressions in check. This guide lists the
|
||||
fixture sets, where they live, and how to regenerate them safely.
|
||||
|
||||
---
|
||||
|
||||
## GHSA ↔ OSV parity fixtures
|
||||
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/osv-ghsa.*.json`
|
||||
- **Purpose:** Exercised by `OsvGhsaParityRegressionTests` to ensure OSV + GHSA outputs stay aligned on aliases,
|
||||
ranges, references, and credits.
|
||||
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`)
|
||||
or execute the fixture updater (`dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
|
||||
normalise timestamps and canonical ordering.
|
||||
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
|
||||
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
|
||||
constraints and notes line up with GHSA/NVD composites before committing.
|
||||
- **Verification:** Inspect the diff, then re-run `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj` to confirm parity.
|
||||
|
||||
## GHSA credit parity fixtures
|
||||
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
|
||||
- **Purpose:** Exercised by `GhsaCreditParityRegressionTests` to guarantee GHSA/NVD/OSV acknowledgements remain in lockstep.
|
||||
- **Regeneration:** `dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj` rewrites all three canonical snapshots.
|
||||
- **Verification:** `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj`.
|
||||
|
||||
> Always commit fixture changes together with the code that motivated them and reference the regression test that guards the behaviour.
|
||||
|
||||
## Apple security update fixtures
|
||||
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
|
||||
- **Purpose:** Exercised by `AppleLiveRegressionTests` to guarantee the Apple HTML parser and mapper stay deterministic while covering Rapid Security Responses and multi-device advisories.
|
||||
- **Regeneration:** Use the helper scripts (`scripts/update-apple-fixtures.sh` or `scripts/update-apple-fixtures.ps1`). They export `UPDATE_APPLE_FIXTURES=1`, propagate the flag through `WSLENV`, touch `.update-apple-fixtures`, and then run the Apple test project. This keeps WSL/VSCode test invocations in sync while the refresh workflow fetches live Apple support pages, sanitises them, and rewrites both the HTML and expected DTO snapshots with normalised ordering.
|
||||
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
|
||||
|
||||
> **Tip for other connector owners:** mirror the sentinel + `WSLENV` pattern (`touch .update-<connector>-fixtures`, append the env var via `WSLENV`) when you add fixture refresh scripts so contributors running under WSL inherit the regeneration flag automatically.
|
||||
|
||||
## KISA advisory fixtures
|
||||
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-{feed,detail}.(xml|json)`
|
||||
- **Purpose:** Used by `KisaConnectorTests` to verify Hangul-aware fetch → parse → map flows and to assert telemetry counters stay wired.
|
||||
- **Regeneration:** `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`
|
||||
- **Verification:** Re-run the same test suite without the env var; confirm advisory content remains NFC-normalised and HTML is sanitised. Metrics assertions will fail if counters drift.
|
||||
- **Localisation note:** RSS `category` values (e.g. `취약점정보`) remain in Hangul—do not translate them in fixtures; they feed directly into metrics/log tags.
|
||||
# Concelier Fixture Maintenance
|
||||
|
||||
Concelier uses a handful of deterministic fixtures to keep connector regressions in check. This guide lists the
|
||||
fixture sets, where they live, and how to regenerate them safely.
|
||||
|
||||
---
|
||||
|
||||
## GHSA ↔ OSV parity fixtures
|
||||
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/osv-ghsa.*.json`
|
||||
- **Purpose:** Exercised by `OsvGhsaParityRegressionTests` to ensure OSV + GHSA outputs stay aligned on aliases,
|
||||
ranges, references, and credits.
|
||||
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`)
|
||||
or execute the fixture updater (`dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
|
||||
normalise timestamps and canonical ordering.
|
||||
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
|
||||
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
|
||||
constraints and notes line up with GHSA/NVD composites before committing.
|
||||
- **Verification:** Inspect the diff, then re-run `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj` to confirm parity.
|
||||
|
||||
## GHSA credit parity fixtures
|
||||
|
||||
- **Location:** `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/credit-parity.{ghsa,osv,nvd}.json`
|
||||
- **Purpose:** Exercised by `GhsaCreditParityRegressionTests` to guarantee GHSA/NVD/OSV acknowledgements remain in lockstep.
|
||||
- **Regeneration:** `dotnet run --project src/Tools/FixtureUpdater/FixtureUpdater.csproj` rewrites all three canonical snapshots.
|
||||
- **Verification:** `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj`.
|
||||
|
||||
> Always commit fixture changes together with the code that motivated them and reference the regression test that guards the behaviour.
|
||||
|
||||
## Apple security update fixtures
|
||||
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures/*.html` and `.expected.json`.
|
||||
- **Purpose:** Exercised by `AppleLiveRegressionTests` to guarantee the Apple HTML parser and mapper stay deterministic while covering Rapid Security Responses and multi-device advisories.
|
||||
- **Regeneration:** Use the helper scripts (`scripts/update-apple-fixtures.sh` or `scripts/update-apple-fixtures.ps1`). They export `UPDATE_APPLE_FIXTURES=1`, propagate the flag through `WSLENV`, touch `.update-apple-fixtures`, and then run the Apple test project. This keeps WSL/VSCode test invocations in sync while the refresh workflow fetches live Apple support pages, sanitises them, and rewrites both the HTML and expected DTO snapshots with normalised ordering.
|
||||
- **Verification:** Inspect the generated diffs and re-run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj` without the env var to confirm determinism.
|
||||
|
||||
> **Tip for other connector owners:** mirror the sentinel + `WSLENV` pattern (`touch .update-<connector>-fixtures`, append the env var via `WSLENV`) when you add fixture refresh scripts so contributors running under WSL inherit the regeneration flag automatically.
|
||||
|
||||
## KISA advisory fixtures
|
||||
|
||||
- **Location:** `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-{feed,detail}.(xml|json)`
|
||||
- **Purpose:** Used by `KisaConnectorTests` to verify Hangul-aware fetch → parse → map flows and to assert telemetry counters stay wired.
|
||||
- **Regeneration:** `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`
|
||||
- **Verification:** Re-run the same test suite without the env var; confirm advisory content remains NFC-normalised and HTML is sanitised. Metrics assertions will fail if counters drift.
|
||||
- **Localisation note:** RSS `category` values (e.g. `취약점정보`) remain in Hangul—do not translate them in fixtures; they feed directly into metrics/log tags.
|
||||
|
||||
@@ -1,154 +1,154 @@
|
||||
# Concelier SemVer Merge Playbook (Sprint 1–2)
|
||||
|
||||
This playbook describes how the merge layer and connector teams should emit the new SemVer primitives introduced in Sprint 1–2, how those primitives become normalized version rules, and how downstream jobs query them deterministically.
|
||||
|
||||
## 1. What landed in Sprint 1–2
|
||||
|
||||
- `RangePrimitives.SemVer` now infers a canonical `style` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`) and captures `exactValue` when the constraint is a single version.
|
||||
- `NormalizedVersionRule` documents the analytics-friendly projection of each `AffectedPackage` coverage entry and is persisted alongside legacy `versionRanges`.
|
||||
- `AdvisoryProvenance.decisionReason` records whether merge resolution favored precedence, freshness, or a tie-breaker comparison.
|
||||
|
||||
See `src/Concelier/__Libraries/StellaOps.Concelier.Models/CANONICAL_RECORDS.md` for the full schema and field descriptions.
|
||||
|
||||
## 2. Mapper pattern
|
||||
|
||||
Connectors should emit SemVer primitives as soon as they can normalize a vendor constraint. The helper `SemVerPrimitiveExtensions.ToNormalizedVersionRule` turns those primitives into the persisted rules:
|
||||
|
||||
```csharp
|
||||
var primitive = new SemVerPrimitive(
|
||||
introduced: "1.2.3",
|
||||
introducedInclusive: true,
|
||||
fixed: "2.0.0",
|
||||
fixedInclusive: false,
|
||||
lastAffected: null,
|
||||
lastAffectedInclusive: false,
|
||||
constraintExpression: ">=1.2.3 <2.0.0",
|
||||
exactValue: null);
|
||||
|
||||
var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
|
||||
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
|
||||
```
|
||||
|
||||
If you omit the optional `notes` argument, `ToNormalizedVersionRule` now falls back to the primitive’s `ConstraintExpression`, ensuring the original comparator expression is preserved for provenance/audit queries.
|
||||
|
||||
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
|
||||
|
||||
## 3. Merge dedupe flow
|
||||
|
||||
During merge, feed all package candidates through `NormalizedVersionRuleComparer.Instance` prior to persistence. The comparer orders by scheme → type → min → minInclusive → max → maxInclusive → value → notes, guaranteeing consistent document layout and making `$unwind` pipelines deterministic.
|
||||
|
||||
If multiple connectors emit identical constraints, the merge layer should:
|
||||
|
||||
1. Combine provenance entries (preserving one per source).
|
||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||
|
||||
## 4. Example Mongo pipeline
|
||||
|
||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { "affectedPackages.type": "semver", "affectedPackages.identifier": "pkg:npm/lodash" } },
|
||||
{ $unwind: "$affectedPackages" },
|
||||
{ $unwind: "$affectedPackages.normalizedVersions" },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "affectedPackages.normalizedVersions.type": "exact",
|
||||
"affectedPackages.normalizedVersions.value": "4.17.21" },
|
||||
{ "affectedPackages.normalizedVersions.type": "range",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" },
|
||||
"affectedPackages.normalizedVersions.max": { $gt: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "gte",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "lte",
|
||||
"affectedPackages.normalizedVersions.max": { $gte: "4.17.21" } }
|
||||
]
|
||||
}},
|
||||
{ $project: { advisoryKey: 1, title: 1, "affectedPackages.identifier": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 5. Recommended indexes
|
||||
|
||||
| Collection | Index | Purpose |
|
||||
|------------|-------|---------|
|
||||
| `advisory` | `{ "affectedPackages.identifier": 1, "affectedPackages.normalizedVersions.scheme": 1, "affectedPackages.normalizedVersions.type": 1 }` (compound, multikey) | Speeds up `$match` on identifier + rule style. |
|
||||
| `advisory` | `{ "affectedPackages.normalizedVersions.value": 1 }` (sparse) | Optimizes lookups for exact version hits. |
|
||||
|
||||
Coordinate with the Storage team when enabling these indexes so deployment windows account for collection size.
|
||||
|
||||
## 6. Dual-write rollout
|
||||
|
||||
Follow the operational checklist in `docs/modules/devops/migrations/semver-style.md`. The summary:
|
||||
|
||||
1. **Dual write (now)** – emit both legacy `versionRanges` and the new `normalizedVersions`.
|
||||
2. **Backfill** – follow the storage migration in `docs/modules/devops/migrations/semver-style.md` to rewrite historical advisories before switching consumers.
|
||||
3. **Verify** – run the aggregation above (with `explain("executionStats")`) to ensure the new indexes are used.
|
||||
4. **Cutover** – after consumers switch to normalized rules, mark the old `rangeExpression` as deprecated.
|
||||
|
||||
## 7. Checklist for connectors & merge
|
||||
|
||||
- [ ] Populate `SemVerPrimitive` for every SemVer-friendly constraint.
|
||||
- [ ] Call `ToNormalizedVersionRule` and store the result.
|
||||
- [ ] Emit provenance masks covering both `versionRanges[].primitives.semver` and `normalizedVersions[]`.
|
||||
- [ ] Ensure merge deduping relies on the canonical comparer.
|
||||
- [ ] Capture merge decisions via `decisionReason`.
|
||||
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
|
||||
|
||||
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 8. Storage projection reference
|
||||
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "pkg:npm/example",
|
||||
"packageType": "npm",
|
||||
"scheme": "semver",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "1.2.3",
|
||||
"minInclusive": true,
|
||||
"max": "2.0.0",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "ghsa:GHSA-xxxx-yyyy",
|
||||
"decisionReason": "ghsa-precedence-over-nvd",
|
||||
"constraint": ">= 1.2.3 < 2.0.0",
|
||||
"source": "ghsa",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
For distro-specific ranges (`nevra`, `evr`) the same envelope applies with `scheme` switched accordingly. Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "bash",
|
||||
"packageType": "rpm",
|
||||
"scheme": "nevra",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "0:4.4.18-2.el7",
|
||||
"minInclusive": true,
|
||||
"max": "0:4.4.20-1.el7",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "redhat:RHSA-2025:1234",
|
||||
"decisionReason": "rhel-priority-over-nvd",
|
||||
"constraint": "<= 0:4.4.20-1.el7",
|
||||
"source": "redhat",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
If a new scheme is required (for example, `apple.build` or `ios.semver`), raise it with the Models team before emitting documents so merge comparers and hashing logic can incorporate the change deterministically.
|
||||
|
||||
## 9. Observability signals
|
||||
|
||||
- `concelier.merge.normalized_rules` (counter, tags: `package_type`, `scheme`) – increments once per normalized rule retained after precedence merge.
|
||||
- `concelier.merge.normalized_rules_missing` (counter, tags: `package_type`) – increments when a merged package still carries version ranges but no normalized rules; watch for spikes to catch connectors that have not emitted normalized arrays yet.
|
||||
# Concelier SemVer Merge Playbook (Sprint 1–2)
|
||||
|
||||
This playbook describes how the merge layer and connector teams should emit the new SemVer primitives introduced in Sprint 1–2, how those primitives become normalized version rules, and how downstream jobs query them deterministically.
|
||||
|
||||
## 1. What landed in Sprint 1–2
|
||||
|
||||
- `RangePrimitives.SemVer` now infers a canonical `style` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`) and captures `exactValue` when the constraint is a single version.
|
||||
- `NormalizedVersionRule` documents the analytics-friendly projection of each `AffectedPackage` coverage entry and is persisted alongside legacy `versionRanges`.
|
||||
- `AdvisoryProvenance.decisionReason` records whether merge resolution favored precedence, freshness, or a tie-breaker comparison.
|
||||
|
||||
See `src/Concelier/__Libraries/StellaOps.Concelier.Models/CANONICAL_RECORDS.md` for the full schema and field descriptions.
|
||||
|
||||
## 2. Mapper pattern
|
||||
|
||||
Connectors should emit SemVer primitives as soon as they can normalize a vendor constraint. The helper `SemVerPrimitiveExtensions.ToNormalizedVersionRule` turns those primitives into the persisted rules:
|
||||
|
||||
```csharp
|
||||
var primitive = new SemVerPrimitive(
|
||||
introduced: "1.2.3",
|
||||
introducedInclusive: true,
|
||||
fixed: "2.0.0",
|
||||
fixedInclusive: false,
|
||||
lastAffected: null,
|
||||
lastAffectedInclusive: false,
|
||||
constraintExpression: ">=1.2.3 <2.0.0",
|
||||
exactValue: null);
|
||||
|
||||
var rule = primitive.ToNormalizedVersionRule(notes: "nvd:CVE-2025-1234");
|
||||
// rule => scheme=semver, type=range, min=1.2.3, minInclusive=true, max=2.0.0, maxInclusive=false
|
||||
```
|
||||
|
||||
If you omit the optional `notes` argument, `ToNormalizedVersionRule` now falls back to the primitive’s `ConstraintExpression`, ensuring the original comparator expression is preserved for provenance/audit queries.
|
||||
|
||||
Emit the resulting rule inside `AffectedPackage.NormalizedVersions` while continuing to populate `AffectedVersionRange.RangeExpression` for backward compatibility.
|
||||
|
||||
## 3. Merge dedupe flow
|
||||
|
||||
During merge, feed all package candidates through `NormalizedVersionRuleComparer.Instance` prior to persistence. The comparer orders by scheme → type → min → minInclusive → max → maxInclusive → value → notes, guaranteeing consistent document layout and making `$unwind` pipelines deterministic.
|
||||
|
||||
If multiple connectors emit identical constraints, the merge layer should:
|
||||
|
||||
1. Combine provenance entries (preserving one per source).
|
||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||
|
||||
## 4. Example Mongo pipeline
|
||||
|
||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||
|
||||
```javascript
|
||||
db.advisories.aggregate([
|
||||
{ $match: { "affectedPackages.type": "semver", "affectedPackages.identifier": "pkg:npm/lodash" } },
|
||||
{ $unwind: "$affectedPackages" },
|
||||
{ $unwind: "$affectedPackages.normalizedVersions" },
|
||||
{ $match: {
|
||||
$or: [
|
||||
{ "affectedPackages.normalizedVersions.type": "exact",
|
||||
"affectedPackages.normalizedVersions.value": "4.17.21" },
|
||||
{ "affectedPackages.normalizedVersions.type": "range",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" },
|
||||
"affectedPackages.normalizedVersions.max": { $gt: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "gte",
|
||||
"affectedPackages.normalizedVersions.min": { $lte: "4.17.21" } },
|
||||
{ "affectedPackages.normalizedVersions.type": "lte",
|
||||
"affectedPackages.normalizedVersions.max": { $gte: "4.17.21" } }
|
||||
]
|
||||
}},
|
||||
{ $project: { advisoryKey: 1, title: 1, "affectedPackages.identifier": 1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
Pair this query with the indexes listed in [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 5. Recommended indexes
|
||||
|
||||
| Collection | Index | Purpose |
|
||||
|------------|-------|---------|
|
||||
| `advisory` | `{ "affectedPackages.identifier": 1, "affectedPackages.normalizedVersions.scheme": 1, "affectedPackages.normalizedVersions.type": 1 }` (compound, multikey) | Speeds up `$match` on identifier + rule style. |
|
||||
| `advisory` | `{ "affectedPackages.normalizedVersions.value": 1 }` (sparse) | Optimizes lookups for exact version hits. |
|
||||
|
||||
Coordinate with the Storage team when enabling these indexes so deployment windows account for collection size.
|
||||
|
||||
## 6. Dual-write rollout
|
||||
|
||||
Follow the operational checklist in `docs/modules/devops/migrations/semver-style.md`. The summary:
|
||||
|
||||
1. **Dual write (now)** – emit both legacy `versionRanges` and the new `normalizedVersions`.
|
||||
2. **Backfill** – follow the storage migration in `docs/modules/devops/migrations/semver-style.md` to rewrite historical advisories before switching consumers.
|
||||
3. **Verify** – run the aggregation above (with `explain("executionStats")`) to ensure the new indexes are used.
|
||||
4. **Cutover** – after consumers switch to normalized rules, mark the old `rangeExpression` as deprecated.
|
||||
|
||||
## 7. Checklist for connectors & merge
|
||||
|
||||
- [ ] Populate `SemVerPrimitive` for every SemVer-friendly constraint.
|
||||
- [ ] Call `ToNormalizedVersionRule` and store the result.
|
||||
- [ ] Emit provenance masks covering both `versionRanges[].primitives.semver` and `normalizedVersions[]`.
|
||||
- [ ] Ensure merge deduping relies on the canonical comparer.
|
||||
- [ ] Capture merge decisions via `decisionReason`.
|
||||
- [ ] Confirm integration tests include fixtures with normalized rules and SemVer styles.
|
||||
|
||||
For deeper query examples and maintenance tasks, continue with [Normalized Versions Query Guide](mongo_indices.md).
|
||||
|
||||
## 8. Storage projection reference
|
||||
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "pkg:npm/example",
|
||||
"packageType": "npm",
|
||||
"scheme": "semver",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "1.2.3",
|
||||
"minInclusive": true,
|
||||
"max": "2.0.0",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "ghsa:GHSA-xxxx-yyyy",
|
||||
"decisionReason": "ghsa-precedence-over-nvd",
|
||||
"constraint": ">= 1.2.3 < 2.0.0",
|
||||
"source": "ghsa",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
For distro-specific ranges (`nevra`, `evr`) the same envelope applies with `scheme` switched accordingly. Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"packageId": "bash",
|
||||
"packageType": "rpm",
|
||||
"scheme": "nevra",
|
||||
"type": "range",
|
||||
"style": "range",
|
||||
"min": "0:4.4.18-2.el7",
|
||||
"minInclusive": true,
|
||||
"max": "0:4.4.20-1.el7",
|
||||
"maxInclusive": false,
|
||||
"value": null,
|
||||
"notes": "redhat:RHSA-2025:1234",
|
||||
"decisionReason": "rhel-priority-over-nvd",
|
||||
"constraint": "<= 0:4.4.20-1.el7",
|
||||
"source": "redhat",
|
||||
"recordedAt": "2025-10-11T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
If a new scheme is required (for example, `apple.build` or `ios.semver`), raise it with the Models team before emitting documents so merge comparers and hashing logic can incorporate the change deterministically.
|
||||
|
||||
## 9. Observability signals
|
||||
|
||||
- `concelier.merge.normalized_rules` (counter, tags: `package_type`, `scheme`) – increments once per normalized rule retained after precedence merge.
|
||||
- `concelier.merge.normalized_rules_missing` (counter, tags: `package_type`) – increments when a merged package still carries version ranges but no normalized rules; watch for spikes to catch connectors that have not emitted normalized arrays yet.
|
||||
|
||||
56
docs/dev/raw-linkset-backfill-plan.md
Normal file
56
docs/dev/raw-linkset-backfill-plan.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Raw Linkset Backfill & Adoption Plan
|
||||
|
||||
_Last updated: 2025-10-31_
|
||||
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
|
||||
|
||||
## Context
|
||||
|
||||
- Concelier observations now emit both a **canonical linkset** (deduped, normalised identifiers) and a **raw linkset** (`rawLinkset`) that preserves upstream ordering, duplicates, and original pointer metadata.
|
||||
- Existing `concelier.advisory_observations` documents created before 2025-10-31 do **not** contain the `rawLinkset` field.
|
||||
- Policy Engine selection joiners (`POLICY-ENGINE-20-003`) will switch to the raw projection once backfill completes and consumers validate fixtures.
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Populate `rawLinkset` for historical observations across online clusters and Offline Kit bundles without breaking append-only guarantees.
|
||||
2. Provide migration scripts + runbook so operators can rehearse in staging (and air-gapped deployments) before production rollout.
|
||||
3. Unblock Policy Engine adoption by guaranteeing dual projections exist for all tenants.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- [ ] **Migration script** (`20251104_advisory_observations_raw_linkset_backfill.csx`)
|
||||
- Iterates observations lacking `rawLinkset`
|
||||
- Rehydrates raw document via existing snapshot (or cached DTO)
|
||||
- Reuses `AdvisoryObservationFactory.CreateRawLinkset`
|
||||
- Writes using `$set` with optimistic retry; preserves `updatedAt` via `setOnInsert`
|
||||
- [ ] **Offline Kit updater** (extend `ops/offline-kit/scripts/export_offline_bundle.py`) to patch bundles in-place
|
||||
- [ ] **Runbook** covering:
|
||||
- Pre-check query: `db.concelier.advisory_observations.countDocuments({ rawLinkset: { $exists: false } })`
|
||||
- Backup procedure (`mongodump` or snapshot requirement)
|
||||
- Dry-run mode limiting batches by tenant
|
||||
- Metrics/telemetry expectations (`concelier.migrations.documents_processed_total`)
|
||||
- Rollback (no-op because field addition; note to retain snapshot for verification)
|
||||
- [ ] **Fixture updates** ensuring storage/CLI/Policy tests include `rawLinkset`
|
||||
- [ ] **Policy Engine follow-up** to flip joiners once `rawLinkset` population reaches 100% (tracked via metrics).
|
||||
|
||||
## Timeline
|
||||
|
||||
| Date (UTC) | Milestone | Notes |
|
||||
|------------|-----------|-------|
|
||||
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
|
||||
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
|
||||
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
|
||||
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations. |
|
||||
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
|
||||
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Do we need archival of the canonical-only projection for backwards compatibility exports? (Policy to confirm.)
|
||||
- Offline Kit delta: should we regenerate entire bundle or ship incremental patch? (DevOps reviewing.)
|
||||
- Metrics: add `raw_linkset_missing_total` counter to detect regressions post-backfill?
|
||||
|
||||
## Next Actions
|
||||
|
||||
- [ ] Concelier Storage Guild: prototype migration script, share for review (`2025-11-01`).
|
||||
- [ ] DevOps Guild: schedule staging rehearsal + update `docs/deploy/containers.md` with new runbook section.
|
||||
- [ ] Policy Guild: prepare feature flag/branch to switch joiners once metrics show zero missing `rawLinkset`.
|
||||
@@ -1,39 +1,39 @@
|
||||
# Scanner Orchestrator Events (ORCH-SVC-38-101)
|
||||
|
||||
Last updated: 2025-10-26
|
||||
|
||||
The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events.
|
||||
|
||||
## 1. Envelope overview
|
||||
|
||||
Orchestrator events share a deterministic JSON envelope:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `eventId` | `uuid` | Globally unique identifier generated per occurrence. |
|
||||
| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. |
|
||||
| `version` | `integer` | Schema version. Initial release uses `1`. |
|
||||
| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. |
|
||||
| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). |
|
||||
| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. |
|
||||
| `source` | `string` | Producer identifier (`scanner.webservice`). |
|
||||
| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). |
|
||||
| `correlationId` | `string` | Maps back to the API request or scan identifier. |
|
||||
| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. |
|
||||
| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. |
|
||||
| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). |
|
||||
| `payload` | `object` | Event-specific body (see §2). |
|
||||
|
||||
Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`.
|
||||
|
||||
## 2. Event kinds and payloads
|
||||
|
||||
### 2.1 `scanner.event.report.ready`
|
||||
|
||||
Emitted once a signed report is persisted and attested. Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing.
|
||||
# Scanner Orchestrator Events (ORCH-SVC-38-101)
|
||||
|
||||
Last updated: 2025-10-26
|
||||
|
||||
The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events.
|
||||
|
||||
## 1. Envelope overview
|
||||
|
||||
Orchestrator events share a deterministic JSON envelope:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `eventId` | `uuid` | Globally unique identifier generated per occurrence. |
|
||||
| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. |
|
||||
| `version` | `integer` | Schema version. Initial release uses `1`. |
|
||||
| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. |
|
||||
| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). |
|
||||
| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. |
|
||||
| `source` | `string` | Producer identifier (`scanner.webservice`). |
|
||||
| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). |
|
||||
| `correlationId` | `string` | Maps back to the API request or scan identifier. |
|
||||
| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. |
|
||||
| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. |
|
||||
| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). |
|
||||
| `payload` | `object` | Event-specific body (see §2). |
|
||||
|
||||
Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`.
|
||||
|
||||
## 2. Event kinds and payloads
|
||||
|
||||
### 2.1 `scanner.event.report.ready`
|
||||
|
||||
Emitted once a signed report is persisted and attested. Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing.
|
||||
- **Links:**
|
||||
- `report.ui` → `/ui/reports/{reportId}` on the current host.
|
||||
- `report.api` → `{apiBasePath}/{reportsSegment}/{reportId}` (defaults to `/api/v1/reports/{reportId}`).
|
||||
@@ -41,83 +41,83 @@ Emitted once a signed report is persisted and attested. Payload highlights:
|
||||
- `policy.api` → `{apiBasePath}/{policySegment}/revisions/{revisionId}` when a revision is present.
|
||||
- `attestation.ui` → `/ui/attestations/{reportId}` when a DSSE envelope is included.
|
||||
- `attestation.api` → `{apiBasePath}/{reportsSegment}/{reportId}/attestation` when a DSSE envelope is included.
|
||||
- `imageDigest` — OCI image digest associated with the analysis.
|
||||
- `generatedAt` — report generation timestamp (ISO-8601 UTC).
|
||||
- `verdict` — `pass`, `warn`, or `fail` after policy evaluation.
|
||||
- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers).
|
||||
- `delta` — newly critical/high counts and optional `kev` array.
|
||||
- `quietedFindingCount` — mirrors `summary.quieted`.
|
||||
- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing.
|
||||
- `links` — UI/report/policy URLs suitable for operators.
|
||||
- `dsse` — embedded DSSE envelope (payload, type, signature list).
|
||||
- `report` — canonical report document; identical to the DSSE payload.
|
||||
|
||||
Schema: `docs/events/scanner.event.report.ready@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json`
|
||||
|
||||
### 2.2 `scanner.event.scan.completed`
|
||||
|
||||
Emitted after scan execution finishes (success or policy failure). Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`.
|
||||
- `imageDigest` — OCI image digest associated with the analysis.
|
||||
- `generatedAt` — report generation timestamp (ISO-8601 UTC).
|
||||
- `verdict` — `pass`, `warn`, or `fail` after policy evaluation.
|
||||
- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers).
|
||||
- `delta` — newly critical/high counts and optional `kev` array.
|
||||
- `quietedFindingCount` — mirrors `summary.quieted`.
|
||||
- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing.
|
||||
- `links` — UI/report/policy URLs suitable for operators.
|
||||
- `dsse` — embedded DSSE envelope (payload, type, signature list).
|
||||
- `report` — canonical report document; identical to the DSSE payload.
|
||||
|
||||
Schema: `docs/events/scanner.event.report.ready@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json`
|
||||
|
||||
### 2.2 `scanner.event.scan.completed`
|
||||
|
||||
Emitted after scan execution finishes (success or policy failure). Payload highlights:
|
||||
|
||||
- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim.
|
||||
- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`.
|
||||
- **Links:** same as above (`report.*`, `policy.*`) with `attestation.*` populated when DSSE metadata exists.
|
||||
- `verdict`, `summary`, `delta`, `policy` — same semantics as above.
|
||||
- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`.
|
||||
- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures).
|
||||
|
||||
Schema: `docs/events/scanner.event.scan.completed@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json`
|
||||
|
||||
### 2.3 Relationship to legacy events
|
||||
|
||||
| Legacy Redis event | Replacement orchestrator event | Notes |
|
||||
|--------------------|-------------------------------|-------|
|
||||
| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. |
|
||||
| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. |
|
||||
|
||||
Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants.
|
||||
|
||||
## 3. Deterministic serialization
|
||||
|
||||
- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace.
|
||||
- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`.
|
||||
- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable.
|
||||
- Optional fields are omitted rather than emitted as `null`.
|
||||
|
||||
## 4. Idempotency and correlation
|
||||
|
||||
Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern:
|
||||
|
||||
| Event kind | Idempotency key template |
|
||||
|------------|-------------------------|
|
||||
| `scanner.event.report.ready` | `scanner.event.report.ready:<tenant>:<reportId>` |
|
||||
| `scanner.event.scan.completed` | `scanner.event.scan.completed:<tenant>:<scanId>` |
|
||||
|
||||
Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key.
|
||||
|
||||
`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together.
|
||||
|
||||
## 5. Versioning and evolution
|
||||
|
||||
- Increment the `version` field and the `@<version>` suffix for **breaking** changes (field removals, type changes, semantic shifts).
|
||||
- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly.
|
||||
- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration.
|
||||
|
||||
## 6. Consumer checklist
|
||||
|
||||
1. Validate incoming payloads against the schema for the targeted version.
|
||||
2. Use `idempotencyKey` for dedupe, not `eventId`.
|
||||
3. Map `traceId`/`spanId` into telemetry spans to preserve causality.
|
||||
4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing.
|
||||
5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101.
|
||||
|
||||
## 7. Implementation status and next actions
|
||||
|
||||
- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (doing) track the production of these envelopes. The remaining blocker is the .NET 10 preview OpenAPI/Auth dependency drift that currently breaks `dotnet test`. Once Gateway and Notifier owners land the replacement packages, rerun the full test suite and capture fresh fixtures under `docs/events/samples/`.
|
||||
- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here.
|
||||
- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/modules/devops/runbooks/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/modules/devops/runbooks/launch-readiness.md` so they land in the launch checklist.
|
||||
|
||||
---
|
||||
|
||||
**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
- `verdict`, `summary`, `delta`, `policy` — same semantics as above.
|
||||
- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`.
|
||||
- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures).
|
||||
|
||||
Schema: `docs/events/scanner.event.scan.completed@1.json`
|
||||
Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json`
|
||||
|
||||
### 2.3 Relationship to legacy events
|
||||
|
||||
| Legacy Redis event | Replacement orchestrator event | Notes |
|
||||
|--------------------|-------------------------------|-------|
|
||||
| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. |
|
||||
| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. |
|
||||
|
||||
Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants.
|
||||
|
||||
## 3. Deterministic serialization
|
||||
|
||||
- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace.
|
||||
- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`.
|
||||
- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable.
|
||||
- Optional fields are omitted rather than emitted as `null`.
|
||||
|
||||
## 4. Idempotency and correlation
|
||||
|
||||
Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern:
|
||||
|
||||
| Event kind | Idempotency key template |
|
||||
|------------|-------------------------|
|
||||
| `scanner.event.report.ready` | `scanner.event.report.ready:<tenant>:<reportId>` |
|
||||
| `scanner.event.scan.completed` | `scanner.event.scan.completed:<tenant>:<scanId>` |
|
||||
|
||||
Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key.
|
||||
|
||||
`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together.
|
||||
|
||||
## 5. Versioning and evolution
|
||||
|
||||
- Increment the `version` field and the `@<version>` suffix for **breaking** changes (field removals, type changes, semantic shifts).
|
||||
- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly.
|
||||
- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration.
|
||||
|
||||
## 6. Consumer checklist
|
||||
|
||||
1. Validate incoming payloads against the schema for the targeted version.
|
||||
2. Use `idempotencyKey` for dedupe, not `eventId`.
|
||||
3. Map `traceId`/`spanId` into telemetry spans to preserve causality.
|
||||
4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing.
|
||||
5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101.
|
||||
|
||||
## 7. Implementation status and next actions
|
||||
|
||||
- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (doing) track the production of these envelopes. The remaining blocker is the .NET 10 preview OpenAPI/Auth dependency drift that currently breaks `dotnet test`. Once Gateway and Notifier owners land the replacement packages, rerun the full test suite and capture fresh fixtures under `docs/events/samples/`.
|
||||
- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here.
|
||||
- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/modules/devops/runbooks/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/modules/devops/runbooks/launch-readiness.md` so they land in the launch checklist.
|
||||
|
||||
---
|
||||
|
||||
**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,208 +1,208 @@
|
||||
Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field<br>Instructions to work:<br>AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/Concelier/__Libraries/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection<br>Instructions to work:<br>`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence<br>Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Updated constructors/tests keep storage suites passing with the new feature flag defaults. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options<br>WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding<br>Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through<br>`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/__Libraries/__Tests/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/Authority/StellaOps.Authority/StellaOps.Authority.sln` returns success. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish<br>Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking<br>Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.<br>Team instructions: Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher<br>Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation<br>Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives<br>VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests<br>Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation<br>`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation<br>Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff<br>Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up<br>Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan<br>Staged reintegration plan published in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation<br>Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep<br>Instructions to work:<br>Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives<br>Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests<br>Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation<br>Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep<br>Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling<br>`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements<br>Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs, tests, and docs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption<br>Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. |
|
||||
| Sprint 1 | Bootstrap & Replay Hardening | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. |
|
||||
| Sprint 1 | Developer Tooling | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.<br>CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe<br>Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/modules/concelier/operations/connectors/cccs.md` with fixtures validating EN/FR list handling. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/modules/concelier/operations/connectors/certbund.md` captures locale guidance and offline packaging. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/modules/concelier/operations/connectors/msrc.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh<br>Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook<br>Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide<br>Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.<br>Instructions to work:<br>DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation<br>`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map<br>Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.<br>Instructions to work:<br>Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep<br>Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>Dual-write/backfill flag delivered; migration + options validated in tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Storage tests adjusted for normalized versions/decision reasons.<br>Instructions to work:<br>Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules<br>Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation<br>Merge events capture per-field decisions; counters/logs align with conflict rules. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline<br>Canonical merge preserves unions with updated tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite<br>Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.<br>Instructions to work:<br>Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures<br>Instructions to work:<br>Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules<br>Runbook published at `docs/modules/concelier/operations/conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout<br>Ops review completed, alert thresholds applied, and change log appended in `docs/modules/concelier/operations/conflict-resolution.md`; task closed after connector signals verified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)<br>Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields<br>Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup<br>Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields<br>Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields<br>GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter<br>Update schemas/offline bundle + fixtures once model/core parity lands.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package<br>Extend Bolt builder, metadata, and regression tests for the expanded schema.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `modules/scanner/architecture.md` §3–§4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-004 | Stabilize descriptor nonce derivation so repeated builds emit deterministic placeholders. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-005 | Integrate determinism guard into GitHub/Gitea workflows and archive proof artifacts. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-205 | Harden heartbeat jitter so lease safety margin stays ≥3× and cover with regression tests + optional live queue smoke run. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). |
|
||||
| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors.<br>Progress 2025-10-20: Coordination matrix + rollout dashboard refreshed; upcoming deadlines tracked (Cccs/Cisco 2025-10-21, CertBund 2025-10-22, ICS-CISA 2025-10-23, KISA 2025-10-24) with escalation path documented in FEEDMERGE-COORD-02-900.|
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – immutable VEX statements store, consensus signal fields, and migration `20251019-consensus-signals-statements` with tests (`dotnet test src/Excititor/__Tests/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj`, `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Concelier WebService Guild | FEEDWEB-EVENTS-07-001 | Advisory event replay API – expose `/concelier/advisories/{key}/replay` with `asOf` filter, hex hashes, and conflict data. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-20) | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. |
|
||||
| Sprint 8 | Mongo strengthening | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions<br>Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. |
|
||||
| Sprint 8 | Mongo strengthening | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage<br>Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. |
|
||||
| Sprint 8 | Mongo strengthening | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories<br>Session-scoped repositories shipped with new Mongo records, orchestrators/workers now share scoped sessions, and replica-set failover coverage added via `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`. |
|
||||
| Sprint 8 | Platform Maintenance | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling – shipped admin backfill endpoint, CLI hook (`stellaops excititor backfill-statements`), integration tests, and operator runbook (`docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`). |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-20) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. |
|
||||
| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-003 | Refactor Authority identity-provider registry to resolve scoped plugin services on-demand.<br>Introduce factory pattern aligned with scoped lifetimes decided in coordination workshop. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-004 | Update Authority plugin loader to activate registrars with DI support and scoped service awareness.<br>Add two-phase initialization allowing scoped dependencies post-container build. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-005 | Provide scoped-safe bootstrap execution for Authority plugins.<br>Implement scope-per-run pattern for hosted bootstrap tasks and document migration guidance. |
|
||||
| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DONE (2025-10-20) | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave 0A prerequisites confirmed complete before remediation work. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-20) | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/StellaOps.Notify.WebService/TASKS.md | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-604 | Teams health endpoint metadata alignment. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Email/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-20) | Scanner WebService Guild | SCANNER-EVENTS-15-201 | Emit `scanner.report.ready` + `scanner.scan.completed` events. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Webhook/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-20) | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-20) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). |
|
||||
Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field<br>Instructions to work:<br>AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/Concelier/__Libraries/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection<br>Instructions to work:<br>`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/__Libraries/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence<br>Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Updated constructors/tests keep storage suites passing with the new feature flag defaults. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options<br>WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding<br>Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through<br>`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/Authority/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/__Libraries/__Tests/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/Authority/StellaOps.Authority/StellaOps.Authority.sln` returns success. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish<br>Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking<br>Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.<br>Team instructions: Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher<br>Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation<br>Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives<br>VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests<br>Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation<br>`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation<br>Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff<br>Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up<br>Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan<br>Staged reintegration plan published in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation<br>Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep<br>Instructions to work:<br>Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives<br>Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests<br>Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation<br>Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep<br>Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling<br>`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements<br>Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning<br>Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns<br>Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff<br>Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs, tests, and docs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption<br>Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. |
|
||||
| Sprint 1 | Bootstrap & Replay Hardening | src/__Libraries/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. |
|
||||
| Sprint 1 | Developer Tooling | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.<br>CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter<br>`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing<br>Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe<br>Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/modules/concelier/operations/connectors/cccs.md` with fixtures validating EN/FR list handling. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/modules/concelier/operations/connectors/certbund.md` captures locale guidance and offline packaging. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/modules/concelier/operations/connectors/msrc.md`. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh<br>Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook<br>Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. |
|
||||
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide<br>Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.<br>Instructions to work:<br>DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation<br>`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map<br>Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.<br>Instructions to work:<br>Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep<br>Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill<br>Dual-write/backfill flag delivered; migration + options validated in tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor<br>Storage tests adjusted for normalized versions/decision reasons.<br>Instructions to work:<br>Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules<br>Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation<br>Merge events capture per-field decisions; counters/logs align with conflict rules. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline<br>Canonical merge preserves unions with updated tests. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite<br>Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.<br>Instructions to work:<br>Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures<br>Instructions to work:<br>Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules<br>Runbook published at `docs/modules/concelier/operations/conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. |
|
||||
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout<br>Ops review completed, alert thresholds applied, and change log appended in `docs/modules/concelier/operations/conflict-resolution.md`; task closed after connector signals verified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)<br>Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields<br>Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup<br>Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields<br>Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields<br>GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter<br>Update schemas/offline bundle + fixtures once model/core parity lands.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package<br>Extend Bolt builder, metadata, and regression tests for the expanded schema.<br>2025-10-15: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. |
|
||||
| Sprint 5 | Excititor Core Foundations | src/Excititor/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Excititor/__Libraries/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. |
|
||||
| Sprint 6 | Excititor Ingest & Formats | src/Cli/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `modules/scanner/architecture.md` §3–§4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-004 | Stabilize descriptor nonce derivation so repeated builds emit deterministic placeholders. |
|
||||
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-005 | Integrate determinism guard into GitHub/Gitea workflows and archive proof artifacts. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-205 | Harden heartbeat jitter so lease safety margin stays ≥3× and cover with regression tests + optional live queue smoke run. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. |
|
||||
| Sprint 9 | Policy Foundations | src/Policy/__Libraries/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). |
|
||||
| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. |
|
||||
| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. |
|
||||
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. |
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors.<br>Progress 2025-10-20: Coordination matrix + rollout dashboard refreshed; upcoming deadlines tracked (Cccs/Cisco 2025-10-21, CertBund 2025-10-22, ICS-CISA 2025-10-23, KISA 2025-10-24) with escalation path documented in FEEDMERGE-COORD-02-900.|
|
||||
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – immutable VEX statements store, consensus signal fields, and migration `20251019-consensus-signals-statements` with tests (`dotnet test src/Excititor/__Tests/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj`, `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Concelier WebService Guild | FEEDWEB-EVENTS-07-001 | Advisory event replay API – expose `/concelier/advisories/{key}/replay` with `asOf` filter, hex hashes, and conflict data. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-20) | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. |
|
||||
| Sprint 8 | Mongo strengthening | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions<br>Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. |
|
||||
| Sprint 8 | Mongo strengthening | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage<br>Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. |
|
||||
| Sprint 8 | Mongo strengthening | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories<br>Session-scoped repositories shipped with new Mongo records, orchestrators/workers now share scoped sessions, and replica-set failover coverage added via `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`. |
|
||||
| Sprint 8 | Platform Maintenance | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling – shipped admin backfill endpoint, CLI hook (`stellaops excititor backfill-statements`), integration tests, and operator runbook (`docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`). |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. |
|
||||
| Sprint 8 | Mirror Distribution | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-20) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. |
|
||||
| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-003 | Refactor Authority identity-provider registry to resolve scoped plugin services on-demand.<br>Introduce factory pattern aligned with scoped lifetimes decided in coordination workshop. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-004 | Update Authority plugin loader to activate registrars with DI support and scoped service awareness.<br>Add two-phase initialization allowing scoped dependencies post-container build. |
|
||||
| Sprint 8 | Plugin Infrastructure | src/__Libraries/StellaOps.Plugin/TASKS.md | DONE (2025-10-20) | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-005 | Provide scoped-safe bootstrap execution for Authority plugins.<br>Implement scope-per-run pattern for hosted bootstrap tasks and document migration guidance. |
|
||||
| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DONE (2025-10-20) | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave 0A prerequisites confirmed complete before remediation work. |
|
||||
| Sprint 11 | Signing Chain Bring-up | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-20) | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/StellaOps.Notify.WebService/TASKS.md | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-604 | Teams health endpoint metadata alignment. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Slack/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Teams/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Email/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 15 | Notify Foundations | src/Scanner/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-20) | Scanner WebService Guild | SCANNER-EVENTS-15-201 | Emit `scanner.report.ready` + `scanner.scan.completed` events. |
|
||||
| Sprint 15 | Notify Foundations | src/Notify/__Libraries/StellaOps.Notify.Connectors.Webhook/TASKS.md | DONE (2025-10-20) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-20) | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Queue/TASKS.md | DONE (2025-10-20) | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-20) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). |
|
||||
|
||||
@@ -1,84 +1,84 @@
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-NUGET-13-002 | Ensure all solutions/projects prioritize `local-nuget` before public feeds and add restore-order validation. |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-003 | Upgrade `Microsoft.*` dependencies pinned to 8.* to their latest .NET 10 (or 9.x) releases and refresh guidance. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | DONE (2025-10-26) | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Scanner Guild | DEVOPS-REL-14-004 | Extend release/offline smoke jobs to cover Python analyzer plug-ins (warm/cold, determinism, signing). |
|
||||
| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | DONE (2025-10-26) | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. |
|
||||
| Sprint 15 | Benchmarks | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit/run stats materialization for UI. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, DevOps Guild | DEVOPS-OFFLINE-17-003 | Mirror release debug-store artefacts into Offline Kit packaging and document validation. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-26) | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. |
|
||||
| Sprint 18 | Launch Readiness | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-LAUNCH-18-001 | Production launch cutover rehearsal and runbook publication. |
|
||||
| Sprint 18 | Launch Readiness | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, Scanner Guild | DEVOPS-OFFLINE-18-005 | Rebuild Offline Kit with Python analyzer artefacts and refreshed manifest/signature pair. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-AOC-19-001 | Publish aggregation-only contract reference documentation. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-AOC-19-002 | Update architecture overview with AOC boundary diagrams. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-AOC-19-003 | Refresh policy engine doc with raw ingestion constraints. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-AOC-19-004 | Document console AOC dashboard and drill-down flow. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-AOC-19-005 | Document CLI AOC commands and exit codes. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-AOC-19-006 | Document new AOC metrics, traces, and logs. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-AOC-19-007 | Document new Authority scopes and tenancy enforcement. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, DevOps Guild | DOCS-AOC-19-008 | Update deployment guide with validator enablement and verify user guidance. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-AOC-19-001 | Introduce new ingestion/auth scopes across Authority. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-001 | Publish `/docs/policy/overview.md` with compliance checklist. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-002 | Document DSL grammar + examples in `/docs/policy/dsl.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-POLICY-20-003 | Write `/docs/policy/lifecycle.md` covering workflow + roles. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Scheduler Guild | DOCS-POLICY-20-004 | Document policy run modes + cursors in `/docs/policy/runs.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Platform Guild | DOCS-POLICY-20-005 | Produce `/docs/api/policy.md` with endpoint schemas + errors. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-POLICY-20-006 | Author `/docs/modules/cli/guides/policy.md` with commands, exit codes, JSON output. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-POLICY-20-007 | Create `/docs/ui/policy-editor.md` covering editor, simulation, approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-POLICY-20-008 | Publish `/docs/modules/policy/architecture.md` with sequence diagrams. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-POLICY-20-009 | Document metrics/traces/logs in `/docs/observability/policy.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Security Guild | DOCS-POLICY-20-010 | Publish `/docs/security/policy-governance.md` for scopes + approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-POLICY-20-011 | Add example policies under `/docs/examples/policies/` with commentary. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Support Guild | DOCS-POLICY-20-012 | Draft `/docs/faq/policy-faq.md` covering conflicts, determinism, pitfalls. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-POLICY-20-001 | Add DSL lint + compile checks to CI pipelines. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, QA Guild | DEVOPS-POLICY-20-003 | Add determinism CI job diffing repeated policy runs. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, Policy Guild | SAMPLES-POLICY-20-001 | Commit baseline/serverless/internal-only policy samples + fixtures. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, UI Guild | SAMPLES-POLICY-20-002 | Produce simulation diff fixtures for UI/CLI tests. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-001 | Add new policy scopes (`policy:*`, `findings:read`, `effective:write`). |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-002 | Enforce Policy Engine service identity and scope checks at gateway. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-POLICY-20-003 | Update Authority docs/config samples for policy scopes + workflows. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Policy Guild | BENCH-POLICY-20-001 | Create policy evaluation benchmark suite + baseline metrics. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild, Platform Guild | POLICY-ENGINE-20-000 | Spin up new Policy Engine service host with DI bootstrap and Authority wiring. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild | POLICY-ENGINE-20-001 | Deliver `stella-dsl@1` parser + IR compiler with diagnostics and checksums. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-20-001 | Define policy run/diff DTOs + validation helpers. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-001 | Introduce graph scopes (`graph:*`) with configuration binding and defaults. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-002 | Enforce graph scopes/identities at gateway with tenant propagation. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-GRAPH-21-003 | Update security docs/config samples for graph access and least privilege. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-001 | Define job DTOs for graph builds/overlay refresh (`GraphBuildJob`, `GraphOverlayJob`) with deterministic serialization and status enums; document in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-21-001-GRAPH-JOBS.md`. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-002 | Publish schema docs/sample payloads for graph job lifecycle. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-001 | Benchmark advisory observation ingest/correlation throughput. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-002 | Benchmark VEX ingest/correlation latency and event emission. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-001 | Publish `/docs/ui/console-overview.md` (IA, tenant model, filters, AOC alignment). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-002 | Author `/docs/ui/navigation.md` with route map, filters, keyboard shortcuts, deep links. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-003 | Document `/docs/ui/sbom-explorer.md` covering catalog, graph, overlays, exports. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-004 | Produce `/docs/ui/advisories-and-vex.md` detailing aggregation-not-merge UX. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-005 | Write `/docs/ui/findings.md` with filters, explain, exports, CLI parity notes. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-006 | Publish `/docs/ui/policies.md` (editor, simulation, approvals, RBAC). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-007 | Document `/docs/ui/runs.md` with SSE monitoring, diff, retries, evidence downloads. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-008 | Draft `/docs/ui/admin.md` covering tenants, roles, tokens, integrations, fresh-auth. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild | DOCS-CONSOLE-23-009 | Publish `/docs/ui/downloads.md` aligning manifest with commands and offline flow. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild, Deployment Guild, Console Guild | DOCS-CONSOLE-23-010 | Write `/docs/deploy/console.md` (Helm, ingress, TLS, env vars, health checks). |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-001 | Provide graph build/overlay job APIs; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-002 | Provide overlay lag metrics endpoint/webhook; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild, Authority Core Guild | SCHED-WEB-21-003 | Replace header auth with Authority scopes using `StellaOpsScopes`; dev fallback only when `Scheduler:Authority:Enabled=false`. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-001 | Deploy default OpenTelemetry collector manifests with secure OTLP pipeline. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-003 | Package telemetry stack configs for offline/air-gapped installs with signatures. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. |
|
||||
This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not).
|
||||
|
||||
| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-NUGET-13-002 | Ensure all solutions/projects prioritize `local-nuget` before public feeds and add restore-order validation. |
|
||||
| Sprint 13 | Platform Reliability | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Platform Leads | DEVOPS-NUGET-13-003 | Upgrade `Microsoft.*` dependencies pinned to 8.* to their latest .NET 10 (or 9.x) releases and refresh guidance. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | DONE (2025-10-26) | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, Scanner Guild | DEVOPS-REL-14-004 | Extend release/offline smoke jobs to cover Python analyzer plug-ins (warm/cold, determinism, signing). |
|
||||
| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | DONE (2025-10-26) | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. |
|
||||
| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. |
|
||||
| Sprint 15 | Benchmarks | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-19) | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Storage.Mongo/TASKS.md | DONE (2025-10-26) | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit/run stats materialization for UI. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md | DONE (2025-10-26) | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, DevOps Guild | DEVOPS-OFFLINE-17-003 | Mirror release debug-store artefacts into Offline Kit packaging and document validation. |
|
||||
| Sprint 17 | Symbol Intelligence & Forensics | src/Scanner/__Libraries/StellaOps.Scanner.Emit/TASKS.md | DONE (2025-10-26) | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. |
|
||||
| Sprint 18 | Launch Readiness | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-LAUNCH-18-001 | Production launch cutover rehearsal and runbook publication. |
|
||||
| Sprint 18 | Launch Readiness | ops/offline-kit/TASKS.md | DONE (2025-10-26) | Offline Kit Guild, Scanner Guild | DEVOPS-OFFLINE-18-005 | Rebuild Offline Kit with Python analyzer artefacts and refreshed manifest/signature pair. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-AOC-19-001 | Publish aggregation-only contract reference documentation. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-AOC-19-002 | Update architecture overview with AOC boundary diagrams. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-AOC-19-003 | Refresh policy engine doc with raw ingestion constraints. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-AOC-19-004 | Document console AOC dashboard and drill-down flow. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-AOC-19-005 | Document CLI AOC commands and exit codes. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-AOC-19-006 | Document new AOC metrics, traces, and logs. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-AOC-19-007 | Document new Authority scopes and tenancy enforcement. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, DevOps Guild | DOCS-AOC-19-008 | Update deployment guide with validator enablement and verify user guidance. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-AOC-19-001 | Introduce new ingestion/auth scopes across Authority. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-001 | Publish `/docs/policy/overview.md` with compliance checklist. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-POLICY-20-002 | Document DSL grammar + examples in `/docs/policy/dsl.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-POLICY-20-003 | Write `/docs/policy/lifecycle.md` covering workflow + roles. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Scheduler Guild | DOCS-POLICY-20-004 | Document policy run modes + cursors in `/docs/policy/runs.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Platform Guild | DOCS-POLICY-20-005 | Produce `/docs/api/policy.md` with endpoint schemas + errors. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-POLICY-20-006 | Author `/docs/modules/cli/guides/policy.md` with commands, exit codes, JSON output. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-POLICY-20-007 | Create `/docs/ui/policy-editor.md` covering editor, simulation, approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-POLICY-20-008 | Publish `/docs/modules/policy/architecture.md` with sequence diagrams. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-POLICY-20-009 | Document metrics/traces/logs in `/docs/observability/policy.md`. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Security Guild | DOCS-POLICY-20-010 | Publish `/docs/security/policy-governance.md` for scopes + approvals. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-POLICY-20-011 | Add example policies under `/docs/examples/policies/` with commentary. |
|
||||
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Support Guild | DOCS-POLICY-20-012 | Draft `/docs/faq/policy-faq.md` covering conflicts, determinism, pitfalls. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-POLICY-20-001 | Add DSL lint + compile checks to CI pipelines. |
|
||||
| Sprint 20 | Policy Engine v2 | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild, QA Guild | DEVOPS-POLICY-20-003 | Add determinism CI job diffing repeated policy runs. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, Policy Guild | SAMPLES-POLICY-20-001 | Commit baseline/serverless/internal-only policy samples + fixtures. |
|
||||
| Sprint 20 | Policy Engine v2 | samples/TASKS.md | DONE (2025-10-26) | Samples Guild, UI Guild | SAMPLES-POLICY-20-002 | Produce simulation diff fixtures for UI/CLI tests. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-001 | Add new policy scopes (`policy:*`, `findings:read`, `effective:write`). |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Security Guild | AUTH-POLICY-20-002 | Enforce Policy Engine service identity and scope checks at gateway. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-POLICY-20-003 | Update Authority docs/config samples for policy scopes + workflows. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild, Policy Guild | BENCH-POLICY-20-001 | Create policy evaluation benchmark suite + baseline metrics. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild, Platform Guild | POLICY-ENGINE-20-000 | Spin up new Policy Engine service host with DI bootstrap and Authority wiring. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | DONE (2025-10-26) | Policy Guild | POLICY-ENGINE-20-001 | Deliver `stella-dsl@1` parser + IR compiler with diagnostics and checksums. |
|
||||
| Sprint 20 | Policy Engine v2 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-20-001 | Define policy run/diff DTOs + validation helpers. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-001 | Introduce graph scopes (`graph:*`) with configuration binding and defaults. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core Guild | AUTH-GRAPH-21-002 | Enforce graph scopes/identities at gateway with tenant propagation. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Authority/StellaOps.Authority/TASKS.md | DONE (2025-10-26) | Authority Core & Docs Guild | AUTH-GRAPH-21-003 | Update security docs/config samples for graph access and least privilege. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-001 | Define job DTOs for graph builds/overlay refresh (`GraphBuildJob`, `GraphOverlayJob`) with deterministic serialization and status enums; document in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-21-001-GRAPH-JOBS.md`. |
|
||||
| Sprint 21 | Graph Explorer v1 | src/Scheduler/__Libraries/StellaOps.Scheduler.Models/TASKS.md | DONE (2025-10-26) | Scheduler Models Guild | SCHED-MODELS-21-002 | Publish schema docs/sample payloads for graph job lifecycle. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-001 | Benchmark advisory observation ingest/correlation throughput. |
|
||||
| Sprint 22 | Link-Not-Merge v1 | src/Bench/StellaOps.Bench/TASKS.md | DONE (2025-10-26) | Bench Guild | BENCH-LNM-22-002 | Benchmark VEX ingest/correlation latency and event emission. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-001 | Publish `/docs/ui/console-overview.md` (IA, tenant model, filters, AOC alignment). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-002 | Author `/docs/ui/navigation.md` with route map, filters, keyboard shortcuts, deep links. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-003 | Document `/docs/ui/sbom-explorer.md` covering catalog, graph, overlays, exports. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-004 | Produce `/docs/ui/advisories-and-vex.md` detailing aggregation-not-merge UX. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-005 | Write `/docs/ui/findings.md` with filters, explain, exports, CLI parity notes. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-006 | Publish `/docs/ui/policies.md` (editor, simulation, approvals, RBAC). |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-007 | Document `/docs/ui/runs.md` with SSE monitoring, diff, retries, evidence downloads. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-26) | Docs Guild | DOCS-CONSOLE-23-008 | Draft `/docs/ui/admin.md` covering tenants, roles, tokens, integrations, fresh-auth. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild | DOCS-CONSOLE-23-009 | Publish `/docs/ui/downloads.md` aligning manifest with commands and offline flow. |
|
||||
| Sprint 23 | StellaOps Console | docs/TASKS.md | DONE (2025-10-27) | Docs Guild, Deployment Guild, Console Guild | DOCS-CONSOLE-23-010 | Write `/docs/deploy/console.md` (Helm, ingress, TLS, env vars, health checks). |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-001 | Provide graph build/overlay job APIs; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild | SCHED-WEB-21-002 | Provide overlay lag metrics endpoint/webhook; see `docs/SCHED-WEB-21-001-GRAPH-APIS.md`. |
|
||||
| Sprint 28 | Graph Explorer | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-26) | Scheduler WebService Guild, Authority Core Guild | SCHED-WEB-21-003 | Replace header auth with Authority scopes using `StellaOpsScopes`; dev fallback only when `Scheduler:Authority:Enabled=false`. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-001 | Deploy default OpenTelemetry collector manifests with secure OTLP pipeline. |
|
||||
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DONE (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-003 | Package telemetry stack configs for offline/air-gapped installs with signatures. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md | DONE (2025-10-27) | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. |
|
||||
| Sprint 16 | Scheduler Intelligence | src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md | DONE (2025-10-27) | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. |
|
||||
|
||||
@@ -1,181 +1,181 @@
|
||||
# Aggregation-Only Contract Reference
|
||||
|
||||
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
|
||||
|
||||
## 1. Purpose and Scope
|
||||
|
||||
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
|
||||
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
|
||||
- Complements the high-level architecture in [Concelier](../modules/concelier/architecture.md) and Authority enforcement documented in [Authority Architecture](../modules/authority/architecture.md).
|
||||
# Aggregation-Only Contract Reference
|
||||
|
||||
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
|
||||
|
||||
## 1. Purpose and Scope
|
||||
|
||||
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
|
||||
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
|
||||
- Complements the high-level architecture in [Concelier](../modules/concelier/architecture.md) and Authority enforcement documented in [Authority Architecture](../modules/authority/architecture.md).
|
||||
- Paired guidance: see the guard-rail checkpoints in [AOC Guardrails](../aoc/aoc-guardrails.md), the implementation reference in [AOC Guard Library](../aoc/guard-library.md), and CLI usage that will land in `/docs/modules/cli/guides/` as part of Sprint 19 follow-up.
|
||||
|
||||
## 2. Philosophy and Goals
|
||||
|
||||
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
|
||||
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
|
||||
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
|
||||
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
|
||||
|
||||
## 3. Contract Invariants
|
||||
|
||||
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|
||||
|---|-----------|-----------------------------|----------------------|
|
||||
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
|
||||
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
|
||||
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
|
||||
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
|
||||
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. | Linkset builders reviewed via fixtures and analyzers. |
|
||||
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
|
||||
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
|
||||
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
|
||||
|
||||
## 4. Raw Schemas
|
||||
|
||||
### 4.1 `advisory_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
|
||||
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
|
||||
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
|
||||
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
|
||||
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
|
||||
| `source.collector_version` | string | Semantic version of the collector. |
|
||||
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
|
||||
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
|
||||
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
|
||||
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
|
||||
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
|
||||
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
|
||||
| `content.spec_version` | string | Upstream spec version when known. |
|
||||
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
|
||||
| `identifiers` | object | Upstream identifiers (`cve`, `ghsa`, `aliases`, etc.) captured as provided (trimmed, order preserved, duplicates allowed). |
|
||||
| `linkset` | object | Join hints (see section 4.3). |
|
||||
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
|
||||
|
||||
### 4.2 `vex_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
|
||||
| `tenant` | string | Required; matches advisory collection requirements. |
|
||||
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
|
||||
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
|
||||
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
|
||||
| `content.raw` | object | Entire upstream VEX payload. |
|
||||
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
|
||||
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
|
||||
| `supersedes` | string or null | Same convention as advisory documents. |
|
||||
|
||||
### 4.3 Linkset Fields
|
||||
|
||||
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
|
||||
- `cpes`: Common Platform Enumerations when upstream docs provide them.
|
||||
- `aliases`: Any alternate advisory identifiers present in the payload.
|
||||
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
|
||||
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
|
||||
|
||||
Canonicalisation rules:
|
||||
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
|
||||
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
|
||||
- Connector mapping stages are responsible for the canonical form; ingestion trims whitespace but otherwise preserves the original order and duplicate entries so downstream policy can reason about upstream intent.
|
||||
|
||||
### 4.4 `advisory_observations`
|
||||
|
||||
`advisory_observations` is an immutable projection of the validated raw document used by Link‑Not‑Merge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
|
||||
| `tenant` | string | Lower-case tenant identifier. |
|
||||
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
|
||||
| `source.api` | string | Absolute URI the connector fetched from. |
|
||||
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
|
||||
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
|
||||
| `upstream.document_version` | string | Upstream revision/version string. |
|
||||
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
|
||||
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
|
||||
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
|
||||
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
|
||||
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
|
||||
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
|
||||
| `linkset.aliases` | array | Connector-supplied aliases (trimmed, order preserved, duplicates allowed). |
|
||||
| `linkset.purls` | array | Connector-supplied PURLs (ingestion preserves order and duplicates). |
|
||||
| `linkset.cpes` | array | Connector-supplied CPE URIs (trimmed, order preserved). |
|
||||
| `linkset.references` | array | `{ type, url }` pairs (trimmed; ingestion preserves order). |
|
||||
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
|
||||
| `attributes` | object | Optional provenance attributes keyed by connector. |
|
||||
|
||||
## 5. Error Model
|
||||
|
||||
| Code | Description | HTTP status | Surfaces |
|
||||
|------|-------------|-------------|----------|
|
||||
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
|
||||
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
|
||||
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
|
||||
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
|
||||
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
|
||||
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
|
||||
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
|
||||
|
||||
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
|
||||
|
||||
## 6. API and Tooling Interfaces
|
||||
|
||||
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
|
||||
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
|
||||
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
|
||||
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
|
||||
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
|
||||
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/modules/cli/guides/` alongside Sprint 19 CLI updates.
|
||||
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../modules/authority/architecture.md) for scope grammar.
|
||||
|
||||
## 7. Idempotency and Supersedes Rules
|
||||
|
||||
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
|
||||
2. If a document with the same hash already exists, skip the write and log a no-op.
|
||||
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
|
||||
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
|
||||
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
|
||||
|
||||
## 8. Migration Playbook
|
||||
|
||||
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
|
||||
2. Snapshot existing collections to `_backup_*` for rollback safety.
|
||||
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
|
||||
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
|
||||
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
|
||||
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
|
||||
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
|
||||
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
|
||||
|
||||
## 9. Observability and Diagnostics
|
||||
|
||||
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
|
||||
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
|
||||
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
|
||||
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
|
||||
|
||||
## 10. Security and Tenancy Checklist
|
||||
|
||||
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
|
||||
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
|
||||
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
|
||||
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
|
||||
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
|
||||
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
|
||||
|
||||
## 11. Compliance Checklist
|
||||
|
||||
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
|
||||
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
|
||||
- [ ] Authority scopes and tenant enforcement verified via integration tests.
|
||||
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
|
||||
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
|
||||
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
|
||||
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 19).*
|
||||
|
||||
## 2. Philosophy and Goals
|
||||
|
||||
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
|
||||
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
|
||||
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
|
||||
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
|
||||
|
||||
## 3. Contract Invariants
|
||||
|
||||
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|
||||
|---|-----------|-----------------------------|----------------------|
|
||||
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
|
||||
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
|
||||
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
|
||||
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
|
||||
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. Observations now persist both canonical linksets (for indexed queries) and raw linksets (preserving upstream order/duplicates) so downstream policy can decide how to normalise. | Linkset builders reviewed via fixtures/analyzers; raw-vs-canonical parity covered by observation fixtures. |
|
||||
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
|
||||
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
|
||||
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
|
||||
|
||||
## 4. Raw Schemas
|
||||
|
||||
### 4.1 `advisory_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
|
||||
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
|
||||
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
|
||||
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
|
||||
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
|
||||
| `source.collector_version` | string | Semantic version of the collector. |
|
||||
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
|
||||
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
|
||||
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
|
||||
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
|
||||
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
|
||||
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
|
||||
| `content.spec_version` | string | Upstream spec version when known. |
|
||||
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
|
||||
| `identifiers` | object | Upstream identifiers (`cve`, `ghsa`, `aliases`, etc.) captured as provided (trimmed, order preserved, duplicates allowed). |
|
||||
| `linkset` | object | Join hints (see section 4.3). |
|
||||
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
|
||||
|
||||
### 4.2 `vex_raw`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
|
||||
| `tenant` | string | Required; matches advisory collection requirements. |
|
||||
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
|
||||
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
|
||||
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
|
||||
| `content.raw` | object | Entire upstream VEX payload. |
|
||||
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
|
||||
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
|
||||
| `supersedes` | string or null | Same convention as advisory documents. |
|
||||
|
||||
### 4.3 Linkset Fields
|
||||
|
||||
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
|
||||
- `cpes`: Common Platform Enumerations when upstream docs provide them.
|
||||
- `aliases`: Any alternate advisory identifiers present in the payload.
|
||||
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
|
||||
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
|
||||
|
||||
Canonicalisation rules:
|
||||
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
|
||||
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
|
||||
- Connector mapping stages are responsible for the canonical form; ingestion trims whitespace but otherwise preserves the original order and duplicate entries so downstream policy can reason about upstream intent.
|
||||
|
||||
### 4.4 `advisory_observations`
|
||||
|
||||
`advisory_observations` is an immutable projection of the validated raw document used by Link‑Not‑Merge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
|
||||
| `tenant` | string | Lower-case tenant identifier. |
|
||||
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
|
||||
| `source.api` | string | Absolute URI the connector fetched from. |
|
||||
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
|
||||
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
|
||||
| `upstream.document_version` | string | Upstream revision/version string. |
|
||||
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
|
||||
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
|
||||
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
|
||||
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
|
||||
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
|
||||
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
|
||||
| `linkset.aliases` | array | Connector-supplied aliases (trimmed, order preserved, duplicates allowed). |
|
||||
| `linkset.purls` | array | Connector-supplied PURLs (ingestion preserves order and duplicates). |
|
||||
| `linkset.cpes` | array | Connector-supplied CPE URIs (trimmed, order preserved). |
|
||||
| `linkset.references` | array | `{ type, url }` pairs (trimmed; ingestion preserves order). |
|
||||
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
|
||||
| `attributes` | object | Optional provenance attributes keyed by connector. |
|
||||
|
||||
## 5. Error Model
|
||||
|
||||
| Code | Description | HTTP status | Surfaces |
|
||||
|------|-------------|-------------|----------|
|
||||
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
|
||||
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
|
||||
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
|
||||
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
|
||||
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
|
||||
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
|
||||
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
|
||||
|
||||
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
|
||||
|
||||
## 6. API and Tooling Interfaces
|
||||
|
||||
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
|
||||
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
|
||||
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
|
||||
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
|
||||
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
|
||||
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/modules/cli/guides/` alongside Sprint 19 CLI updates.
|
||||
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../modules/authority/architecture.md) for scope grammar.
|
||||
|
||||
## 7. Idempotency and Supersedes Rules
|
||||
|
||||
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
|
||||
2. If a document with the same hash already exists, skip the write and log a no-op.
|
||||
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
|
||||
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
|
||||
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
|
||||
|
||||
## 8. Migration Playbook
|
||||
|
||||
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
|
||||
2. Snapshot existing collections to `_backup_*` for rollback safety.
|
||||
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
|
||||
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
|
||||
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
|
||||
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
|
||||
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
|
||||
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
|
||||
|
||||
## 9. Observability and Diagnostics
|
||||
|
||||
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
|
||||
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
|
||||
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
|
||||
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
|
||||
|
||||
## 10. Security and Tenancy Checklist
|
||||
|
||||
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
|
||||
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
|
||||
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
|
||||
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
|
||||
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
|
||||
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
|
||||
|
||||
## 11. Compliance Checklist
|
||||
|
||||
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
|
||||
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
|
||||
- [ ] Authority scopes and tenant enforcement verified via integration tests.
|
||||
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
|
||||
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
|
||||
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
|
||||
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 19).*
|
||||
|
||||
860
docs/moat.md
860
docs/moat.md
@@ -1,430 +1,430 @@
|
||||
# StellaOps Moat Track — Spec Outline v0.3
|
||||
|
||||
**Scope of this doc:**
|
||||
(1) Deterministic Replayable Scans (SRM), (2) Policy Engine & Lattice UI, (3) Sovereign Readiness (CryptoProfile + RootPack), (4) Attestation Observability Graph (AOG), (5) Procurement‑Grade Trust Statement, (6) Third‑Party Proof Channel, (7) Zastava differential SBOM + AI scheduler.
|
||||
|
||||
Cross‑cutting principles: offline‑first, cryptographic determinism, evidence‑bound decisions, regional crypto compliance, minimal operational friction.
|
||||
|
||||
---
|
||||
|
||||
## 0) Shared Concepts (applies to all 7)
|
||||
|
||||
* **Artifact identity:** digest-first (OCI image digest, file sha256).
|
||||
* **Canonicalization:** all structured payloads (SBOM, SRM, Trust Statement JSON, VEX) are normalized via Canonical JSON (RFC‑8785‑like) prior to hashing/signing.
|
||||
* **Signatures:** DSSE envelopes; **dual‑signing** supported (e.g., FIPS ECDSA + GOST R 34.10; or ECDSA + SM2).
|
||||
* **Attestation chain:** each decision (scan, VEX merge, policy evaluation) yields a signed, replayable record.
|
||||
* **Profiles & Packs:** **CryptoProfile** (algorithm + root policy) and **RootPack** (trust anchors + OCSP/CRL/TSA mirrors) are versioned and importable.
|
||||
* **Policy Unit Tests:** any policy/lattice bundle ships with fixtures expected to pass during CI.
|
||||
|
||||
---
|
||||
|
||||
## 1) Deterministic Replayable Scans — SRM
|
||||
|
||||
### Objective
|
||||
|
||||
Make every scan a **provable, re‑executable fact**. Auditors can replay; results are bit‑for‑bit reproducible.
|
||||
|
||||
### Deliverables
|
||||
|
||||
* **SRM v0.1** schema (YAML/JSON)
|
||||
* Deterministic executor + `stella replay`
|
||||
* Replay diffing and result hashing
|
||||
|
||||
### SRM (Stella Replay Manifest) — schema (abridged)
|
||||
|
||||
```yaml
|
||||
apiVersion: srm.stellaops.dev/v0.1
|
||||
scan:
|
||||
id: uuid
|
||||
timestamp: ISO8601
|
||||
engine: { name: "stella-scan", version: "1.7.3", build_sha: "<sha>" }
|
||||
environment:
|
||||
os_image: <digest>
|
||||
kernel: { uname: "...", cgroups: "v2" }
|
||||
cpu_features: [avx2, sse4.2]
|
||||
inputs:
|
||||
image: { name: "reg/app:1.9.2", digest: "<sha>", layers: ["<sha>", ...] }
|
||||
sbom: { type: "cyclonedx@1.5", digest: "<sha>" }
|
||||
vex_set: [{ type: "openvex", digest: "<sha>" }]
|
||||
lattice_policy: { id: "corp-policy@2025-08-15", digest: "<sha>" }
|
||||
rules_and_feeds:
|
||||
rulesets: [{ name: "vuln-core", version: "2025.08.30", digest: "<sha>" }]
|
||||
feeds:
|
||||
- { name: "nvd", snapshot_date: "2025-08-30", archive_digest: "<sha>" }
|
||||
execution:
|
||||
mode: deterministic
|
||||
random_seed: 314159
|
||||
ordering: lexical
|
||||
heuristics: { binary_scan: true, secrets: false }
|
||||
evidence:
|
||||
files_hashed: 12873
|
||||
samples: [{ path: "/usr/lib/libssl.so.3", sha256: "<sha>" }]
|
||||
outputs:
|
||||
report_digest: "<sha>" # canonical JSON root hash
|
||||
artifacts:
|
||||
- { name: "findings.json", sha256: "<sha>" }
|
||||
signatures:
|
||||
- { scheme: "DSSE", CryptoProfile: "FIPS-140-3", signer: "build-ca@corp" }
|
||||
- { scheme: "DSSE", CryptoProfile: "GOST-2012", signer: "ru-ca@corp" }
|
||||
rekor: { entries: ["<uuid>", ...] } # optional (offline allowed)
|
||||
```
|
||||
|
||||
### CLI & API
|
||||
|
||||
* `stella scan --image reg/app@sha256:... --srm-out srm.yaml --findings findings.json`
|
||||
* `stella replay srm.yaml --out replay.json --assert-digest <sha>`
|
||||
* `POST /v1/srm/replay` → returns `ok`, `replay_report_digest`, `diff` (if any).
|
||||
|
||||
### Determinism Rules
|
||||
|
||||
* Single thread or ordered parallel with stable scheduling; sorted inputs; fixed random seed; pinned rules/feeds/policies from SRM; identical canonicalization routines.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Replaying SRM on a different host returns identical `report_digest`.
|
||||
* If any feed archive differs by 1 bit, replay fails with a precise diff.
|
||||
* SRM size ≤ 25 MB for a typical microservice image (excludes large feed archives, which may be referenced by digest and bundled as a side‑car tar).
|
||||
|
||||
---
|
||||
|
||||
## 2) Policy Engine & Lattice UI
|
||||
|
||||
### Objective
|
||||
|
||||
Turn VEX merging and severity logic into **programmable, testable algebra** with explainability.
|
||||
|
||||
### Model
|
||||
|
||||
* **Domain:** partial order over vulnerability states:
|
||||
`unknown < under_investigation < affected || not_affected < fixed`.
|
||||
Cross‑product with *scope*: `{runtime_path, build_path, optional_path}` and *confidence*: `{low, med, high}`.
|
||||
* **Merge semantics:** monotonic lattice joins; conflict resolution rules prioritized by signed source trust and policy precedence.
|
||||
|
||||
### DSL (sketch)
|
||||
|
||||
```hocon
|
||||
policy "corp-runtime" version "2025.08.15" {
|
||||
sources {
|
||||
trust_order = ["vendor:redhat", "internal:appsec", "public:nvd"]
|
||||
require_signatures = true
|
||||
}
|
||||
|
||||
rules {
|
||||
when vex.statement == "not_affected"
|
||||
and evidence.entrypoint_exposes == false
|
||||
then state := not_affected with confidence := high;
|
||||
|
||||
when package.is_dev_dependency == true
|
||||
then scope := optional_path;
|
||||
|
||||
when cvss >= 9.0 and reachable == true
|
||||
then priority := "block";
|
||||
}
|
||||
|
||||
guards {
|
||||
forbid unsigned_sources;
|
||||
forbid downgrade_of_state_below previous_state;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### UI (“Trust Algebra Studio”)
|
||||
|
||||
* Drag‑and‑drop rule blocks, precedence editor, **simulation mode** on sample SBOM/VEX; **policy unit tests**.
|
||||
* Export **signed** `.lattice.json`; importable into CI.
|
||||
|
||||
### CLI & API
|
||||
|
||||
* `stella policy lint corp-runtime.lattice.json`
|
||||
* `stella policy test --fixtures fixtures/`
|
||||
* `POST /v1/policy/evaluate` → normalized decision + proof trail.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Given same inputs, policy evaluation yields identical decision + proof trail hash.
|
||||
* UI can round‑trip DSL ⇄ JSON with no semantic drift.
|
||||
* Policy pack signature required to run in “enforced” mode.
|
||||
|
||||
---
|
||||
|
||||
## 3) Sovereign Readiness — CryptoProfile + RootPack
|
||||
|
||||
### Objective
|
||||
|
||||
**Drop‑in regional cryptography** (Russia/China/EU/US) with offline operation.
|
||||
|
||||
### CryptoProfile (attached to every signature/attestation)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "GOST-2012@v1",
|
||||
"algorithms": {"sign":"GOST R 34.10-2012","hash":"GOST R 34.11-2012","cipher":"GOST 34.12-2015"},
|
||||
"key_policy": {"curve":"id-tc26-gost-3410-2012-256","hsm_required": true},
|
||||
"time_stamping": {"tsa": "rootpack://ru/tsa1"},
|
||||
"roots": ["rootpack://ru/trustanchors/gost-ca1"]
|
||||
}
|
||||
```
|
||||
|
||||
### RootPack
|
||||
|
||||
* Tarball containing: trust anchors, intermediate CAs, OCSP/CRL snapshots, TSA profiles, policy constraints (e.g., no cross‑sign with foreign roots), and region metadata.
|
||||
* Installed via: `stella rootpack import rootpack_ru_v1.tar.gz`.
|
||||
|
||||
### Dual‑Signing & Guardrails
|
||||
|
||||
* Policy flags like `allow_dual_signing FIPS+GOST`, `forbid_sm2_for_us_exports` (example only).
|
||||
* Enforcement happens at signing time and verification time.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Offline verification using RootPack succeeds; online OCSP disabled per policy.
|
||||
* Mis‑profiled signatures are rejected with explicit reason codes.
|
||||
* Dual‑signed DSSE verifies under both profiles when allowed.
|
||||
|
||||
---
|
||||
|
||||
## 4) Attestation Observability Graph (AOG)
|
||||
|
||||
### Objective
|
||||
|
||||
Make trust **observable**. Expose SLIs/SLOs for cryptographic posture & policy compliance.
|
||||
|
||||
### Data Model
|
||||
|
||||
* Nodes: `{artifact, sbom, policy, vex, srm, signature, rootpack, runtime_instance}`
|
||||
* Edges: `derived_from`, `signed_by`, `evaluated_by`, `replayed_by`, `runs_as`
|
||||
|
||||
### Metrics (OpenTelemetry/Prometheus)
|
||||
|
||||
* `stella_trust_sli{service,env}` = fraction of running pods whose image has a valid SRM‑backed attestation chain under the active policy.
|
||||
* `stella_attestation_latency_seconds` (P50/P95) from build to verified‑ready.
|
||||
* `stella_policy_drift_events_total` (increment when running policy != signed policy).
|
||||
* `stella_exception_without_proof_total` (must be 0).
|
||||
* `stella_replay_success_ratio` (per week).
|
||||
|
||||
**SLO Example**
|
||||
|
||||
* **Trust SLO ≥ 99.9%** measured hourly; error budget resets monthly.
|
||||
|
||||
### Interfaces
|
||||
|
||||
* `stella aog export --format otlp`
|
||||
* `GET /v1/aog/graph?artifactDigest=...` → subgraph JSON (with signed edge proofs).
|
||||
* Grafana dashboards (packaged).
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* AOG can reconstruct the full trust lineage for any running pod in ≤ 2s (p95) on a 1k‑service cluster.
|
||||
* Metrics cardinality bounded (service/env/policy only).
|
||||
|
||||
---
|
||||
|
||||
## 5) Procurement‑Grade “Trust Statement”
|
||||
|
||||
### Objective
|
||||
|
||||
One **board‑ready** artifact that unifies security posture across vendors; machine‑readable twin for ERP/GRC.
|
||||
|
||||
### Outputs
|
||||
|
||||
* **PDF** (human): signed, watermark, controlled fields for vendor name/version/date, summary graphs, SLOs, exceptions with PCE (proof‑carrying exceptions).
|
||||
* **JSON** (machine): normalized schema below; DSSE‑signed; includes SRM and policy references.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "trust-statement.stellaops.dev/v1",
|
||||
"vendor": {"name": "Acme","product":"Payments API","version":"1.9.2"},
|
||||
"build": {"image_digest":"sha256:...","srm_digest":"sha256:..."},
|
||||
"policy": {"id":"corp-runtime@2025-08-15","digest":"sha256:..."},
|
||||
"summary": {
|
||||
"trust_sli": 0.9992,
|
||||
"exceptions": 1,
|
||||
"open_findings": {"critical":0,"high":2,"medium":4,"low":12}
|
||||
},
|
||||
"exceptions": [{
|
||||
"id":"EXC-2025-0912",
|
||||
"reason":"not_affected-via-vex",
|
||||
"proof_digest":"sha256:...",
|
||||
"expiry":"2026-01-15"
|
||||
}],
|
||||
"signatures": [{ "CryptoProfile":"FIPS-140-3" }]
|
||||
}
|
||||
```
|
||||
|
||||
### Integrations
|
||||
|
||||
* Push connectors: **SAP Ariba, ServiceNow, Archer, Jira** (webhooks or SFTP in offline flows).
|
||||
* CLI: `stella trust-statement generate --srm srm.yaml --policy corp-runtime.lattice.json --out acme-1.9.2.trust.json --pdf`.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* JSON validates against schema; PDF and JSON hashes match the DSSE statement.
|
||||
* ERP ingest POC: Ariba/ServiceNow field mapping validated.
|
||||
|
||||
---
|
||||
|
||||
## 6) Third‑Party Proof Channel
|
||||
|
||||
### Objective
|
||||
|
||||
Create a **publisher ecosystem** for upstream proofs: SBOM, VEX, and **VDR (Vulnerability Derivation Reason)**.
|
||||
|
||||
### Publisher Model
|
||||
|
||||
* **Identity:** publishers obtain a **Publisher Certificate** (could be verified via RootPack‑anchored CA or cross‑signed).
|
||||
* **Submission:** `stella ledger publish --type {sbom|vex|vdr} --artifact <digest> --file proof.json --sign`.
|
||||
* **Moderation & Revocation:** CRL‑like **Proof Revocation List (PRL)** with signed reasons.
|
||||
|
||||
### VDR (schema sketch)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema":"vdr.stellaops.dev/v1",
|
||||
"artifact":"sha256:...",
|
||||
"cve":"CVE-2025-12345",
|
||||
"claim":"not_affected",
|
||||
"method":"entrypoint_unreachable|abi_mismatch|dead_code",
|
||||
"evidence_refs":[{"type":"symbol_map","digest":"sha256:..."}],
|
||||
"publisher":"redhat://rhel",
|
||||
"signatures":[...]
|
||||
}
|
||||
```
|
||||
|
||||
### Consumption
|
||||
|
||||
* Policies can **prioritize** publisher channels by trust level.
|
||||
* AOG shows which proofs originated from which publishers.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* At least one upstream (e.g., base image vendor) can publish and your policy can consume & rank it.
|
||||
* PRL revokes a proof and AOG reflects the change within 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 7) Zastava — differential SBOM + AI enrichment scheduler
|
||||
|
||||
### Objective
|
||||
|
||||
Produce **entrypoint‑aware differential SBOMs** and continually **re‑enrich** new/old SBOMs with AI context and exposure‑aware prioritization.
|
||||
|
||||
### Concepts
|
||||
|
||||
* **dSBOM:** SBOM that reflects effective dependency set for a specific `ENTRYPOINT/CMD` and runtime flags (e.g., `--server.urls`, `DOTNET_...`).
|
||||
* **Scheduler:** rescans **old SBOMs** when: (a) new CVE feeds arrive, (b) new VEX/VDR appear, (c) policy changes, or (d) AI models learn new exploitability signals.
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. **Static slice:** infer reachable packages from entrypoint (e.g., `.NET Kestrel` vs CLI tool).
|
||||
2. **Runtime slice (optional):** collect process tree, open sockets, and imported modules at startup in a **shadow run** or mirrored traffic.
|
||||
3. **Diff:** `dSBOM = SBOM ∩ (static_reachable ∪ runtime_observed)`.
|
||||
4. **AI Enrichment:** Zastava annotates each finding with *context weights* (exposed/not exposed, network scope, RBAC, secrets proximity).
|
||||
5. **Plan:** produce PRs (Dockerfile base bump, package pin, k8s Service change).
|
||||
6. **Scheduler:**
|
||||
|
||||
* `stella zastava schedule --query 'service=payments AND env=prod' --interval 6h`
|
||||
* Triggers re‑evaluation and emits updated SRM + Trust deltas.
|
||||
|
||||
### dSBOM format (addon)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema":"cyclonedx+stella-diff@1.0",
|
||||
"base_sbom":"sha256:...",
|
||||
"entrypoint": ["/app/bin/Release/net8.0/app.dll"],
|
||||
"cmd": ["--urls","http://127.0.0.1:8080"],
|
||||
"static_reachable": ["pkg:nuget/Kestrel@*", "pkg:nuget/System.Data@*"],
|
||||
"runtime_observed": ["pkg:rpm/openssl@3.0.9"],
|
||||
"excluded_paths": ["/usr/share/docs/**"],
|
||||
"digest":"sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
### Kestrel example (priority logic)
|
||||
|
||||
* If Kestrel present but Service is `ClusterIP` and no Ingress, **deprioritize**; if `LoadBalancer`/`NodePort` with 0‑RTT TLS disabled, **prioritize**.
|
||||
* Zastava produces an **explainable card**: “Priority lowered due to non‑exposed runtime path; evidence: `kubectl get svc`, `netstat`, policy rule #42.”
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Changing `ENTRYPOINT` produces a different, signed dSBOM and updated priorities.
|
||||
* Scheduler updates stale SBOMs and issues signed deltas without network access (when RootPacks + feed mirrors are available).
|
||||
|
||||
---
|
||||
|
||||
## Cross‑Cutting Security & Compliance
|
||||
|
||||
* **Sanctions & Guardrails:** per‑profile constraints (e.g., forbid dual‑signing across certain jurisdictions). Policy‑enforced at sign/verify time.
|
||||
* **HSM Integrations:** PKCS#11 providers for each profile (FIPS, GOST, SM2).
|
||||
* **Data handling:** SRMs and Trust Statements may contain paths and hashes; optional redaction profiles for vendor sharing.
|
||||
|
||||
---
|
||||
|
||||
## Interfaces Summary (CLI)
|
||||
|
||||
```bash
|
||||
# Scans / Replay
|
||||
stella scan --image <digest> --srm-out srm.yaml
|
||||
stella replay srm.yaml --assert-digest <sha>
|
||||
|
||||
# Policy & Lattice
|
||||
stella policy lint corp.lattice.json
|
||||
stella policy test --fixtures fixtures/
|
||||
stella policy sign --profile FIPS-140-3
|
||||
|
||||
# Crypto Sovereign
|
||||
stella rootpack import rootpack_ru_v1.tar.gz
|
||||
stella attest sign --profile GOST-2012 --in srm.yaml
|
||||
stella attest verify --profiles GOST-2012,FIPS-140-3
|
||||
|
||||
# AOG
|
||||
stella aog export --format otlp
|
||||
stella aog graph --artifact <digest>
|
||||
|
||||
# Trust Statement
|
||||
stella trust-statement generate --srm srm.yaml --policy corp.lattice.json --pdf out.pdf --json out.json
|
||||
|
||||
# Third-Party Proof Channel
|
||||
stella ledger publish --type vdr --artifact <digest> --file vdr.json --sign
|
||||
stella ledger revoke --id <proof-id> --reason "superseded"
|
||||
|
||||
# Zastava
|
||||
stella zastava diff-sbom --image <digest> --entrypoint "<cmd>" --out dsbom.json
|
||||
stella zastava enrich --sbom dsbom.json --findings findings.json
|
||||
stella zastava schedule --query 'env=prod' --interval 6h
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics (per pillar)
|
||||
|
||||
* **SRM:** ≥ 99% of production images have SRM attached; **Replay Success Ratio** ≥ 0.99 weekly.
|
||||
* **Policy/Lattice:** 100% of exceptions carry proof; policy test coverage ≥ 90% on top CVE classes.
|
||||
* **Sovereign:** RootPacks for RU/CN/EU/US verified; dual‑signing working in CI; offline verification median < 200 ms.
|
||||
* **AOG:** Trust SLO ≥ 99.9%; lineage lookup p95 ≤ 2s.
|
||||
* **Trust Statement:** Accepted by at least one ERP (pilot); generation time ≤ 15s.
|
||||
* **Third‑Party Channel:** ≥ 3 upstream publishers integrated; PRL revocations flow to AOG within 5 minutes.
|
||||
* **Zastava:** dSBOM reduces non‑reachable high/critical findings by ≥ 35% without raising exposure incidents.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
* **Crypto complexity:** profile misuse → strict guardrails + default safe profiles, strong linting.
|
||||
* **Cardinality/telemetry blow‑ups:** AOG label hygiene + sampling.
|
||||
* **Vendor adoption (Proof Channel):** seed with your own base images and internal frameworks; provide SDKs and reference publishers.
|
||||
* **Determinism regressions:** CI runs replay tests on golden SRMs; any drift fails the build.
|
||||
|
||||
---
|
||||
|
||||
## 90‑Day Moat‑First Milestones
|
||||
|
||||
1. **SRM v0.1**: schema, deterministic executor, CLI replay, golden tests.
|
||||
2. **Policy Engine MVP**: DSL + evaluator + UI simulation; policy unit tests; signed policy packs.
|
||||
3. **CryptoProfile/RootPack MVP**: FIPS + GOST working; dual‑signing; offline verify.
|
||||
4. **AOG MVP**: lineage service + OTLP exporter + Grafana pack; Trust SLI.
|
||||
5. **Trust Statement MVP**: JSON + PDF; ServiceNow ingest POC.
|
||||
6. **Proof Channel alpha**: publisher identity + VDR schema + local ledger; PRL.
|
||||
7. **Zastava α**: `diff-sbom` + exposure heuristics for `.NET Kestrel`; scheduler with offline mirrors.
|
||||
|
||||
---
|
||||
# StellaOps Moat Track — Spec Outline v0.3
|
||||
|
||||
**Scope of this doc:**
|
||||
(1) Deterministic Replayable Scans (SRM), (2) Policy Engine & Lattice UI, (3) Sovereign Readiness (CryptoProfile + RootPack), (4) Attestation Observability Graph (AOG), (5) Procurement‑Grade Trust Statement, (6) Third‑Party Proof Channel, (7) Zastava differential SBOM + AI scheduler.
|
||||
|
||||
Cross‑cutting principles: offline‑first, cryptographic determinism, evidence‑bound decisions, regional crypto compliance, minimal operational friction.
|
||||
|
||||
---
|
||||
|
||||
## 0) Shared Concepts (applies to all 7)
|
||||
|
||||
* **Artifact identity:** digest-first (OCI image digest, file sha256).
|
||||
* **Canonicalization:** all structured payloads (SBOM, SRM, Trust Statement JSON, VEX) are normalized via Canonical JSON (RFC‑8785‑like) prior to hashing/signing.
|
||||
* **Signatures:** DSSE envelopes; **dual‑signing** supported (e.g., FIPS ECDSA + GOST R 34.10; or ECDSA + SM2).
|
||||
* **Attestation chain:** each decision (scan, VEX merge, policy evaluation) yields a signed, replayable record.
|
||||
* **Profiles & Packs:** **CryptoProfile** (algorithm + root policy) and **RootPack** (trust anchors + OCSP/CRL/TSA mirrors) are versioned and importable.
|
||||
* **Policy Unit Tests:** any policy/lattice bundle ships with fixtures expected to pass during CI.
|
||||
|
||||
---
|
||||
|
||||
## 1) Deterministic Replayable Scans — SRM
|
||||
|
||||
### Objective
|
||||
|
||||
Make every scan a **provable, re‑executable fact**. Auditors can replay; results are bit‑for‑bit reproducible.
|
||||
|
||||
### Deliverables
|
||||
|
||||
* **SRM v0.1** schema (YAML/JSON)
|
||||
* Deterministic executor + `stella replay`
|
||||
* Replay diffing and result hashing
|
||||
|
||||
### SRM (Stella Replay Manifest) — schema (abridged)
|
||||
|
||||
```yaml
|
||||
apiVersion: srm.stellaops.dev/v0.1
|
||||
scan:
|
||||
id: uuid
|
||||
timestamp: ISO8601
|
||||
engine: { name: "stella-scan", version: "1.7.3", build_sha: "<sha>" }
|
||||
environment:
|
||||
os_image: <digest>
|
||||
kernel: { uname: "...", cgroups: "v2" }
|
||||
cpu_features: [avx2, sse4.2]
|
||||
inputs:
|
||||
image: { name: "reg/app:1.9.2", digest: "<sha>", layers: ["<sha>", ...] }
|
||||
sbom: { type: "cyclonedx@1.5", digest: "<sha>" }
|
||||
vex_set: [{ type: "openvex", digest: "<sha>" }]
|
||||
lattice_policy: { id: "corp-policy@2025-08-15", digest: "<sha>" }
|
||||
rules_and_feeds:
|
||||
rulesets: [{ name: "vuln-core", version: "2025.08.30", digest: "<sha>" }]
|
||||
feeds:
|
||||
- { name: "nvd", snapshot_date: "2025-08-30", archive_digest: "<sha>" }
|
||||
execution:
|
||||
mode: deterministic
|
||||
random_seed: 314159
|
||||
ordering: lexical
|
||||
heuristics: { binary_scan: true, secrets: false }
|
||||
evidence:
|
||||
files_hashed: 12873
|
||||
samples: [{ path: "/usr/lib/libssl.so.3", sha256: "<sha>" }]
|
||||
outputs:
|
||||
report_digest: "<sha>" # canonical JSON root hash
|
||||
artifacts:
|
||||
- { name: "findings.json", sha256: "<sha>" }
|
||||
signatures:
|
||||
- { scheme: "DSSE", CryptoProfile: "FIPS-140-3", signer: "build-ca@corp" }
|
||||
- { scheme: "DSSE", CryptoProfile: "GOST-2012", signer: "ru-ca@corp" }
|
||||
rekor: { entries: ["<uuid>", ...] } # optional (offline allowed)
|
||||
```
|
||||
|
||||
### CLI & API
|
||||
|
||||
* `stella scan --image reg/app@sha256:... --srm-out srm.yaml --findings findings.json`
|
||||
* `stella replay srm.yaml --out replay.json --assert-digest <sha>`
|
||||
* `POST /v1/srm/replay` → returns `ok`, `replay_report_digest`, `diff` (if any).
|
||||
|
||||
### Determinism Rules
|
||||
|
||||
* Single thread or ordered parallel with stable scheduling; sorted inputs; fixed random seed; pinned rules/feeds/policies from SRM; identical canonicalization routines.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Replaying SRM on a different host returns identical `report_digest`.
|
||||
* If any feed archive differs by 1 bit, replay fails with a precise diff.
|
||||
* SRM size ≤ 25 MB for a typical microservice image (excludes large feed archives, which may be referenced by digest and bundled as a side‑car tar).
|
||||
|
||||
---
|
||||
|
||||
## 2) Policy Engine & Lattice UI
|
||||
|
||||
### Objective
|
||||
|
||||
Turn VEX merging and severity logic into **programmable, testable algebra** with explainability.
|
||||
|
||||
### Model
|
||||
|
||||
* **Domain:** partial order over vulnerability states:
|
||||
`unknown < under_investigation < affected || not_affected < fixed`.
|
||||
Cross‑product with *scope*: `{runtime_path, build_path, optional_path}` and *confidence*: `{low, med, high}`.
|
||||
* **Merge semantics:** monotonic lattice joins; conflict resolution rules prioritized by signed source trust and policy precedence.
|
||||
|
||||
### DSL (sketch)
|
||||
|
||||
```hocon
|
||||
policy "corp-runtime" version "2025.08.15" {
|
||||
sources {
|
||||
trust_order = ["vendor:redhat", "internal:appsec", "public:nvd"]
|
||||
require_signatures = true
|
||||
}
|
||||
|
||||
rules {
|
||||
when vex.statement == "not_affected"
|
||||
and evidence.entrypoint_exposes == false
|
||||
then state := not_affected with confidence := high;
|
||||
|
||||
when package.is_dev_dependency == true
|
||||
then scope := optional_path;
|
||||
|
||||
when cvss >= 9.0 and reachable == true
|
||||
then priority := "block";
|
||||
}
|
||||
|
||||
guards {
|
||||
forbid unsigned_sources;
|
||||
forbid downgrade_of_state_below previous_state;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### UI (“Trust Algebra Studio”)
|
||||
|
||||
* Drag‑and‑drop rule blocks, precedence editor, **simulation mode** on sample SBOM/VEX; **policy unit tests**.
|
||||
* Export **signed** `.lattice.json`; importable into CI.
|
||||
|
||||
### CLI & API
|
||||
|
||||
* `stella policy lint corp-runtime.lattice.json`
|
||||
* `stella policy test --fixtures fixtures/`
|
||||
* `POST /v1/policy/evaluate` → normalized decision + proof trail.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Given same inputs, policy evaluation yields identical decision + proof trail hash.
|
||||
* UI can round‑trip DSL ⇄ JSON with no semantic drift.
|
||||
* Policy pack signature required to run in “enforced” mode.
|
||||
|
||||
---
|
||||
|
||||
## 3) Sovereign Readiness — CryptoProfile + RootPack
|
||||
|
||||
### Objective
|
||||
|
||||
**Drop‑in regional cryptography** (Russia/China/EU/US) with offline operation.
|
||||
|
||||
### CryptoProfile (attached to every signature/attestation)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "GOST-2012@v1",
|
||||
"algorithms": {"sign":"GOST R 34.10-2012","hash":"GOST R 34.11-2012","cipher":"GOST 34.12-2015"},
|
||||
"key_policy": {"curve":"id-tc26-gost-3410-2012-256","hsm_required": true},
|
||||
"time_stamping": {"tsa": "rootpack://ru/tsa1"},
|
||||
"roots": ["rootpack://ru/trustanchors/gost-ca1"]
|
||||
}
|
||||
```
|
||||
|
||||
### RootPack
|
||||
|
||||
* Tarball containing: trust anchors, intermediate CAs, OCSP/CRL snapshots, TSA profiles, policy constraints (e.g., no cross‑sign with foreign roots), and region metadata.
|
||||
* Installed via: `stella rootpack import rootpack_ru_v1.tar.gz`.
|
||||
|
||||
### Dual‑Signing & Guardrails
|
||||
|
||||
* Policy flags like `allow_dual_signing FIPS+GOST`, `forbid_sm2_for_us_exports` (example only).
|
||||
* Enforcement happens at signing time and verification time.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Offline verification using RootPack succeeds; online OCSP disabled per policy.
|
||||
* Mis‑profiled signatures are rejected with explicit reason codes.
|
||||
* Dual‑signed DSSE verifies under both profiles when allowed.
|
||||
|
||||
---
|
||||
|
||||
## 4) Attestation Observability Graph (AOG)
|
||||
|
||||
### Objective
|
||||
|
||||
Make trust **observable**. Expose SLIs/SLOs for cryptographic posture & policy compliance.
|
||||
|
||||
### Data Model
|
||||
|
||||
* Nodes: `{artifact, sbom, policy, vex, srm, signature, rootpack, runtime_instance}`
|
||||
* Edges: `derived_from`, `signed_by`, `evaluated_by`, `replayed_by`, `runs_as`
|
||||
|
||||
### Metrics (OpenTelemetry/Prometheus)
|
||||
|
||||
* `stella_trust_sli{service,env}` = fraction of running pods whose image has a valid SRM‑backed attestation chain under the active policy.
|
||||
* `stella_attestation_latency_seconds` (P50/P95) from build to verified‑ready.
|
||||
* `stella_policy_drift_events_total` (increment when running policy != signed policy).
|
||||
* `stella_exception_without_proof_total` (must be 0).
|
||||
* `stella_replay_success_ratio` (per week).
|
||||
|
||||
**SLO Example**
|
||||
|
||||
* **Trust SLO ≥ 99.9%** measured hourly; error budget resets monthly.
|
||||
|
||||
### Interfaces
|
||||
|
||||
* `stella aog export --format otlp`
|
||||
* `GET /v1/aog/graph?artifactDigest=...` → subgraph JSON (with signed edge proofs).
|
||||
* Grafana dashboards (packaged).
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* AOG can reconstruct the full trust lineage for any running pod in ≤ 2s (p95) on a 1k‑service cluster.
|
||||
* Metrics cardinality bounded (service/env/policy only).
|
||||
|
||||
---
|
||||
|
||||
## 5) Procurement‑Grade “Trust Statement”
|
||||
|
||||
### Objective
|
||||
|
||||
One **board‑ready** artifact that unifies security posture across vendors; machine‑readable twin for ERP/GRC.
|
||||
|
||||
### Outputs
|
||||
|
||||
* **PDF** (human): signed, watermark, controlled fields for vendor name/version/date, summary graphs, SLOs, exceptions with PCE (proof‑carrying exceptions).
|
||||
* **JSON** (machine): normalized schema below; DSSE‑signed; includes SRM and policy references.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "trust-statement.stellaops.dev/v1",
|
||||
"vendor": {"name": "Acme","product":"Payments API","version":"1.9.2"},
|
||||
"build": {"image_digest":"sha256:...","srm_digest":"sha256:..."},
|
||||
"policy": {"id":"corp-runtime@2025-08-15","digest":"sha256:..."},
|
||||
"summary": {
|
||||
"trust_sli": 0.9992,
|
||||
"exceptions": 1,
|
||||
"open_findings": {"critical":0,"high":2,"medium":4,"low":12}
|
||||
},
|
||||
"exceptions": [{
|
||||
"id":"EXC-2025-0912",
|
||||
"reason":"not_affected-via-vex",
|
||||
"proof_digest":"sha256:...",
|
||||
"expiry":"2026-01-15"
|
||||
}],
|
||||
"signatures": [{ "CryptoProfile":"FIPS-140-3" }]
|
||||
}
|
||||
```
|
||||
|
||||
### Integrations
|
||||
|
||||
* Push connectors: **SAP Ariba, ServiceNow, Archer, Jira** (webhooks or SFTP in offline flows).
|
||||
* CLI: `stella trust-statement generate --srm srm.yaml --policy corp-runtime.lattice.json --out acme-1.9.2.trust.json --pdf`.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* JSON validates against schema; PDF and JSON hashes match the DSSE statement.
|
||||
* ERP ingest POC: Ariba/ServiceNow field mapping validated.
|
||||
|
||||
---
|
||||
|
||||
## 6) Third‑Party Proof Channel
|
||||
|
||||
### Objective
|
||||
|
||||
Create a **publisher ecosystem** for upstream proofs: SBOM, VEX, and **VDR (Vulnerability Derivation Reason)**.
|
||||
|
||||
### Publisher Model
|
||||
|
||||
* **Identity:** publishers obtain a **Publisher Certificate** (could be verified via RootPack‑anchored CA or cross‑signed).
|
||||
* **Submission:** `stella ledger publish --type {sbom|vex|vdr} --artifact <digest> --file proof.json --sign`.
|
||||
* **Moderation & Revocation:** CRL‑like **Proof Revocation List (PRL)** with signed reasons.
|
||||
|
||||
### VDR (schema sketch)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema":"vdr.stellaops.dev/v1",
|
||||
"artifact":"sha256:...",
|
||||
"cve":"CVE-2025-12345",
|
||||
"claim":"not_affected",
|
||||
"method":"entrypoint_unreachable|abi_mismatch|dead_code",
|
||||
"evidence_refs":[{"type":"symbol_map","digest":"sha256:..."}],
|
||||
"publisher":"redhat://rhel",
|
||||
"signatures":[...]
|
||||
}
|
||||
```
|
||||
|
||||
### Consumption
|
||||
|
||||
* Policies can **prioritize** publisher channels by trust level.
|
||||
* AOG shows which proofs originated from which publishers.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* At least one upstream (e.g., base image vendor) can publish and your policy can consume & rank it.
|
||||
* PRL revokes a proof and AOG reflects the change within 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 7) Zastava — differential SBOM + AI enrichment scheduler
|
||||
|
||||
### Objective
|
||||
|
||||
Produce **entrypoint‑aware differential SBOMs** and continually **re‑enrich** new/old SBOMs with AI context and exposure‑aware prioritization.
|
||||
|
||||
### Concepts
|
||||
|
||||
* **dSBOM:** SBOM that reflects effective dependency set for a specific `ENTRYPOINT/CMD` and runtime flags (e.g., `--server.urls`, `DOTNET_...`).
|
||||
* **Scheduler:** rescans **old SBOMs** when: (a) new CVE feeds arrive, (b) new VEX/VDR appear, (c) policy changes, or (d) AI models learn new exploitability signals.
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. **Static slice:** infer reachable packages from entrypoint (e.g., `.NET Kestrel` vs CLI tool).
|
||||
2. **Runtime slice (optional):** collect process tree, open sockets, and imported modules at startup in a **shadow run** or mirrored traffic.
|
||||
3. **Diff:** `dSBOM = SBOM ∩ (static_reachable ∪ runtime_observed)`.
|
||||
4. **AI Enrichment:** Zastava annotates each finding with *context weights* (exposed/not exposed, network scope, RBAC, secrets proximity).
|
||||
5. **Plan:** produce PRs (Dockerfile base bump, package pin, k8s Service change).
|
||||
6. **Scheduler:**
|
||||
|
||||
* `stella zastava schedule --query 'service=payments AND env=prod' --interval 6h`
|
||||
* Triggers re‑evaluation and emits updated SRM + Trust deltas.
|
||||
|
||||
### dSBOM format (addon)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema":"cyclonedx+stella-diff@1.0",
|
||||
"base_sbom":"sha256:...",
|
||||
"entrypoint": ["/app/bin/Release/net8.0/app.dll"],
|
||||
"cmd": ["--urls","http://127.0.0.1:8080"],
|
||||
"static_reachable": ["pkg:nuget/Kestrel@*", "pkg:nuget/System.Data@*"],
|
||||
"runtime_observed": ["pkg:rpm/openssl@3.0.9"],
|
||||
"excluded_paths": ["/usr/share/docs/**"],
|
||||
"digest":"sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
### Kestrel example (priority logic)
|
||||
|
||||
* If Kestrel present but Service is `ClusterIP` and no Ingress, **deprioritize**; if `LoadBalancer`/`NodePort` with 0‑RTT TLS disabled, **prioritize**.
|
||||
* Zastava produces an **explainable card**: “Priority lowered due to non‑exposed runtime path; evidence: `kubectl get svc`, `netstat`, policy rule #42.”
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
* Changing `ENTRYPOINT` produces a different, signed dSBOM and updated priorities.
|
||||
* Scheduler updates stale SBOMs and issues signed deltas without network access (when RootPacks + feed mirrors are available).
|
||||
|
||||
---
|
||||
|
||||
## Cross‑Cutting Security & Compliance
|
||||
|
||||
* **Sanctions & Guardrails:** per‑profile constraints (e.g., forbid dual‑signing across certain jurisdictions). Policy‑enforced at sign/verify time.
|
||||
* **HSM Integrations:** PKCS#11 providers for each profile (FIPS, GOST, SM2).
|
||||
* **Data handling:** SRMs and Trust Statements may contain paths and hashes; optional redaction profiles for vendor sharing.
|
||||
|
||||
---
|
||||
|
||||
## Interfaces Summary (CLI)
|
||||
|
||||
```bash
|
||||
# Scans / Replay
|
||||
stella scan --image <digest> --srm-out srm.yaml
|
||||
stella replay srm.yaml --assert-digest <sha>
|
||||
|
||||
# Policy & Lattice
|
||||
stella policy lint corp.lattice.json
|
||||
stella policy test --fixtures fixtures/
|
||||
stella policy sign --profile FIPS-140-3
|
||||
|
||||
# Crypto Sovereign
|
||||
stella rootpack import rootpack_ru_v1.tar.gz
|
||||
stella attest sign --profile GOST-2012 --in srm.yaml
|
||||
stella attest verify --profiles GOST-2012,FIPS-140-3
|
||||
|
||||
# AOG
|
||||
stella aog export --format otlp
|
||||
stella aog graph --artifact <digest>
|
||||
|
||||
# Trust Statement
|
||||
stella trust-statement generate --srm srm.yaml --policy corp.lattice.json --pdf out.pdf --json out.json
|
||||
|
||||
# Third-Party Proof Channel
|
||||
stella ledger publish --type vdr --artifact <digest> --file vdr.json --sign
|
||||
stella ledger revoke --id <proof-id> --reason "superseded"
|
||||
|
||||
# Zastava
|
||||
stella zastava diff-sbom --image <digest> --entrypoint "<cmd>" --out dsbom.json
|
||||
stella zastava enrich --sbom dsbom.json --findings findings.json
|
||||
stella zastava schedule --query 'env=prod' --interval 6h
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics (per pillar)
|
||||
|
||||
* **SRM:** ≥ 99% of production images have SRM attached; **Replay Success Ratio** ≥ 0.99 weekly.
|
||||
* **Policy/Lattice:** 100% of exceptions carry proof; policy test coverage ≥ 90% on top CVE classes.
|
||||
* **Sovereign:** RootPacks for RU/CN/EU/US verified; dual‑signing working in CI; offline verification median < 200 ms.
|
||||
* **AOG:** Trust SLO ≥ 99.9%; lineage lookup p95 ≤ 2s.
|
||||
* **Trust Statement:** Accepted by at least one ERP (pilot); generation time ≤ 15s.
|
||||
* **Third‑Party Channel:** ≥ 3 upstream publishers integrated; PRL revocations flow to AOG within 5 minutes.
|
||||
* **Zastava:** dSBOM reduces non‑reachable high/critical findings by ≥ 35% without raising exposure incidents.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
* **Crypto complexity:** profile misuse → strict guardrails + default safe profiles, strong linting.
|
||||
* **Cardinality/telemetry blow‑ups:** AOG label hygiene + sampling.
|
||||
* **Vendor adoption (Proof Channel):** seed with your own base images and internal frameworks; provide SDKs and reference publishers.
|
||||
* **Determinism regressions:** CI runs replay tests on golden SRMs; any drift fails the build.
|
||||
|
||||
---
|
||||
|
||||
## 90‑Day Moat‑First Milestones
|
||||
|
||||
1. **SRM v0.1**: schema, deterministic executor, CLI replay, golden tests.
|
||||
2. **Policy Engine MVP**: DSL + evaluator + UI simulation; policy unit tests; signed policy packs.
|
||||
3. **CryptoProfile/RootPack MVP**: FIPS + GOST working; dual‑signing; offline verify.
|
||||
4. **AOG MVP**: lineage service + OTLP exporter + Grafana pack; Trust SLI.
|
||||
5. **Trust Statement MVP**: JSON + PDF; ServiceNow ingest POC.
|
||||
6. **Proof Channel alpha**: publisher identity + VDR schema + local ledger; PRL.
|
||||
7. **Zastava α**: `diff-sbom` + exposure heuristics for `.NET Kestrel`; scheduler with offline mirrors.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Advisory AI agent guide
|
||||
|
||||
## Mission
|
||||
Advisory AI is the retrieval-augmented assistant that synthesizes advisory and VEX evidence into operator-ready summaries, conflict explanations, and remediation plans with strict provenance.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Review ./architecture.md for retrieval pipeline, guardrails, and profile support.
|
||||
2. Open ../../implplan/SPRINTS.md and locate stories for this component.
|
||||
3. Check ./TASKS.md and update status before/after work.
|
||||
4. Read README/architecture for design context and update as the implementation evolves.
|
||||
|
||||
## Guardrails
|
||||
- Uphold Aggregation-Only Contract boundaries when consuming ingestion data.
|
||||
- Preserve determinism and provenance in all derived outputs.
|
||||
- Document offline/air-gap pathways for any new feature.
|
||||
- Update telemetry/observability assets alongside feature work.
|
||||
# Advisory AI agent guide
|
||||
|
||||
## Mission
|
||||
Advisory AI is the retrieval-augmented assistant that synthesizes advisory and VEX evidence into operator-ready summaries, conflict explanations, and remediation plans with strict provenance.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Review ./architecture.md for retrieval pipeline, guardrails, and profile support.
|
||||
2. Open ../../implplan/SPRINTS.md and locate stories for this component.
|
||||
3. Check ./TASKS.md and update status before/after work.
|
||||
4. Read README/architecture for design context and update as the implementation evolves.
|
||||
|
||||
## Guardrails
|
||||
- Uphold Aggregation-Only Contract boundaries when consuming ingestion data.
|
||||
- Preserve determinism and provenance in all derived outputs.
|
||||
- Document offline/air-gap pathways for any new feature.
|
||||
- Update telemetry/observability assets alongside feature work.
|
||||
|
||||
@@ -1,29 +1,29 @@
|
||||
# StellaOps Advisory AI
|
||||
|
||||
Advisory AI is the retrieval-augmented assistant that synthesizes advisory and VEX evidence into operator-ready summaries, conflict explanations, and remediation plans with strict provenance.
|
||||
|
||||
## Responsibilities
|
||||
- Generate policy-aware advisory summaries with citations back to Conseiller and Excititor evidence.
|
||||
- Explain conflicting advisories/VEX statements using weights from VEX Lens and Policy Engine.
|
||||
- Propose remediation hints aligned with Offline Kit staging and export bundles.
|
||||
- Expose API/UI surfaces with guardrails on model prompts, outputs, and retention.
|
||||
|
||||
## Key components
|
||||
- RAG pipeline drawing from Conseiller, Excititor, VEX Lens, Policy Engine, and SBOM Service data.
|
||||
- Prompt templates and guard models enforcing provenance and redaction policies.
|
||||
- Vercel/offline inference workers with deterministic caching of generated artefacts.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Authority for tenant-aware access control.
|
||||
- Policy Engine for context-specific decisions and explain traces.
|
||||
- Console/CLI for interaction surfaces.
|
||||
- Export Center/Vuln Explorer for embedding generated briefs.
|
||||
|
||||
## Operational notes
|
||||
- Model cache management and offline bundle packaging per Epic 8 requirements.
|
||||
- Usage/latency dashboards for prompt/response monitoring.
|
||||
- Redaction policies validated against security/LLM guardrail tests.
|
||||
|
||||
## Epic alignment
|
||||
- Epic 8: Advisory AI Assistant.
|
||||
- DOCS-AI stories to be tracked in ../../TASKS.md.
|
||||
# StellaOps Advisory AI
|
||||
|
||||
Advisory AI is the retrieval-augmented assistant that synthesizes advisory and VEX evidence into operator-ready summaries, conflict explanations, and remediation plans with strict provenance.
|
||||
|
||||
## Responsibilities
|
||||
- Generate policy-aware advisory summaries with citations back to Conseiller and Excititor evidence.
|
||||
- Explain conflicting advisories/VEX statements using weights from VEX Lens and Policy Engine.
|
||||
- Propose remediation hints aligned with Offline Kit staging and export bundles.
|
||||
- Expose API/UI surfaces with guardrails on model prompts, outputs, and retention.
|
||||
|
||||
## Key components
|
||||
- RAG pipeline drawing from Conseiller, Excititor, VEX Lens, Policy Engine, and SBOM Service data.
|
||||
- Prompt templates and guard models enforcing provenance and redaction policies.
|
||||
- Vercel/offline inference workers with deterministic caching of generated artefacts.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Authority for tenant-aware access control.
|
||||
- Policy Engine for context-specific decisions and explain traces.
|
||||
- Console/CLI for interaction surfaces.
|
||||
- Export Center/Vuln Explorer for embedding generated briefs.
|
||||
|
||||
## Operational notes
|
||||
- Model cache management and offline bundle packaging per Epic 8 requirements.
|
||||
- Usage/latency dashboards for prompt/response monitoring.
|
||||
- Redaction policies validated against security/LLM guardrail tests.
|
||||
|
||||
## Epic alignment
|
||||
- Epic 8: Advisory AI Assistant.
|
||||
- DOCS-AI stories to be tracked in ../../TASKS.md.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — Advisory AI
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| ADVISORY-AI-DOCS-0001 | TODO | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | Align with ./AGENTS.md |
|
||||
| ADVISORY-AI-ENG-0001 | TODO | Module Team | Break down epic milestones into actionable stories. | Sync into ../../TASKS.md |
|
||||
| ADVISORY-AI-OPS-0001 | TODO | Ops Guild | Prepare runbooks/observability assets once MVP lands. | Document outputs in ./README.md |
|
||||
# Task board — Advisory AI
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| ADVISORY-AI-DOCS-0001 | TODO | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | Align with ./AGENTS.md |
|
||||
| ADVISORY-AI-ENG-0001 | TODO | Module Team | Break down epic milestones into actionable stories. | Sync into ../../TASKS.md |
|
||||
| ADVISORY-AI-OPS-0001 | TODO | Ops Guild | Prepare runbooks/observability assets once MVP lands. | Document outputs in ./README.md |
|
||||
|
||||
@@ -1,100 +1,100 @@
|
||||
# Advisory AI architecture
|
||||
|
||||
> Captures the retrieval, guardrail, and inference packaging requirements defined in the Advisory AI implementation plan and related module guides.
|
||||
|
||||
## 1) Goals
|
||||
|
||||
- Summarise advisories/VEX evidence into operator-ready briefs with citations.
|
||||
- Explain conflicting statements with provenance and trust weights (using VEX Lens & Excititor data).
|
||||
- Suggest remediation plans aligned with Offline Kit deployment models and scheduler follow-ups.
|
||||
- Operate deterministically where possible; cache generated artefacts with digests for audit.
|
||||
|
||||
## 2) Pipeline overview
|
||||
|
||||
```
|
||||
+---------------------+
|
||||
Concelier/VEX Lens | Evidence Retriever |
|
||||
Policy Engine ----> | (vector + keyword) | ---> Context Pack (JSON)
|
||||
Zastava runtime +---------------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Prompt |
|
||||
| Assembler |
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Guarded LLM |
|
||||
| (local/host)|
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| Citation & |
|
||||
| Validation |
|
||||
+-----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| Output cache |
|
||||
| (hash, bundle) |
|
||||
+----------------+
|
||||
```
|
||||
|
||||
## 3) Retrieval & context
|
||||
|
||||
- Hybrid search: vector embeddings (SBERT-compatible) + keyword filters for advisory IDs, PURLs, CVEs.
|
||||
- Context packs include:
|
||||
- Advisory raw excerpts with highlighted sections and source URLs.
|
||||
- VEX statements (normalized tuples + trust metadata).
|
||||
- Policy explain traces for the affected finding.
|
||||
- Runtime/impact hints from Zastava (exposure, entrypoints).
|
||||
- Export-ready remediation data (fixed versions, patches).
|
||||
|
||||
All context references include `content_hash` and `source_id` enabling verifiable citations.
|
||||
|
||||
## 4) Guardrails
|
||||
|
||||
- Prompt templates enforce structure: summary, conflicts, remediation, references.
|
||||
- Response validator ensures:
|
||||
- No hallucinated advisories (every fact must map to input context).
|
||||
- Citations follow `[n]` indexing referencing actual sources.
|
||||
- Remediation suggestions only cite policy-approved sources (fixed versions, vendor hotfixes).
|
||||
- Moderation/PII filters prevent leaking secrets; responses failing validation are rejected and logged.
|
||||
|
||||
## 5) Output persistence
|
||||
|
||||
- Cached artefacts stored in `advisory_ai_outputs` with fields:
|
||||
- `output_hash` (sha256 of JSON response).
|
||||
- `input_digest` (hash of context pack).
|
||||
- `summary`, `conflicts`, `remediation`, `citations`.
|
||||
- `generated_at`, `model_id`, `profile` (Sovereign/FIPS etc.).
|
||||
- `signatures` (optional DSSE if run in deterministic mode).
|
||||
- Offline bundle format contains `summary.md`, `citations.json`, `context_manifest.json`, `signatures/`.
|
||||
|
||||
## 6) Profiles & sovereignty
|
||||
|
||||
- **Profiles:** `default`, `fips-local` (FIPS-compliant local model), `gost-local`, `cloud-openai` (optional, disabled by default). Each profile defines allowed models, key management, and telemetry endpoints.
|
||||
- **CryptoProfile/RootPack integration:** generated artefacts can be signed using configured CryptoProfile to satisfy procurement/trust requirements.
|
||||
|
||||
## 7) APIs
|
||||
|
||||
- `POST /v1/advisory-ai/summaries` — generate (or retrieve cached) summary for `{advisoryKey, artifactId, policyVersion}`.
|
||||
- `POST /v1/advisory-ai/conflicts` — explain conflicting VEX statements with trust ranking.
|
||||
- `POST /v1/advisory-ai/remediation` — fetch remediation plan with target fix versions, prerequisites, verification steps.
|
||||
- `GET /v1/advisory-ai/outputs/{hash}` — retrieve cached artefact (used by CLI/Console/Export Center).
|
||||
|
||||
All endpoints accept `profile` parameter (default `fips-local`) and return `output_hash`, `input_digest`, and `citations` for verification.
|
||||
|
||||
## 8) Observability
|
||||
|
||||
- Metrics: `advisory_ai_requests_total{profile,type}`, `advisory_ai_latency_seconds`, `advisory_ai_validation_failures_total`.
|
||||
- Logs: include `output_hash`, `input_digest`, `profile`, `model_id`, `tenant`, `artifacts`. Sensitive context is not logged.
|
||||
- Traces: spans for retrieval, prompt assembly, model inference, validation, cache write.
|
||||
|
||||
## 9) Operational controls
|
||||
|
||||
- Feature flags per tenant (`ai.summary.enabled`, `ai.remediation.enabled`).
|
||||
- Rate limits (per tenant, per profile) enforced by Orchestrator to prevent runaway usage.
|
||||
- Offline/air-gapped deployments run local models packaged with Offline Kit; model weights validated via manifest digests.
|
||||
# Advisory AI architecture
|
||||
|
||||
> Captures the retrieval, guardrail, and inference packaging requirements defined in the Advisory AI implementation plan and related module guides.
|
||||
|
||||
## 1) Goals
|
||||
|
||||
- Summarise advisories/VEX evidence into operator-ready briefs with citations.
|
||||
- Explain conflicting statements with provenance and trust weights (using VEX Lens & Excititor data).
|
||||
- Suggest remediation plans aligned with Offline Kit deployment models and scheduler follow-ups.
|
||||
- Operate deterministically where possible; cache generated artefacts with digests for audit.
|
||||
|
||||
## 2) Pipeline overview
|
||||
|
||||
```
|
||||
+---------------------+
|
||||
Concelier/VEX Lens | Evidence Retriever |
|
||||
Policy Engine ----> | (vector + keyword) | ---> Context Pack (JSON)
|
||||
Zastava runtime +---------------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Prompt |
|
||||
| Assembler |
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-------------+
|
||||
| Guarded LLM |
|
||||
| (local/host)|
|
||||
+-------------+
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| Citation & |
|
||||
| Validation |
|
||||
+-----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| Output cache |
|
||||
| (hash, bundle) |
|
||||
+----------------+
|
||||
```
|
||||
|
||||
## 3) Retrieval & context
|
||||
|
||||
- Hybrid search: vector embeddings (SBERT-compatible) + keyword filters for advisory IDs, PURLs, CVEs.
|
||||
- Context packs include:
|
||||
- Advisory raw excerpts with highlighted sections and source URLs.
|
||||
- VEX statements (normalized tuples + trust metadata).
|
||||
- Policy explain traces for the affected finding.
|
||||
- Runtime/impact hints from Zastava (exposure, entrypoints).
|
||||
- Export-ready remediation data (fixed versions, patches).
|
||||
|
||||
All context references include `content_hash` and `source_id` enabling verifiable citations.
|
||||
|
||||
## 4) Guardrails
|
||||
|
||||
- Prompt templates enforce structure: summary, conflicts, remediation, references.
|
||||
- Response validator ensures:
|
||||
- No hallucinated advisories (every fact must map to input context).
|
||||
- Citations follow `[n]` indexing referencing actual sources.
|
||||
- Remediation suggestions only cite policy-approved sources (fixed versions, vendor hotfixes).
|
||||
- Moderation/PII filters prevent leaking secrets; responses failing validation are rejected and logged.
|
||||
|
||||
## 5) Output persistence
|
||||
|
||||
- Cached artefacts stored in `advisory_ai_outputs` with fields:
|
||||
- `output_hash` (sha256 of JSON response).
|
||||
- `input_digest` (hash of context pack).
|
||||
- `summary`, `conflicts`, `remediation`, `citations`.
|
||||
- `generated_at`, `model_id`, `profile` (Sovereign/FIPS etc.).
|
||||
- `signatures` (optional DSSE if run in deterministic mode).
|
||||
- Offline bundle format contains `summary.md`, `citations.json`, `context_manifest.json`, `signatures/`.
|
||||
|
||||
## 6) Profiles & sovereignty
|
||||
|
||||
- **Profiles:** `default`, `fips-local` (FIPS-compliant local model), `gost-local`, `cloud-openai` (optional, disabled by default). Each profile defines allowed models, key management, and telemetry endpoints.
|
||||
- **CryptoProfile/RootPack integration:** generated artefacts can be signed using configured CryptoProfile to satisfy procurement/trust requirements.
|
||||
|
||||
## 7) APIs
|
||||
|
||||
- `POST /v1/advisory-ai/summaries` — generate (or retrieve cached) summary for `{advisoryKey, artifactId, policyVersion}`.
|
||||
- `POST /v1/advisory-ai/conflicts` — explain conflicting VEX statements with trust ranking.
|
||||
- `POST /v1/advisory-ai/remediation` — fetch remediation plan with target fix versions, prerequisites, verification steps.
|
||||
- `GET /v1/advisory-ai/outputs/{hash}` — retrieve cached artefact (used by CLI/Console/Export Center).
|
||||
|
||||
All endpoints accept `profile` parameter (default `fips-local`) and return `output_hash`, `input_digest`, and `citations` for verification.
|
||||
|
||||
## 8) Observability
|
||||
|
||||
- Metrics: `advisory_ai_requests_total{profile,type}`, `advisory_ai_latency_seconds`, `advisory_ai_validation_failures_total`.
|
||||
- Logs: include `output_hash`, `input_digest`, `profile`, `model_id`, `tenant`, `artifacts`. Sensitive context is not logged.
|
||||
- Traces: spans for retrieval, prompt assembly, model inference, validation, cache write.
|
||||
|
||||
## 9) Operational controls
|
||||
|
||||
- Feature flags per tenant (`ai.summary.enabled`, `ai.remediation.enabled`).
|
||||
- Rate limits (per tenant, per profile) enforced by Orchestrator to prevent runaway usage.
|
||||
- Offline/air-gapped deployments run local models packaged with Offline Kit; model weights validated via manifest digests.
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
# Implementation plan — Advisory AI
|
||||
|
||||
## Current objectives
|
||||
- Deliver Epic milestones summarised below while maintaining determinism and offline parity.
|
||||
- Keep documentation, telemetry, and runbooks aligned with sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Roadmap: reconcile open stories in ../../TASKS.md with module backlog.
|
||||
- Delivery: ship features outlined in the epic while preserving AOC guardrails.
|
||||
- Validation: extend tests/fixtures to guarantee reproducibility and provenance.
|
||||
|
||||
## Epic milestones
|
||||
- Epic 8: Advisory AI Assistant.
|
||||
- DOCS-AI stories to be tracked in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up work.
|
||||
- Sync with owners listed in docs/implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — Advisory AI
|
||||
|
||||
## Current objectives
|
||||
- Deliver Epic milestones summarised below while maintaining determinism and offline parity.
|
||||
- Keep documentation, telemetry, and runbooks aligned with sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Roadmap: reconcile open stories in ../../TASKS.md with module backlog.
|
||||
- Delivery: ship features outlined in the epic while preserving AOC guardrails.
|
||||
- Validation: extend tests/fixtures to guarantee reproducibility and provenance.
|
||||
|
||||
## Epic milestones
|
||||
- Epic 8: Advisory AI Assistant.
|
||||
- DOCS-AI stories to be tracked in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up work.
|
||||
- Sync with owners listed in docs/implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Attestor agent guide
|
||||
|
||||
## Mission
|
||||
Attestor moves signed evidence through the trust chain by accepting DSSE bundles from Signer, registering them with Rekor v2, and serving deterministic verification payloads to other services.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
# Attestor agent guide
|
||||
|
||||
## Mission
|
||||
Attestor moves signed evidence through the trust chain by accepting DSSE bundles from Signer, registering them with Rekor v2, and serving deterministic verification payloads to other services.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
|
||||
@@ -1,54 +1,54 @@
|
||||
# StellaOps Attestor
|
||||
|
||||
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
|
||||
|
||||
## Why it exists
|
||||
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
|
||||
- **Policy enforcement:** verification policies ensure only approved issuers, key types, witnesses, and freshness windows are accepted.
|
||||
- **Sovereign/offline-ready:** Attestor archives envelopes, signatures, and proofs so air-gapped deployments can replay verification without contacting external services.
|
||||
|
||||
## Roles & surfaces
|
||||
- **Subjects:** immutable digests for container images, SBOMs, reports, and policy bundles.
|
||||
- **Issuers:** builders, scanners, policy engines, or operators signing DSSE envelopes using keyless (Fulcio), KMS/HSM, or FIDO2 keys.
|
||||
- **Consumers:** CLI/SDK, Console, Export Center, Scanner, Policy Engine, and Notify retrieving verification bundles or triggering policy checks.
|
||||
- **Scopes:** Authority issues `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for issuer/key management; every call is bound with mTLS + DPoP.
|
||||
|
||||
## Supported payloads
|
||||
- `StellaOps.BuildProvenance@1`, `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`, `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.PolicyEvaluation@1`, `StellaOps.RiskProfileEvidence@1`
|
||||
All predicates capture subjects, issuer metadata, policy context, materials, optional witnesses, and versioned schemas. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
## Trust & envelope model
|
||||
- DSSE envelopes are canonicalised, hashed, and stored alongside the Rekor UUID, index, and proof.
|
||||
- Signature modes span keyless (Fulcio), keyful (KMS/HSM), and hardware-backed (FIDO2). Multiple signatures are supported per envelope.
|
||||
- Proofs include Merkle inclusion path, checkpoint metadata, optional witness endorsements, and cached verification verdicts.
|
||||
- CAS/object storage retains envelopes + provenance for later replay; Rekor backends may be primary plus mirrors.
|
||||
|
||||
## UI, CLI, and SDK workflows
|
||||
- **Console:** Evidence browser, verification reports, chain-of-custody graph, issuer/key management, attestation workbench, and bulk verification flows.
|
||||
- **CLI / SDK:** `stella attest sign|verify|list|fetch|key` commands plus language SDKs to integrate build pipelines and offline verification scripts.
|
||||
- **Policy Studio:** Verification policies author required predicate types, issuers, witness requirements, and freshness windows; simulations show enforcement impact.
|
||||
|
||||
## Storage, offline & air-gap posture
|
||||
- MongoDB stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||
|
||||
## Observability & performance
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_cache_hit_ratio`, `attestor_rekor_latency_seconds`.
|
||||
- Logs capture tenant, issuer, subject digests, Rekor UUID, proof status, and policy verdict.
|
||||
- Performance target: ≥1 000 envelopes/minute per worker with cached verification, batched operations, and concurrency controls.
|
||||
|
||||
## Key integrations
|
||||
- Signer (DSSE source), Authority (scopes & tenancy), Export Center (attestation bundles), Policy Engine (verification policies), Scanner/Excititor (subject evidence), Notify (key rotation & verification alerts), Observability stack (dashboards/alerts).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-ATTEST-73-001 … DOCS-ATTEST-75-002 (Attestor console, key management, air-gap bundles) in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 (Export Center attestation packaging) in ../export-center/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 19 – Attestor Console:** console experience, verification APIs, issuer/key governance, transparency integration, and offline bundles.
|
||||
- **Epic 10 – Export Center:** provenance alignment so exports carry signed manifests and attestation bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
# StellaOps Attestor
|
||||
|
||||
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
|
||||
|
||||
## Why it exists
|
||||
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
|
||||
- **Policy enforcement:** verification policies ensure only approved issuers, key types, witnesses, and freshness windows are accepted.
|
||||
- **Sovereign/offline-ready:** Attestor archives envelopes, signatures, and proofs so air-gapped deployments can replay verification without contacting external services.
|
||||
|
||||
## Roles & surfaces
|
||||
- **Subjects:** immutable digests for container images, SBOMs, reports, and policy bundles.
|
||||
- **Issuers:** builders, scanners, policy engines, or operators signing DSSE envelopes using keyless (Fulcio), KMS/HSM, or FIDO2 keys.
|
||||
- **Consumers:** CLI/SDK, Console, Export Center, Scanner, Policy Engine, and Notify retrieving verification bundles or triggering policy checks.
|
||||
- **Scopes:** Authority issues `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for issuer/key management; every call is bound with mTLS + DPoP.
|
||||
|
||||
## Supported payloads
|
||||
- `StellaOps.BuildProvenance@1`, `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`, `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.PolicyEvaluation@1`, `StellaOps.RiskProfileEvidence@1`
|
||||
All predicates capture subjects, issuer metadata, policy context, materials, optional witnesses, and versioned schemas. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
## Trust & envelope model
|
||||
- DSSE envelopes are canonicalised, hashed, and stored alongside the Rekor UUID, index, and proof.
|
||||
- Signature modes span keyless (Fulcio), keyful (KMS/HSM), and hardware-backed (FIDO2). Multiple signatures are supported per envelope.
|
||||
- Proofs include Merkle inclusion path, checkpoint metadata, optional witness endorsements, and cached verification verdicts.
|
||||
- CAS/object storage retains envelopes + provenance for later replay; Rekor backends may be primary plus mirrors.
|
||||
|
||||
## UI, CLI, and SDK workflows
|
||||
- **Console:** Evidence browser, verification reports, chain-of-custody graph, issuer/key management, attestation workbench, and bulk verification flows.
|
||||
- **CLI / SDK:** `stella attest sign|verify|list|fetch|key` commands plus language SDKs to integrate build pipelines and offline verification scripts.
|
||||
- **Policy Studio:** Verification policies author required predicate types, issuers, witness requirements, and freshness windows; simulations show enforcement impact.
|
||||
|
||||
## Storage, offline & air-gap posture
|
||||
- MongoDB stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||
|
||||
## Observability & performance
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_cache_hit_ratio`, `attestor_rekor_latency_seconds`.
|
||||
- Logs capture tenant, issuer, subject digests, Rekor UUID, proof status, and policy verdict.
|
||||
- Performance target: ≥1 000 envelopes/minute per worker with cached verification, batched operations, and concurrency controls.
|
||||
|
||||
## Key integrations
|
||||
- Signer (DSSE source), Authority (scopes & tenancy), Export Center (attestation bundles), Policy Engine (verification policies), Scanner/Excititor (subject evidence), Notify (key rotation & verification alerts), Observability stack (dashboards/alerts).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-ATTEST-73-001 … DOCS-ATTEST-75-002 (Attestor console, key management, air-gap bundles) in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 (Export Center attestation packaging) in ../export-center/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 19 – Attestor Console:** console experience, verification APIs, issuer/key governance, transparency integration, and offline bundles.
|
||||
- **Epic 10 – Export Center:** provenance alignment so exports carry signed manifests and attestation bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — Attestor
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| ATTESTOR-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| ATTESTOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| ATTESTOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — Attestor
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| ATTESTOR-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| ATTESTOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| ATTESTOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
@@ -1,432 +1,432 @@
|
||||
# component_architecture_attestor.md — **Stella Ops Attestor** (2025Q4)
|
||||
|
||||
> Derived from Epic 19 – Attestor Console with provenance hooks aligned to the Export Center bundle workflows scoped in Epic 10.
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chain‑of‑trust to Stella Ops roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Excititor exports when configured).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Turn a signed DSSE envelope from the Signer into a **transparency‑logged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Attestor **does not sign**; it **must not** accept unsigned or third‑party‑signed bundles.
|
||||
* Attestor **does not decide PASS/FAIL**; it logs attestations for SBOMs, reports, and export artifacts.
|
||||
* Rekor v2 backends may be **local** (self‑hosted) or **remote**; Attestor handles both with retries, backoff, and idempotency.
|
||||
|
||||
---
|
||||
|
||||
## 1) Topology & dependencies
|
||||
|
||||
**Process shape:** single stateless service `stellaops/attestor` behind mTLS.
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks.
|
||||
* **Rekor v2** — tile‑backed transparency log endpoint(s).
|
||||
* **MinIO (S3)** — optional archive store for DSSE envelopes & verification bundles.
|
||||
* **MongoDB** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
|
||||
* **Redis** — dedupe/idempotency keys and short‑lived rate‑limit buckets.
|
||||
* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in.
|
||||
|
||||
Trust boundary: **Only the Signer** is allowed to call submission endpoints; enforced by **mTLS peer cert allowlist** + `aud=attestor` OpTok.
|
||||
|
||||
---
|
||||
|
||||
### Roles, identities & scopes
|
||||
- **Subjects** — immutable digests for artifacts (container images, SBOMs, reports) referenced in DSSE envelopes.
|
||||
- **Issuers** — authenticated builders/scanners/policy engines signing evidence; tracked with mode (`keyless`, `kms`, `hsm`, `fido2`) and tenant scope.
|
||||
- **Consumers** — Scanner, Export Center, CLI, Console, Policy Engine that verify proofs using Attestor APIs.
|
||||
- **Authority scopes** — `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for key management; all calls mTLS/DPoP-bound.
|
||||
|
||||
### Supported predicate types
|
||||
- `StellaOps.BuildProvenance@1`
|
||||
- `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`
|
||||
- `StellaOps.PolicyEvaluation@1`
|
||||
- `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.RiskProfileEvidence@1`
|
||||
|
||||
Each predicate embeds subject digests, issuer metadata, policy context, materials, and optional transparency hints. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
### Envelope & signature model
|
||||
- DSSE envelopes canonicalised (stable JSON ordering) prior to hashing.
|
||||
- Signature modes: keyless (Fulcio cert chain), keyful (KMS/HSM), hardware (FIDO2/WebAuthn). Multiple signatures allowed.
|
||||
- Rekor entry stores bundle hash, certificate chain, and optional witness endorsements.
|
||||
- Archive CAS retains original envelope plus metadata for offline verification.
|
||||
|
||||
### Verification pipeline overview
|
||||
1. Fetch envelope (from request, cache, or storage) and validate DSSE structure.
|
||||
2. Verify signature(s) against configured trust roots; evaluate issuer policy.
|
||||
3. Retrieve or acquire inclusion proof from Rekor (primary + optional mirror).
|
||||
4. Validate Merkle proof against checkpoint; optionally verify witness endorsement.
|
||||
5. Return cached verification bundle including policy verdict and timestamps.
|
||||
|
||||
### UI & CLI touchpoints
|
||||
- Console: Evidence browser, verification report, chain-of-custody graph, issuer/key management, attestation workbench, bulk verification views.
|
||||
- CLI: `stella attest sign|verify|list|fetch|key` with offline verification and export bundle support.
|
||||
- SDKs expose sign/verify primitives for build pipelines.
|
||||
|
||||
### Performance & observability targets
|
||||
- Throughput goal: ≥1 000 envelopes/minute per worker with cached verification.
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_rekor_latency_seconds`, `attestor_cache_hit_ratio`.
|
||||
- Logs include `tenant`, `issuer`, `subjectDigest`, `rekorUuid`, `proofStatus`; traces cover submission → Rekor → cache → response path.
|
||||
|
||||
---
|
||||
|
||||
## 2) Data model (Mongo)
|
||||
|
||||
Database: `attestor`
|
||||
|
||||
**Collections & schemas**
|
||||
|
||||
* `entries`
|
||||
|
||||
```
|
||||
{ _id: "<rekor-uuid>",
|
||||
artifact: { sha256: "<sha256>", kind: "sbom|report|vex-export", imageDigest?, subjectUri? },
|
||||
bundleSha256: "<sha256>", // canonicalized DSSE
|
||||
index: <int>, // log index/sequence if provided by backend
|
||||
proof: { // inclusion proof
|
||||
checkpoint: { origin, size, rootHash, timestamp },
|
||||
inclusion: { leafHash, path[] } // Merkle path (tiles)
|
||||
},
|
||||
log: { url, logId? },
|
||||
createdAt, status: "included|pending|failed",
|
||||
signerIdentity: { mode: "keyless|kms", issuer, san?, kid? }
|
||||
}
|
||||
```
|
||||
|
||||
* `dedupe`
|
||||
|
||||
```
|
||||
{ key: "bundle:<sha256>", rekorUuid, createdAt, ttlAt } // idempotency key
|
||||
```
|
||||
|
||||
* `audit`
|
||||
|
||||
```
|
||||
{ _id, ts, caller: { cn, mTLSThumbprint, sub, aud }, // from mTLS + OpTok
|
||||
action: "submit|verify|fetch",
|
||||
artifactSha256, bundleSha256, rekorUuid?, index?, result, latencyMs, backend }
|
||||
```
|
||||
|
||||
Indexes:
|
||||
|
||||
* `entries` on `artifact.sha256`, `bundleSha256`, `createdAt`, and `{status:1, createdAt:-1}`.
|
||||
* `dedupe.key` unique (TTL 24–48h).
|
||||
* `audit.ts` for time‑range queries.
|
||||
|
||||
---
|
||||
|
||||
## 3) Input contract (from Signer)
|
||||
|
||||
**Attestor accepts only** DSSE envelopes that satisfy all of:
|
||||
|
||||
1. **mTLS** peer certificate maps to `signer` service (CA‑pinned).
|
||||
2. **Authority** OpTok with `aud=attestor`, `scope=attestor.write`, DPoP or mTLS bound.
|
||||
3. DSSE envelope is **signed by the Signer’s key** (or includes a **Fulcio‑issued** cert chain) and **chains to configured roots** (Fulcio/KMS).
|
||||
4. **Predicate type** is one of Stella Ops types (sbom/report/vex‑export) with valid schema.
|
||||
5. `subject[*].digest.sha256` is present and canonicalized.
|
||||
|
||||
**Wire shape (JSON):**
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle": { "dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "<b64>", "signatures": [ ... ] },
|
||||
"certificateChain": [ "-----BEGIN CERTIFICATE-----..." ],
|
||||
"mode": "keyless" },
|
||||
"meta": {
|
||||
"artifact": { "sha256": "<subject sha256>", "kind": "sbom|report|vex-export", "imageDigest": "sha256:..." },
|
||||
"bundleSha256": "<sha256 of canonical dsse>",
|
||||
"logPreference": "primary", // "primary" | "mirror" | "both"
|
||||
"archive": true // whether Attestor should archive bundle to S3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) APIs
|
||||
|
||||
### 4.1 Submission
|
||||
|
||||
`POST /api/v1/rekor/entries` *(mTLS + OpTok required)*
|
||||
|
||||
* **Body**: as above.
|
||||
* **Behavior**:
|
||||
|
||||
* Verify caller (mTLS + OpTok).
|
||||
* Validate DSSE bundle (signature, cert chain to Fulcio/KMS; DSSE structure; payloadType allowed).
|
||||
* Idempotency: compute `bundleSha256`; check `dedupe`. If present, return existing `rekorUuid`.
|
||||
* Submit canonicalized bundle to Rekor v2 (primary or mirror according to `logPreference`).
|
||||
* Retrieve **inclusion proof** (blocking until inclusion or up to `proofTimeoutMs`); if backend returns promise only, return `status=pending` and retry asynchronously.
|
||||
* Persist `entries` record; archive DSSE to S3 if `archive=true`.
|
||||
* **Response 200**:
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "…",
|
||||
"index": 123456,
|
||||
"proof": {
|
||||
"checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "…", "timestamp": "…" },
|
||||
"inclusion": { "leafHash": "…", "path": ["…","…"] }
|
||||
},
|
||||
"logURL": "https://rekor…/api/v2/log/…/entries/…",
|
||||
"status": "included"
|
||||
}
|
||||
```
|
||||
* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`.
|
||||
|
||||
### 4.2 Proof retrieval
|
||||
|
||||
`GET /api/v1/rekor/entries/{uuid}`
|
||||
|
||||
* Returns `entries` row (refreshes proof from Rekor if stale/missing).
|
||||
* Accepts `?refresh=true` to force backend query.
|
||||
|
||||
### 4.3 Verification (third‑party or internal)
|
||||
|
||||
`POST /api/v1/rekor/verify`
|
||||
|
||||
* **Body** (one of):
|
||||
|
||||
* `{ "uuid": "…" }`
|
||||
* `{ "bundle": { …DSSE… } }`
|
||||
* `{ "artifactSha256": "…" }` *(looks up most recent entry)*
|
||||
|
||||
* **Checks**:
|
||||
|
||||
1. **Bundle signature** → cert chain to Fulcio/KMS roots configured.
|
||||
2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root.
|
||||
3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints).
|
||||
4. Confirm **subject.digest** matches caller‑provided hash (when given).
|
||||
|
||||
* **Response**:
|
||||
|
||||
```json
|
||||
{ "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
|
||||
```
|
||||
|
||||
### 4.4 Batch submission (optional)
|
||||
|
||||
`POST /api/v1/rekor/batch` accepts an array of submission objects; processes with per‑item results.
|
||||
|
||||
---
|
||||
|
||||
## 5) Rekor v2 driver (backend)
|
||||
|
||||
* **Canonicalization**: DSSE envelopes are **normalized** (stable JSON ordering, no insignificant whitespace) before hashing and submission.
|
||||
* **Transport**: HTTP/2 with retries (exponential backoff, jitter), budgeted timeouts.
|
||||
* **Idempotency**: if backend returns “already exists,” map to existing `uuid`.
|
||||
* **Proof acquisition**:
|
||||
|
||||
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff).
|
||||
* **Mirrors/dual logs**:
|
||||
|
||||
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
||||
* Optional **cloud endorsement**: POST to the Stella Ops cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id.
|
||||
|
||||
---
|
||||
|
||||
## 6) Security model
|
||||
|
||||
* **mTLS required** for submission from **Signer** (CA‑pinned).
|
||||
* **Authority token** with `aud=attestor` and DPoP/mTLS binding must be presented; Attestor verifies both.
|
||||
* **Bundle acceptance policy**:
|
||||
|
||||
* DSSE signature must chain to the configured **Fulcio** (keyless) or **KMS/HSM** roots.
|
||||
* SAN (Subject Alternative Name) must match **Signer identity** policy (e.g., `urn:stellaops:signer` or pinned OIDC issuer).
|
||||
* Predicate `predicateType` must be on allowlist (sbom/report/vex-export).
|
||||
* `subject.digest.sha256` values must be present and well‑formed (hex).
|
||||
* **No public submission** path. **Never** accept bundles from untrusted clients.
|
||||
* **Client certificate allowlists**: optional `security.mtls.allowedSubjects` / `allowedThumbprints` tighten peer identity checks beyond CA pinning.
|
||||
* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded.
|
||||
* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.
|
||||
|
||||
---
|
||||
|
||||
## 7) Storage & archival
|
||||
|
||||
* **Entries** in Mongo provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups.
|
||||
* **S3 archival** (if enabled):
|
||||
|
||||
```
|
||||
s3://stellaops/attest/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
bundle/<artifactSha256>.zip # optional verification bundle
|
||||
```
|
||||
* **Verification bundles** (zip):
|
||||
|
||||
* DSSE (`*.dsse.json`), proof (`*.proof.json`), `chain.pem` (certs), `README.txt` with verification steps & hashes.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & audit
|
||||
|
||||
**Metrics** (Prometheus):
|
||||
|
||||
* `attestor.submit_total{result,backend}`
|
||||
* `attestor.submit_latency_seconds{backend}`
|
||||
* `attestor.proof_fetch_total{result}`
|
||||
* `attestor.verify_total{result}`
|
||||
* `attestor.dedupe_hits_total`
|
||||
* `attestor.errors_total{type}`
|
||||
|
||||
**Correlation**:
|
||||
|
||||
* HTTP callers may supply `X-Correlation-Id`; Attestor will echo the header and push `CorrelationId` into the log scope for cross-service tracing.
|
||||
|
||||
**Tracing**:
|
||||
|
||||
* Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`.
|
||||
|
||||
**Audit**:
|
||||
|
||||
* Immutable `audit` rows (ts, caller, action, hashes, uuid, index, backend, result, latency).
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
attestor:
|
||||
listen: "https://0.0.0.0:8444"
|
||||
security:
|
||||
mtls:
|
||||
caBundle: /etc/ssl/signer-ca.pem
|
||||
requireClientCert: true
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
jwksUrl: "https://authority.internal/jwks"
|
||||
requireSenderConstraint: "dpop" # or "mtls"
|
||||
signerIdentity:
|
||||
mode: ["keyless","kms"]
|
||||
fulcioRoots: ["/etc/fulcio/root.pem"]
|
||||
allowedSANs: ["urn:stellaops:signer"]
|
||||
kmsKeys: ["kms://cluster-kms/stellaops-signer"]
|
||||
rekor:
|
||||
primary:
|
||||
url: "https://rekor-v2.internal"
|
||||
proofTimeoutMs: 15000
|
||||
pollIntervalMs: 250
|
||||
maxAttempts: 60
|
||||
mirror:
|
||||
enabled: false
|
||||
url: "https://rekor-v2.mirror"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/attestor"
|
||||
s3:
|
||||
enabled: true
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
prefix: "attest/"
|
||||
objectLock: "governance"
|
||||
redis:
|
||||
url: "redis://redis:6379/2"
|
||||
quotas:
|
||||
perCaller:
|
||||
qps: 50
|
||||
burst: 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) End‑to‑end sequences
|
||||
|
||||
**A) Submit & include (happy path)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
participant RK as Rekor v2
|
||||
|
||||
SW->>SG: POST /sign/dsse (OpTok+PoE)
|
||||
SG-->>SW: DSSE bundle (+certs)
|
||||
SW->>AT: POST /rekor/entries (mTLS + OpTok)
|
||||
AT->>AT: Validate DSSE (chain to Fulcio/KMS; signer identity)
|
||||
AT->>RK: submit(bundle)
|
||||
RK-->>AT: {uuid, index?}
|
||||
AT->>RK: poll inclusion until proof or timeout
|
||||
RK-->>AT: inclusion proof (checkpoint + path)
|
||||
AT-->>SW: {uuid, index, proof, logURL}
|
||||
```
|
||||
|
||||
**B) Verify by artifact digest (CLI)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI as stellaops verify
|
||||
participant SW as Scanner.WebService
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>SW: GET /catalog/artifacts/{id}
|
||||
SW-->>CLI: {artifactSha256, rekor: {uuid}}
|
||||
CLI->>AT: POST /rekor/verify { uuid }
|
||||
AT-->>CLI: { ok: true, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11) Failure modes & responses
|
||||
|
||||
| Condition | Return | Details | | |
|
||||
| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ |
|
||||
| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | |
|
||||
| Bundle not signed by trusted identity | `403 chain_untrusted` | DSSE accepted only from Signer identities | | |
|
||||
| Duplicate bundle | `409 duplicate_bundle` | Return existing `uuid` (idempotent) | | |
|
||||
| Rekor unreachable/timeout | `502 rekor_unavailable` | Retry with backoff; surface `Retry-After` | | |
|
||||
| Inclusion proof timeout | `202 accepted` | `status=pending`, background job continues to fetch proof | | |
|
||||
| Archive failure | `207 multi-status` | Entry recorded; archive will retry asynchronously | | |
|
||||
| Verification mismatch | `400 verify_failed` | Include reason: chain | leafHash | rootMismatch |
|
||||
|
||||
---
|
||||
|
||||
## 12) Performance & scale
|
||||
|
||||
* Stateless; scale horizontally.
|
||||
* **Targets**:
|
||||
|
||||
* Submit+proof P95 ≤ **300 ms** (warm log; local Rekor).
|
||||
* Verify P95 ≤ **30 ms** from cache; ≤ **120 ms** with live proof fetch.
|
||||
* 1k submissions/minute per replica sustained.
|
||||
* **Hot caches**: `dedupe` (bundle hash → uuid), recent `entries` by artifact sha256.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Happy path**: valid DSSE, inclusion within timeout.
|
||||
* **Idempotency**: resubmit same `bundleSha256` → same `uuid`.
|
||||
* **Security**: reject non‑Signer mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType.
|
||||
* **Rekor variants**: promise‑then‑proof, proof delayed, mirror dual‑submit, mirror failure.
|
||||
* **Verification**: corrupt leaf path, wrong root, tampered bundle.
|
||||
* **Throughput**: soak test with 10k submissions; latency SLOs, zero drops.
|
||||
|
||||
---
|
||||
|
||||
## 14) Implementation notes
|
||||
|
||||
* Language: **.NET 10** minimal API; `HttpClient` with **sockets handler** tuned for HTTP/2.
|
||||
* JSON: **canonical writer** for DSSE payload hashing.
|
||||
* Crypto: use **BouncyCastle**/**System.Security.Cryptography**; PEM parsing for cert chains.
|
||||
* Rekor client: pluggable driver; treat backend errors as retryable/non‑retryable with granular mapping.
|
||||
* Safety: size caps on bundles; decompress bombs guarded; strict UTF‑8.
|
||||
* CLI integration: `stellaops verify attestation <uuid|bundle|artifact>` calls `/rekor/verify`.
|
||||
|
||||
---
|
||||
|
||||
## 15) Optional features
|
||||
|
||||
* **Dual‑log** write (primary + mirror) and **cross‑log proof** packaging.
|
||||
* **Cloud endorsement**: send `{uuid, artifactSha256}` to Stella Ops cloud; store returned endorsement id for marketing/chain‑of‑custody.
|
||||
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
|
||||
|
||||
# component_architecture_attestor.md — **Stella Ops Attestor** (2025Q4)
|
||||
|
||||
> Derived from Epic 19 – Attestor Console with provenance hooks aligned to the Export Center bundle workflows scoped in Epic 10.
|
||||
|
||||
> **Scope.** Implementation‑ready architecture for the **Attestor**: the service that **submits** DSSE envelopes to **Rekor v2**, retrieves/validates inclusion proofs, caches results, and exposes verification APIs. It accepts DSSE **only** from the **Signer** over mTLS, enforces chain‑of‑trust to Stella Ops roots, and returns `{uuid, index, proof, logURL}` to calling services (Scanner.WebService for SBOMs; backend for final reports; Excititor exports when configured).
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & boundaries
|
||||
|
||||
**Mission.** Turn a signed DSSE envelope from the Signer into a **transparency‑logged, verifiable fact** with a durable, replayable proof (Merkle inclusion + (optional) checkpoint anchoring). Provide **fast verification** for downstream consumers and a stable retrieval interface for UI/CLI.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Attestor **does not sign**; it **must not** accept unsigned or third‑party‑signed bundles.
|
||||
* Attestor **does not decide PASS/FAIL**; it logs attestations for SBOMs, reports, and export artifacts.
|
||||
* Rekor v2 backends may be **local** (self‑hosted) or **remote**; Attestor handles both with retries, backoff, and idempotency.
|
||||
|
||||
---
|
||||
|
||||
## 1) Topology & dependencies
|
||||
|
||||
**Process shape:** single stateless service `stellaops/attestor` behind mTLS.
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
* **Signer** (caller) — authenticated via **mTLS** and **Authority** OpToks.
|
||||
* **Rekor v2** — tile‑backed transparency log endpoint(s).
|
||||
* **MinIO (S3)** — optional archive store for DSSE envelopes & verification bundles.
|
||||
* **MongoDB** — local cache of `{uuid, index, proof, artifactSha256, bundleSha256}`; job state; audit.
|
||||
* **Redis** — dedupe/idempotency keys and short‑lived rate‑limit buckets.
|
||||
* **Licensing Service (optional)** — “endorse” call for cross‑log publishing when customer opts‑in.
|
||||
|
||||
Trust boundary: **Only the Signer** is allowed to call submission endpoints; enforced by **mTLS peer cert allowlist** + `aud=attestor` OpTok.
|
||||
|
||||
---
|
||||
|
||||
### Roles, identities & scopes
|
||||
- **Subjects** — immutable digests for artifacts (container images, SBOMs, reports) referenced in DSSE envelopes.
|
||||
- **Issuers** — authenticated builders/scanners/policy engines signing evidence; tracked with mode (`keyless`, `kms`, `hsm`, `fido2`) and tenant scope.
|
||||
- **Consumers** — Scanner, Export Center, CLI, Console, Policy Engine that verify proofs using Attestor APIs.
|
||||
- **Authority scopes** — `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for key management; all calls mTLS/DPoP-bound.
|
||||
|
||||
### Supported predicate types
|
||||
- `StellaOps.BuildProvenance@1`
|
||||
- `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`
|
||||
- `StellaOps.PolicyEvaluation@1`
|
||||
- `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.RiskProfileEvidence@1`
|
||||
|
||||
Each predicate embeds subject digests, issuer metadata, policy context, materials, and optional transparency hints. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
### Envelope & signature model
|
||||
- DSSE envelopes canonicalised (stable JSON ordering) prior to hashing.
|
||||
- Signature modes: keyless (Fulcio cert chain), keyful (KMS/HSM), hardware (FIDO2/WebAuthn). Multiple signatures allowed.
|
||||
- Rekor entry stores bundle hash, certificate chain, and optional witness endorsements.
|
||||
- Archive CAS retains original envelope plus metadata for offline verification.
|
||||
|
||||
### Verification pipeline overview
|
||||
1. Fetch envelope (from request, cache, or storage) and validate DSSE structure.
|
||||
2. Verify signature(s) against configured trust roots; evaluate issuer policy.
|
||||
3. Retrieve or acquire inclusion proof from Rekor (primary + optional mirror).
|
||||
4. Validate Merkle proof against checkpoint; optionally verify witness endorsement.
|
||||
5. Return cached verification bundle including policy verdict and timestamps.
|
||||
|
||||
### UI & CLI touchpoints
|
||||
- Console: Evidence browser, verification report, chain-of-custody graph, issuer/key management, attestation workbench, bulk verification views.
|
||||
- CLI: `stella attest sign|verify|list|fetch|key` with offline verification and export bundle support.
|
||||
- SDKs expose sign/verify primitives for build pipelines.
|
||||
|
||||
### Performance & observability targets
|
||||
- Throughput goal: ≥1 000 envelopes/minute per worker with cached verification.
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_rekor_latency_seconds`, `attestor_cache_hit_ratio`.
|
||||
- Logs include `tenant`, `issuer`, `subjectDigest`, `rekorUuid`, `proofStatus`; traces cover submission → Rekor → cache → response path.
|
||||
|
||||
---
|
||||
|
||||
## 2) Data model (Mongo)
|
||||
|
||||
Database: `attestor`
|
||||
|
||||
**Collections & schemas**
|
||||
|
||||
* `entries`
|
||||
|
||||
```
|
||||
{ _id: "<rekor-uuid>",
|
||||
artifact: { sha256: "<sha256>", kind: "sbom|report|vex-export", imageDigest?, subjectUri? },
|
||||
bundleSha256: "<sha256>", // canonicalized DSSE
|
||||
index: <int>, // log index/sequence if provided by backend
|
||||
proof: { // inclusion proof
|
||||
checkpoint: { origin, size, rootHash, timestamp },
|
||||
inclusion: { leafHash, path[] } // Merkle path (tiles)
|
||||
},
|
||||
log: { url, logId? },
|
||||
createdAt, status: "included|pending|failed",
|
||||
signerIdentity: { mode: "keyless|kms", issuer, san?, kid? }
|
||||
}
|
||||
```
|
||||
|
||||
* `dedupe`
|
||||
|
||||
```
|
||||
{ key: "bundle:<sha256>", rekorUuid, createdAt, ttlAt } // idempotency key
|
||||
```
|
||||
|
||||
* `audit`
|
||||
|
||||
```
|
||||
{ _id, ts, caller: { cn, mTLSThumbprint, sub, aud }, // from mTLS + OpTok
|
||||
action: "submit|verify|fetch",
|
||||
artifactSha256, bundleSha256, rekorUuid?, index?, result, latencyMs, backend }
|
||||
```
|
||||
|
||||
Indexes:
|
||||
|
||||
* `entries` on `artifact.sha256`, `bundleSha256`, `createdAt`, and `{status:1, createdAt:-1}`.
|
||||
* `dedupe.key` unique (TTL 24–48h).
|
||||
* `audit.ts` for time‑range queries.
|
||||
|
||||
---
|
||||
|
||||
## 3) Input contract (from Signer)
|
||||
|
||||
**Attestor accepts only** DSSE envelopes that satisfy all of:
|
||||
|
||||
1. **mTLS** peer certificate maps to `signer` service (CA‑pinned).
|
||||
2. **Authority** OpTok with `aud=attestor`, `scope=attestor.write`, DPoP or mTLS bound.
|
||||
3. DSSE envelope is **signed by the Signer’s key** (or includes a **Fulcio‑issued** cert chain) and **chains to configured roots** (Fulcio/KMS).
|
||||
4. **Predicate type** is one of Stella Ops types (sbom/report/vex‑export) with valid schema.
|
||||
5. `subject[*].digest.sha256` is present and canonicalized.
|
||||
|
||||
**Wire shape (JSON):**
|
||||
|
||||
```json
|
||||
{
|
||||
"bundle": { "dsse": { "payloadType": "application/vnd.in-toto+json", "payload": "<b64>", "signatures": [ ... ] },
|
||||
"certificateChain": [ "-----BEGIN CERTIFICATE-----..." ],
|
||||
"mode": "keyless" },
|
||||
"meta": {
|
||||
"artifact": { "sha256": "<subject sha256>", "kind": "sbom|report|vex-export", "imageDigest": "sha256:..." },
|
||||
"bundleSha256": "<sha256 of canonical dsse>",
|
||||
"logPreference": "primary", // "primary" | "mirror" | "both"
|
||||
"archive": true // whether Attestor should archive bundle to S3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) APIs
|
||||
|
||||
### 4.1 Submission
|
||||
|
||||
`POST /api/v1/rekor/entries` *(mTLS + OpTok required)*
|
||||
|
||||
* **Body**: as above.
|
||||
* **Behavior**:
|
||||
|
||||
* Verify caller (mTLS + OpTok).
|
||||
* Validate DSSE bundle (signature, cert chain to Fulcio/KMS; DSSE structure; payloadType allowed).
|
||||
* Idempotency: compute `bundleSha256`; check `dedupe`. If present, return existing `rekorUuid`.
|
||||
* Submit canonicalized bundle to Rekor v2 (primary or mirror according to `logPreference`).
|
||||
* Retrieve **inclusion proof** (blocking until inclusion or up to `proofTimeoutMs`); if backend returns promise only, return `status=pending` and retry asynchronously.
|
||||
* Persist `entries` record; archive DSSE to S3 if `archive=true`.
|
||||
* **Response 200**:
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "…",
|
||||
"index": 123456,
|
||||
"proof": {
|
||||
"checkpoint": { "origin": "rekor@site", "size": 987654, "rootHash": "…", "timestamp": "…" },
|
||||
"inclusion": { "leafHash": "…", "path": ["…","…"] }
|
||||
},
|
||||
"logURL": "https://rekor…/api/v2/log/…/entries/…",
|
||||
"status": "included"
|
||||
}
|
||||
```
|
||||
* **Errors**: `401 invalid_token`, `403 not_signer|chain_untrusted`, `409 duplicate_bundle` (with existing `uuid`), `502 rekor_unavailable`, `504 proof_timeout`.
|
||||
|
||||
### 4.2 Proof retrieval
|
||||
|
||||
`GET /api/v1/rekor/entries/{uuid}`
|
||||
|
||||
* Returns `entries` row (refreshes proof from Rekor if stale/missing).
|
||||
* Accepts `?refresh=true` to force backend query.
|
||||
|
||||
### 4.3 Verification (third‑party or internal)
|
||||
|
||||
`POST /api/v1/rekor/verify`
|
||||
|
||||
* **Body** (one of):
|
||||
|
||||
* `{ "uuid": "…" }`
|
||||
* `{ "bundle": { …DSSE… } }`
|
||||
* `{ "artifactSha256": "…" }` *(looks up most recent entry)*
|
||||
|
||||
* **Checks**:
|
||||
|
||||
1. **Bundle signature** → cert chain to Fulcio/KMS roots configured.
|
||||
2. **Inclusion proof** → recompute leaf hash; verify Merkle path against checkpoint root.
|
||||
3. Optionally verify **checkpoint** against local trust anchors (if Rekor signs checkpoints).
|
||||
4. Confirm **subject.digest** matches caller‑provided hash (when given).
|
||||
|
||||
* **Response**:
|
||||
|
||||
```json
|
||||
{ "ok": true, "uuid": "…", "index": 123, "logURL": "…", "checkedAt": "…" }
|
||||
```
|
||||
|
||||
### 4.4 Batch submission (optional)
|
||||
|
||||
`POST /api/v1/rekor/batch` accepts an array of submission objects; processes with per‑item results.
|
||||
|
||||
---
|
||||
|
||||
## 5) Rekor v2 driver (backend)
|
||||
|
||||
* **Canonicalization**: DSSE envelopes are **normalized** (stable JSON ordering, no insignificant whitespace) before hashing and submission.
|
||||
* **Transport**: HTTP/2 with retries (exponential backoff, jitter), budgeted timeouts.
|
||||
* **Idempotency**: if backend returns “already exists,” map to existing `uuid`.
|
||||
* **Proof acquisition**:
|
||||
|
||||
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff).
|
||||
* **Mirrors/dual logs**:
|
||||
|
||||
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
||||
* Optional **cloud endorsement**: POST to the Stella Ops cloud `/attest/endorse` with `{uuid, artifactSha256}`; store returned endorsement id.
|
||||
|
||||
---
|
||||
|
||||
## 6) Security model
|
||||
|
||||
* **mTLS required** for submission from **Signer** (CA‑pinned).
|
||||
* **Authority token** with `aud=attestor` and DPoP/mTLS binding must be presented; Attestor verifies both.
|
||||
* **Bundle acceptance policy**:
|
||||
|
||||
* DSSE signature must chain to the configured **Fulcio** (keyless) or **KMS/HSM** roots.
|
||||
* SAN (Subject Alternative Name) must match **Signer identity** policy (e.g., `urn:stellaops:signer` or pinned OIDC issuer).
|
||||
* Predicate `predicateType` must be on allowlist (sbom/report/vex-export).
|
||||
* `subject.digest.sha256` values must be present and well‑formed (hex).
|
||||
* **No public submission** path. **Never** accept bundles from untrusted clients.
|
||||
* **Client certificate allowlists**: optional `security.mtls.allowedSubjects` / `allowedThumbprints` tighten peer identity checks beyond CA pinning.
|
||||
* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded.
|
||||
* **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor.
|
||||
|
||||
---
|
||||
|
||||
## 7) Storage & archival
|
||||
|
||||
* **Entries** in Mongo provide a local ledger keyed by `rekorUuid` and **artifact sha256** for quick reverse lookups.
|
||||
* **S3 archival** (if enabled):
|
||||
|
||||
```
|
||||
s3://stellaops/attest/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
bundle/<artifactSha256>.zip # optional verification bundle
|
||||
```
|
||||
* **Verification bundles** (zip):
|
||||
|
||||
* DSSE (`*.dsse.json`), proof (`*.proof.json`), `chain.pem` (certs), `README.txt` with verification steps & hashes.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & audit
|
||||
|
||||
**Metrics** (Prometheus):
|
||||
|
||||
* `attestor.submit_total{result,backend}`
|
||||
* `attestor.submit_latency_seconds{backend}`
|
||||
* `attestor.proof_fetch_total{result}`
|
||||
* `attestor.verify_total{result}`
|
||||
* `attestor.dedupe_hits_total`
|
||||
* `attestor.errors_total{type}`
|
||||
|
||||
**Correlation**:
|
||||
|
||||
* HTTP callers may supply `X-Correlation-Id`; Attestor will echo the header and push `CorrelationId` into the log scope for cross-service tracing.
|
||||
|
||||
**Tracing**:
|
||||
|
||||
* Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`.
|
||||
|
||||
**Audit**:
|
||||
|
||||
* Immutable `audit` rows (ts, caller, action, hashes, uuid, index, backend, result, latency).
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
attestor:
|
||||
listen: "https://0.0.0.0:8444"
|
||||
security:
|
||||
mtls:
|
||||
caBundle: /etc/ssl/signer-ca.pem
|
||||
requireClientCert: true
|
||||
authority:
|
||||
issuer: "https://authority.internal"
|
||||
jwksUrl: "https://authority.internal/jwks"
|
||||
requireSenderConstraint: "dpop" # or "mtls"
|
||||
signerIdentity:
|
||||
mode: ["keyless","kms"]
|
||||
fulcioRoots: ["/etc/fulcio/root.pem"]
|
||||
allowedSANs: ["urn:stellaops:signer"]
|
||||
kmsKeys: ["kms://cluster-kms/stellaops-signer"]
|
||||
rekor:
|
||||
primary:
|
||||
url: "https://rekor-v2.internal"
|
||||
proofTimeoutMs: 15000
|
||||
pollIntervalMs: 250
|
||||
maxAttempts: 60
|
||||
mirror:
|
||||
enabled: false
|
||||
url: "https://rekor-v2.mirror"
|
||||
mongo:
|
||||
uri: "mongodb://mongo/attestor"
|
||||
s3:
|
||||
enabled: true
|
||||
endpoint: "http://minio:9000"
|
||||
bucket: "stellaops"
|
||||
prefix: "attest/"
|
||||
objectLock: "governance"
|
||||
redis:
|
||||
url: "redis://redis:6379/2"
|
||||
quotas:
|
||||
perCaller:
|
||||
qps: 50
|
||||
burst: 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) End‑to‑end sequences
|
||||
|
||||
**A) Submit & include (happy path)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant SW as Scanner.WebService
|
||||
participant SG as Signer
|
||||
participant AT as Attestor
|
||||
participant RK as Rekor v2
|
||||
|
||||
SW->>SG: POST /sign/dsse (OpTok+PoE)
|
||||
SG-->>SW: DSSE bundle (+certs)
|
||||
SW->>AT: POST /rekor/entries (mTLS + OpTok)
|
||||
AT->>AT: Validate DSSE (chain to Fulcio/KMS; signer identity)
|
||||
AT->>RK: submit(bundle)
|
||||
RK-->>AT: {uuid, index?}
|
||||
AT->>RK: poll inclusion until proof or timeout
|
||||
RK-->>AT: inclusion proof (checkpoint + path)
|
||||
AT-->>SW: {uuid, index, proof, logURL}
|
||||
```
|
||||
|
||||
**B) Verify by artifact digest (CLI)**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant CLI as stellaops verify
|
||||
participant SW as Scanner.WebService
|
||||
participant AT as Attestor
|
||||
|
||||
CLI->>SW: GET /catalog/artifacts/{id}
|
||||
SW-->>CLI: {artifactSha256, rekor: {uuid}}
|
||||
CLI->>AT: POST /rekor/verify { uuid }
|
||||
AT-->>CLI: { ok: true, index, logURL }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11) Failure modes & responses
|
||||
|
||||
| Condition | Return | Details | | |
|
||||
| ------------------------------------- | ----------------------- | --------------------------------------------------------- | -------- | ------------ |
|
||||
| mTLS/OpTok invalid | `401 invalid_token` | Include `WWW-Authenticate` DPoP challenge when applicable | | |
|
||||
| Bundle not signed by trusted identity | `403 chain_untrusted` | DSSE accepted only from Signer identities | | |
|
||||
| Duplicate bundle | `409 duplicate_bundle` | Return existing `uuid` (idempotent) | | |
|
||||
| Rekor unreachable/timeout | `502 rekor_unavailable` | Retry with backoff; surface `Retry-After` | | |
|
||||
| Inclusion proof timeout | `202 accepted` | `status=pending`, background job continues to fetch proof | | |
|
||||
| Archive failure | `207 multi-status` | Entry recorded; archive will retry asynchronously | | |
|
||||
| Verification mismatch | `400 verify_failed` | Include reason: chain | leafHash | rootMismatch |
|
||||
|
||||
---
|
||||
|
||||
## 12) Performance & scale
|
||||
|
||||
* Stateless; scale horizontally.
|
||||
* **Targets**:
|
||||
|
||||
* Submit+proof P95 ≤ **300 ms** (warm log; local Rekor).
|
||||
* Verify P95 ≤ **30 ms** from cache; ≤ **120 ms** with live proof fetch.
|
||||
* 1k submissions/minute per replica sustained.
|
||||
* **Hot caches**: `dedupe` (bundle hash → uuid), recent `entries` by artifact sha256.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Happy path**: valid DSSE, inclusion within timeout.
|
||||
* **Idempotency**: resubmit same `bundleSha256` → same `uuid`.
|
||||
* **Security**: reject non‑Signer mTLS, wrong `aud`, DPoP replay, untrusted cert chain, forbidden predicateType.
|
||||
* **Rekor variants**: promise‑then‑proof, proof delayed, mirror dual‑submit, mirror failure.
|
||||
* **Verification**: corrupt leaf path, wrong root, tampered bundle.
|
||||
* **Throughput**: soak test with 10k submissions; latency SLOs, zero drops.
|
||||
|
||||
---
|
||||
|
||||
## 14) Implementation notes
|
||||
|
||||
* Language: **.NET 10** minimal API; `HttpClient` with **sockets handler** tuned for HTTP/2.
|
||||
* JSON: **canonical writer** for DSSE payload hashing.
|
||||
* Crypto: use **BouncyCastle**/**System.Security.Cryptography**; PEM parsing for cert chains.
|
||||
* Rekor client: pluggable driver; treat backend errors as retryable/non‑retryable with granular mapping.
|
||||
* Safety: size caps on bundles; decompress bombs guarded; strict UTF‑8.
|
||||
* CLI integration: `stellaops verify attestation <uuid|bundle|artifact>` calls `/rekor/verify`.
|
||||
|
||||
---
|
||||
|
||||
## 15) Optional features
|
||||
|
||||
* **Dual‑log** write (primary + mirror) and **cross‑log proof** packaging.
|
||||
* **Cloud endorsement**: send `{uuid, artifactSha256}` to Stella Ops cloud; store returned endorsement id for marketing/chain‑of‑custody.
|
||||
* **Checkpoint pinning**: periodically pin latest Rekor checkpoints to an external audit store for independent monitoring.
|
||||
|
||||
|
||||
@@ -1,74 +1,74 @@
|
||||
# Implementation plan — Attestor
|
||||
|
||||
## Delivery phases
|
||||
- **Phase 1 – Foundations**
|
||||
Build the Attestor service skeleton, DSSE bundle ingestion, mTLS/OpTok enforcement, Rekor v2 client, and cache the `{uuid,index,proof}` tuple. Publish base API (`POST /rekor/entries`, `GET /entries/{uuid}`) and Mongo schemas.
|
||||
- **Phase 2 – Policies & UI**
|
||||
Deliver verification policy authoring (Policy Studio integration), console views (evidence browser, verification reports, issuer management), and CLI verbs (`stella attest sign|verify|list|fetch`).
|
||||
- **Phase 3 – Scan & VEX support**
|
||||
Accept SBOM, ScanResults, VEX, and PolicyEvaluation predicates; integrate with Scanner, Export Center, Excititor, and Policy Engine pipelines. Ensure AOC invariants on ingestion.
|
||||
- **Phase 4 – Transparency & keys**
|
||||
Add multi-log submission (primary + mirror), witness endorsements, KMS/HSM/FIDO2 drivers, key rotation/revocation workflows, and audit trails.
|
||||
- **Phase 5 – Bulk & air gap**
|
||||
Implement batch submission/verification, DSSE archival to CAS/object storage, export/import bundles for Offline Kit, and mirror transparency log snapshots.
|
||||
- **Phase 6 – Performance & hardening**
|
||||
Optimise cache usage, parallel verification (target ≥1 k envelopes/minute per worker), extend observability (metrics/logs/traces), fuzz parsers, and finalise incident playbooks.
|
||||
|
||||
## Work breakdown
|
||||
- **Attestor service & libraries**
|
||||
- DSSE validation pipeline (payload whitelist, signature verification, trust roots).
|
||||
- Rekor client with inclusion-proof acquisition, retry/backoff, mirroring controls.
|
||||
- Mongo repositories for entries, dedupe, audit; CAS storage for DSSE envelopes.
|
||||
- Batch submission/verification APIs, verification cache, deterministic serialization.
|
||||
- Observability hooks: metrics (`attestor_submission_total`, `attestor_verify_seconds`), structured logs, OpenTelemetry traces.
|
||||
- **Signer & Authority integration**
|
||||
- Enforce mTLS peer validation, Authority scope mapping (`attestor.write`, `attestor.verify`), and DPoP binding.
|
||||
- Provide signer identity attestation metadata consumed by Attestor.
|
||||
- **Policy & Console**
|
||||
- Extend Policy Studio with `VerificationPolicy` authoring, approvals, and simulated results.
|
||||
- Console workflows: Evidence browser, verification reports, chain-of-custody graph, key management UI, bulk verification screens.
|
||||
- **CLI & SDK**
|
||||
- `stella attest` command group (sign/verify/list/fetch/key management) with DSSE canonicalisation and cosign interoperability.
|
||||
- SDK helpers for DSSE envelope creation, verification, and proof inspection.
|
||||
- **Export Center & Offline Kit**
|
||||
- Export Center adapters for attestation bundles; CLI/Console flows to export & import evidence in air-gapped environments.
|
||||
- Offline Kit scripts for replaying verification, mirroring transparency logs, and reporting gaps.
|
||||
- **Security & key management**
|
||||
- KMS/HSM/FIDO2 driver abstraction, key rotation and revocation runbooks, witness endorsements, and revocation telemetry.
|
||||
- **Docs & training**
|
||||
- Update module dossier (overview, architecture, implementation plan), key management guides, transparency reference, CLI/Console documentation, and air-gap runbooks.
|
||||
|
||||
## Cross-module dependencies
|
||||
- **Policy Studio / Policy Engine:** verification policy artefacts, explain integration, remediation hints.
|
||||
- **Export Center:** attestation bundle export/import, provenance linking.
|
||||
- **Authority & Tenancy:** scopes, identity attestations, tenant-aware issuer catalogues.
|
||||
- **Notifications:** attestation success/failure events, key rotation alerts.
|
||||
- **Observability:** dashboards and alerting for signing/verification pipelines.
|
||||
|
||||
## Acceptance criteria
|
||||
- Service ingests DSSE envelopes for all supported predicate types, logs them to configured transparency logs, and returns proofs with deterministic hashes.
|
||||
- Verification APIs/CLI/UI validate signatures, inclusion proofs, and policy compliance; cached verification accelerates repeated checks.
|
||||
- Verification policies gate attestation usage, enforcing issuer, freshness, signature count, and witness requirements.
|
||||
- Export Center and Offline Kit workflows bundle attestations and replay verification offline.
|
||||
- Observability coverage includes metrics, traces, logs, audit events, and alert triggers for key compromise, log outages, and verification failure spikes.
|
||||
- Performance target met (≥1 k envelopes/minute per worker) with horizontal scaling.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Key compromise or leakage:** enforce hardware-backed keys, rotation procedures, revocation checks, and incident runbooks.
|
||||
- **Parser bugs / malformed DSSE:** fuzz DSSE and predicate schemas, strict schema validation, fail closed.
|
||||
- **Transparency outage:** mirror logs, support witness endorsements, queue submissions for retry with exponential backoff.
|
||||
- **Policy complexity:** ship curated starter policies, provide simulation tooling, and document common scenarios.
|
||||
- **Offline gaps:** archive bundles and proof material, surface gaps to operators, and document compensating controls.
|
||||
|
||||
## Test strategy
|
||||
- **Unit:** DSSE validation, Rekor client, dedupe logic, key drivers, policy enforcement.
|
||||
- **Integration:** submit/verify flows across predicate types, multi-log publishing, batch operations, CLI/UI end-to-end exercises.
|
||||
- **Security:** tenant isolation, scope enforcement, key rotation regression, tamper detection.
|
||||
- **Performance:** throughput benchmarks, cache hit-rate monitoring, large batch verification.
|
||||
- **Chaos:** inject Rekor outages, network failures, corrupt bundles; ensure graceful degradation and auditable alerts.
|
||||
|
||||
## Definition of done
|
||||
- Phased milestones delivered with telemetry, documentation, and runbooks in place.
|
||||
- CLI/Console parity verified; Offline Kit procedures validated in sealed environment.
|
||||
- Cross-module dependencies acknowledged in ./TASKS.md and ../../TASKS.md.
|
||||
- Documentation set refreshed (overview, architecture, key management, transparency, CLI/UI) with imposed rule statement.
|
||||
# Implementation plan — Attestor
|
||||
|
||||
## Delivery phases
|
||||
- **Phase 1 – Foundations**
|
||||
Build the Attestor service skeleton, DSSE bundle ingestion, mTLS/OpTok enforcement, Rekor v2 client, and cache the `{uuid,index,proof}` tuple. Publish base API (`POST /rekor/entries`, `GET /entries/{uuid}`) and Mongo schemas.
|
||||
- **Phase 2 – Policies & UI**
|
||||
Deliver verification policy authoring (Policy Studio integration), console views (evidence browser, verification reports, issuer management), and CLI verbs (`stella attest sign|verify|list|fetch`).
|
||||
- **Phase 3 – Scan & VEX support**
|
||||
Accept SBOM, ScanResults, VEX, and PolicyEvaluation predicates; integrate with Scanner, Export Center, Excititor, and Policy Engine pipelines. Ensure AOC invariants on ingestion.
|
||||
- **Phase 4 – Transparency & keys**
|
||||
Add multi-log submission (primary + mirror), witness endorsements, KMS/HSM/FIDO2 drivers, key rotation/revocation workflows, and audit trails.
|
||||
- **Phase 5 – Bulk & air gap**
|
||||
Implement batch submission/verification, DSSE archival to CAS/object storage, export/import bundles for Offline Kit, and mirror transparency log snapshots.
|
||||
- **Phase 6 – Performance & hardening**
|
||||
Optimise cache usage, parallel verification (target ≥1 k envelopes/minute per worker), extend observability (metrics/logs/traces), fuzz parsers, and finalise incident playbooks.
|
||||
|
||||
## Work breakdown
|
||||
- **Attestor service & libraries**
|
||||
- DSSE validation pipeline (payload whitelist, signature verification, trust roots).
|
||||
- Rekor client with inclusion-proof acquisition, retry/backoff, mirroring controls.
|
||||
- Mongo repositories for entries, dedupe, audit; CAS storage for DSSE envelopes.
|
||||
- Batch submission/verification APIs, verification cache, deterministic serialization.
|
||||
- Observability hooks: metrics (`attestor_submission_total`, `attestor_verify_seconds`), structured logs, OpenTelemetry traces.
|
||||
- **Signer & Authority integration**
|
||||
- Enforce mTLS peer validation, Authority scope mapping (`attestor.write`, `attestor.verify`), and DPoP binding.
|
||||
- Provide signer identity attestation metadata consumed by Attestor.
|
||||
- **Policy & Console**
|
||||
- Extend Policy Studio with `VerificationPolicy` authoring, approvals, and simulated results.
|
||||
- Console workflows: Evidence browser, verification reports, chain-of-custody graph, key management UI, bulk verification screens.
|
||||
- **CLI & SDK**
|
||||
- `stella attest` command group (sign/verify/list/fetch/key management) with DSSE canonicalisation and cosign interoperability.
|
||||
- SDK helpers for DSSE envelope creation, verification, and proof inspection.
|
||||
- **Export Center & Offline Kit**
|
||||
- Export Center adapters for attestation bundles; CLI/Console flows to export & import evidence in air-gapped environments.
|
||||
- Offline Kit scripts for replaying verification, mirroring transparency logs, and reporting gaps.
|
||||
- **Security & key management**
|
||||
- KMS/HSM/FIDO2 driver abstraction, key rotation and revocation runbooks, witness endorsements, and revocation telemetry.
|
||||
- **Docs & training**
|
||||
- Update module dossier (overview, architecture, implementation plan), key management guides, transparency reference, CLI/Console documentation, and air-gap runbooks.
|
||||
|
||||
## Cross-module dependencies
|
||||
- **Policy Studio / Policy Engine:** verification policy artefacts, explain integration, remediation hints.
|
||||
- **Export Center:** attestation bundle export/import, provenance linking.
|
||||
- **Authority & Tenancy:** scopes, identity attestations, tenant-aware issuer catalogues.
|
||||
- **Notifications:** attestation success/failure events, key rotation alerts.
|
||||
- **Observability:** dashboards and alerting for signing/verification pipelines.
|
||||
|
||||
## Acceptance criteria
|
||||
- Service ingests DSSE envelopes for all supported predicate types, logs them to configured transparency logs, and returns proofs with deterministic hashes.
|
||||
- Verification APIs/CLI/UI validate signatures, inclusion proofs, and policy compliance; cached verification accelerates repeated checks.
|
||||
- Verification policies gate attestation usage, enforcing issuer, freshness, signature count, and witness requirements.
|
||||
- Export Center and Offline Kit workflows bundle attestations and replay verification offline.
|
||||
- Observability coverage includes metrics, traces, logs, audit events, and alert triggers for key compromise, log outages, and verification failure spikes.
|
||||
- Performance target met (≥1 k envelopes/minute per worker) with horizontal scaling.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Key compromise or leakage:** enforce hardware-backed keys, rotation procedures, revocation checks, and incident runbooks.
|
||||
- **Parser bugs / malformed DSSE:** fuzz DSSE and predicate schemas, strict schema validation, fail closed.
|
||||
- **Transparency outage:** mirror logs, support witness endorsements, queue submissions for retry with exponential backoff.
|
||||
- **Policy complexity:** ship curated starter policies, provide simulation tooling, and document common scenarios.
|
||||
- **Offline gaps:** archive bundles and proof material, surface gaps to operators, and document compensating controls.
|
||||
|
||||
## Test strategy
|
||||
- **Unit:** DSSE validation, Rekor client, dedupe logic, key drivers, policy enforcement.
|
||||
- **Integration:** submit/verify flows across predicate types, multi-log publishing, batch operations, CLI/UI end-to-end exercises.
|
||||
- **Security:** tenant isolation, scope enforcement, key rotation regression, tamper detection.
|
||||
- **Performance:** throughput benchmarks, cache hit-rate monitoring, large batch verification.
|
||||
- **Chaos:** inject Rekor outages, network failures, corrupt bundles; ensure graceful degradation and auditable alerts.
|
||||
|
||||
## Definition of done
|
||||
- Phased milestones delivered with telemetry, documentation, and runbooks in place.
|
||||
- CLI/Console parity verified; Offline Kit procedures validated in sealed environment.
|
||||
- Cross-module dependencies acknowledged in ./TASKS.md and ../../TASKS.md.
|
||||
- Documentation set refreshed (overview, architecture, key management, transparency, CLI/UI) with imposed rule statement.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Authority agent guide
|
||||
|
||||
## Mission
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# Authority agent guide
|
||||
|
||||
## Mission
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,40 +1,40 @@
|
||||
# StellaOps Authority
|
||||
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Responsibilities
|
||||
- Expose device-code, auth-code, and client-credential flows with DPoP or mTLS binding.
|
||||
- Manage signing keys, JWKS rotation, and PoE integration for plan enforcement.
|
||||
- Emit structured audit events and enforce tenant-aware scope policies.
|
||||
- Provide plugin surface for custom identity providers and credential validators.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Authority` web host.
|
||||
- `StellaOps.Authority.Plugin.*` extensions for secret stores, identity bridges, and OpTok validation.
|
||||
- Telemetry and audit pipeline feeding Security/Observability stacks.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Signer/Attestor for PoE and OpTok introspection.
|
||||
- CLI/UI for login flows and token management.
|
||||
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (see ./operations/key-rotation.md).
|
||||
- Grafana/Prometheus dashboards for auth latency/issuance.
|
||||
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
|
||||
## Backlog references
|
||||
- DOCS-SEC-62-001 (scope hardening doc) in ../../TASKS.md.
|
||||
- AUTH-POLICY-20-001/002 follow-ups in src/Authority/StellaOps.Authority/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** enforce OpTok scopes and guardrails supporting raw ingestion boundaries.
|
||||
- **Epic 2 – Policy Engine & Editor:** supply policy evaluation/principal scopes and short-lived tokens for evaluator workflows.
|
||||
- **Epic 4 – Policy Studio:** integrate approval/promotion signatures and policy registry access controls.
|
||||
- **Epic 14 – Identity & Tenancy:** deliver tenant isolation, RBAC hierarchies, and governance tooling for authentication.
|
||||
# StellaOps Authority
|
||||
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Responsibilities
|
||||
- Expose device-code, auth-code, and client-credential flows with DPoP or mTLS binding.
|
||||
- Manage signing keys, JWKS rotation, and PoE integration for plan enforcement.
|
||||
- Emit structured audit events and enforce tenant-aware scope policies.
|
||||
- Provide plugin surface for custom identity providers and credential validators.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Authority` web host.
|
||||
- `StellaOps.Authority.Plugin.*` extensions for secret stores, identity bridges, and OpTok validation.
|
||||
- Telemetry and audit pipeline feeding Security/Observability stacks.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Signer/Attestor for PoE and OpTok introspection.
|
||||
- CLI/UI for login flows and token management.
|
||||
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (see ./operations/key-rotation.md).
|
||||
- Grafana/Prometheus dashboards for auth latency/issuance.
|
||||
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
|
||||
## Backlog references
|
||||
- DOCS-SEC-62-001 (scope hardening doc) in ../../TASKS.md.
|
||||
- AUTH-POLICY-20-001/002 follow-ups in src/Authority/StellaOps.Authority/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** enforce OpTok scopes and guardrails supporting raw ingestion boundaries.
|
||||
- **Epic 2 – Policy Engine & Editor:** supply policy evaluation/principal scopes and short-lived tokens for evaluator workflows.
|
||||
- **Epic 4 – Policy Studio:** integrate approval/promotion signatures and policy registry access controls.
|
||||
- **Epic 14 – Identity & Tenancy:** deliver tenant isolation, RBAC hierarchies, and governance tooling for authentication.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — Authority
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| AUTHORITY-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| AUTHORITY-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| AUTHORITY-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — Authority
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| AUTHORITY-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| AUTHORITY-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| AUTHORITY-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Implementation plan — Authority
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** deliver OpTok scopes, guardrails, and AOC verifier hooks for ingestion services.
|
||||
- **Epic 2 – Policy Engine & Editor:** support policy evaluator flows (device-code, client credentials, scope sandboxing).
|
||||
- **Epic 4 – Policy Studio:** provide registry/promotion signing, approvals, and fresh-auth prompts.
|
||||
- **Epic 14 – Identity & Tenancy:** implement tenant isolation, RBAC hierarchies, audit trails, and PoE integration.
|
||||
- Track additional work (DOCS-SEC-62-001, AUTH-POLICY-20-001/002) in ../../TASKS.md and src/Authority/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — Authority
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** deliver OpTok scopes, guardrails, and AOC verifier hooks for ingestion services.
|
||||
- **Epic 2 – Policy Engine & Editor:** support policy evaluator flows (device-code, client credentials, scope sandboxing).
|
||||
- **Epic 4 – Policy Studio:** provide registry/promotion signing, approvals, and fresh-auth prompts.
|
||||
- **Epic 14 – Identity & Tenancy:** implement tenant isolation, RBAC hierarchies, audit trails, and PoE integration.
|
||||
- Track additional work (DOCS-SEC-62-001, AUTH-POLICY-20-001/002) in ../../TASKS.md and src/Authority/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,97 +1,97 @@
|
||||
# Authority Backup & Restore Runbook
|
||||
|
||||
## Scope
|
||||
- **Applies to:** StellaOps Authority deployments running the official `ops/authority/docker-compose.authority.yaml` stack or equivalent Kubernetes packaging.
|
||||
- **Artifacts covered:** MongoDB (`stellaops-authority` database), Authority configuration (`etc/authority.yaml`), plugin manifests under `etc/authority.plugins/`, and signing key material stored in the `authority-keys` volume (defaults to `/app/keys` inside the container).
|
||||
- **Frequency:** Run the full procedure prior to upgrades, before rotating keys, and at least once per 24 h in production. Store snapshots in an encrypted, access-controlled vault.
|
||||
|
||||
## Inventory Checklist
|
||||
| Component | Location (compose default) | Notes |
|
||||
| --- | --- | --- |
|
||||
| Mongo data | `mongo-data` volume (`/var/lib/docker/volumes/.../mongo-data`) | Contains all Authority collections (`AuthorityUser`, `AuthorityClient`, `AuthorityToken`, etc.). |
|
||||
| Configuration | `etc/authority.yaml` | Mounted read-only into the container at `/etc/authority.yaml`. |
|
||||
| Plugin manifests | `etc/authority.plugins/*.yaml` | Includes `standard.yaml` with `tokenSigning.keyDirectory`. |
|
||||
| Signing keys | `authority-keys` volume -> `/app/keys` | Path is derived from `tokenSigning.keyDirectory` (defaults to `../keys` relative to the manifest). |
|
||||
|
||||
> **TIP:** Confirm the deployed key directory via `tokenSigning.keyDirectory` in `etc/authority.plugins/standard.yaml`; some installations relocate keys to `/var/lib/stellaops/authority/keys`.
|
||||
|
||||
## Hot Backup (no downtime)
|
||||
1. **Create output directory:** `mkdir -p backup/$(date +%Y-%m-%d)` on the host.
|
||||
2. **Dump Mongo:**
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml exec mongo \
|
||||
mongodump --archive=/dump/authority-$(date +%Y%m%dT%H%M%SZ).gz \
|
||||
--gzip --db stellaops-authority
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml cp \
|
||||
mongo:/dump/authority-$(date +%Y%m%dT%H%M%SZ).gz backup/
|
||||
```
|
||||
The `mongodump` archive preserves indexes and can be restored with `mongorestore --archive --gzip`.
|
||||
3. **Capture configuration + manifests:**
|
||||
```bash
|
||||
cp etc/authority.yaml backup/
|
||||
rsync -a etc/authority.plugins/ backup/authority.plugins/
|
||||
```
|
||||
4. **Export signing keys:** the compose file maps `authority-keys` to a local Docker volume. Snapshot it without stopping the service:
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v authority-keys:/keys \
|
||||
-v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/authority-keys-$(date +%Y%m%dT%H%M%SZ).tar.gz -C /keys .
|
||||
```
|
||||
5. **Checksum:** generate SHA-256 digests for every file and store them alongside the artefacts.
|
||||
6. **Encrypt & upload:** wrap the backup folder using your secrets management standard (e.g., age, GPG) and upload to the designated offline vault.
|
||||
|
||||
## Cold Backup (planned downtime)
|
||||
1. Notify stakeholders and drain traffic (CLI clients should refresh tokens afterwards).
|
||||
2. Stop services:
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml down
|
||||
```
|
||||
3. Back up volumes directly using `tar`:
|
||||
```bash
|
||||
docker run --rm -v mongo-data:/data -v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/mongo-data-$(date +%Y%m%d).tar.gz -C /data .
|
||||
docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/authority-keys-$(date +%Y%m%d).tar.gz -C /keys .
|
||||
```
|
||||
4. Copy configuration + manifests as in the hot backup (steps 3–6).
|
||||
5. Restart services and verify health:
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml up -d
|
||||
curl -fsS http://localhost:8080/ready
|
||||
```
|
||||
|
||||
## Restore Procedure
|
||||
1. **Provision clean volumes:** remove existing volumes if you’re rebuilding a node (`docker volume rm mongo-data authority-keys`), then recreate the compose stack so empty volumes exist.
|
||||
2. **Restore Mongo:**
|
||||
```bash
|
||||
docker compose exec -T mongo mongorestore --archive --gzip --drop < backup/authority-YYYYMMDDTHHMMSSZ.gz
|
||||
```
|
||||
Use `--drop` to replace collections; omit if doing a partial restore.
|
||||
3. **Restore configuration/manifests:** copy `authority.yaml` and `authority.plugins/*` into place before starting the Authority container.
|
||||
4. **Restore signing keys:** untar into the mounted volume:
|
||||
```bash
|
||||
docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
|
||||
busybox tar xzf /backup/authority-keys-YYYYMMDD.tar.gz -C /keys
|
||||
```
|
||||
Ensure file permissions remain `600` for private keys (`chmod -R 600`).
|
||||
5. **Start services & validate:**
|
||||
```bash
|
||||
docker compose up -d
|
||||
curl -fsS http://localhost:8080/health
|
||||
```
|
||||
# Authority Backup & Restore Runbook
|
||||
|
||||
## Scope
|
||||
- **Applies to:** StellaOps Authority deployments running the official `ops/authority/docker-compose.authority.yaml` stack or equivalent Kubernetes packaging.
|
||||
- **Artifacts covered:** MongoDB (`stellaops-authority` database), Authority configuration (`etc/authority.yaml`), plugin manifests under `etc/authority.plugins/`, and signing key material stored in the `authority-keys` volume (defaults to `/app/keys` inside the container).
|
||||
- **Frequency:** Run the full procedure prior to upgrades, before rotating keys, and at least once per 24 h in production. Store snapshots in an encrypted, access-controlled vault.
|
||||
|
||||
## Inventory Checklist
|
||||
| Component | Location (compose default) | Notes |
|
||||
| --- | --- | --- |
|
||||
| Mongo data | `mongo-data` volume (`/var/lib/docker/volumes/.../mongo-data`) | Contains all Authority collections (`AuthorityUser`, `AuthorityClient`, `AuthorityToken`, etc.). |
|
||||
| Configuration | `etc/authority.yaml` | Mounted read-only into the container at `/etc/authority.yaml`. |
|
||||
| Plugin manifests | `etc/authority.plugins/*.yaml` | Includes `standard.yaml` with `tokenSigning.keyDirectory`. |
|
||||
| Signing keys | `authority-keys` volume -> `/app/keys` | Path is derived from `tokenSigning.keyDirectory` (defaults to `../keys` relative to the manifest). |
|
||||
|
||||
> **TIP:** Confirm the deployed key directory via `tokenSigning.keyDirectory` in `etc/authority.plugins/standard.yaml`; some installations relocate keys to `/var/lib/stellaops/authority/keys`.
|
||||
|
||||
## Hot Backup (no downtime)
|
||||
1. **Create output directory:** `mkdir -p backup/$(date +%Y-%m-%d)` on the host.
|
||||
2. **Dump Mongo:**
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml exec mongo \
|
||||
mongodump --archive=/dump/authority-$(date +%Y%m%dT%H%M%SZ).gz \
|
||||
--gzip --db stellaops-authority
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml cp \
|
||||
mongo:/dump/authority-$(date +%Y%m%dT%H%M%SZ).gz backup/
|
||||
```
|
||||
The `mongodump` archive preserves indexes and can be restored with `mongorestore --archive --gzip`.
|
||||
3. **Capture configuration + manifests:**
|
||||
```bash
|
||||
cp etc/authority.yaml backup/
|
||||
rsync -a etc/authority.plugins/ backup/authority.plugins/
|
||||
```
|
||||
4. **Export signing keys:** the compose file maps `authority-keys` to a local Docker volume. Snapshot it without stopping the service:
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v authority-keys:/keys \
|
||||
-v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/authority-keys-$(date +%Y%m%dT%H%M%SZ).tar.gz -C /keys .
|
||||
```
|
||||
5. **Checksum:** generate SHA-256 digests for every file and store them alongside the artefacts.
|
||||
6. **Encrypt & upload:** wrap the backup folder using your secrets management standard (e.g., age, GPG) and upload to the designated offline vault.
|
||||
|
||||
## Cold Backup (planned downtime)
|
||||
1. Notify stakeholders and drain traffic (CLI clients should refresh tokens afterwards).
|
||||
2. Stop services:
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml down
|
||||
```
|
||||
3. Back up volumes directly using `tar`:
|
||||
```bash
|
||||
docker run --rm -v mongo-data:/data -v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/mongo-data-$(date +%Y%m%d).tar.gz -C /data .
|
||||
docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
|
||||
busybox tar czf /backup/authority-keys-$(date +%Y%m%d).tar.gz -C /keys .
|
||||
```
|
||||
4. Copy configuration + manifests as in the hot backup (steps 3–6).
|
||||
5. Restart services and verify health:
|
||||
```bash
|
||||
docker compose -f ops/authority/docker-compose.authority.yaml up -d
|
||||
curl -fsS http://localhost:8080/ready
|
||||
```
|
||||
|
||||
## Restore Procedure
|
||||
1. **Provision clean volumes:** remove existing volumes if you’re rebuilding a node (`docker volume rm mongo-data authority-keys`), then recreate the compose stack so empty volumes exist.
|
||||
2. **Restore Mongo:**
|
||||
```bash
|
||||
docker compose exec -T mongo mongorestore --archive --gzip --drop < backup/authority-YYYYMMDDTHHMMSSZ.gz
|
||||
```
|
||||
Use `--drop` to replace collections; omit if doing a partial restore.
|
||||
3. **Restore configuration/manifests:** copy `authority.yaml` and `authority.plugins/*` into place before starting the Authority container.
|
||||
4. **Restore signing keys:** untar into the mounted volume:
|
||||
```bash
|
||||
docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
|
||||
busybox tar xzf /backup/authority-keys-YYYYMMDD.tar.gz -C /keys
|
||||
```
|
||||
Ensure file permissions remain `600` for private keys (`chmod -R 600`).
|
||||
5. **Start services & validate:**
|
||||
```bash
|
||||
docker compose up -d
|
||||
curl -fsS http://localhost:8080/health
|
||||
```
|
||||
6. **Validate JWKS and tokens:** call `/jwks` and issue a short-lived token via the CLI to confirm key material matches expectations. If the restored environment requires a fresh signing key, follow the rotation SOP in [`docs/11_AUTHORITY.md`](../../../11_AUTHORITY.md) using `ops/authority/key-rotation.sh` to invoke `/internal/signing/rotate`.
|
||||
|
||||
## Disaster Recovery Notes
|
||||
- **Air-gapped replication:** replicate archives via the Offline Update Kit transport channels; never attach USB devices without scanning.
|
||||
- **Retention:** maintain 30 daily snapshots + 12 monthly archival copies. Rotate encryption keys annually.
|
||||
|
||||
## Disaster Recovery Notes
|
||||
- **Air-gapped replication:** replicate archives via the Offline Update Kit transport channels; never attach USB devices without scanning.
|
||||
- **Retention:** maintain 30 daily snapshots + 12 monthly archival copies. Rotate encryption keys annually.
|
||||
- **Key compromise:** if signing keys are suspected compromised, restore from the latest clean backup, rotate via OPS3 (see `ops/authority/key-rotation.sh` and [`docs/11_AUTHORITY.md`](../../../11_AUTHORITY.md)), and publish a revocation notice.
|
||||
- **Mongo version:** keep dump/restore images pinned to the deployment version (compose uses `mongo:7`). Driver 3.5.0 requires MongoDB **4.2+**—clusters still on 4.0 must be upgraded before restore, and future driver releases will drop 4.0 entirely. citeturn1open1
|
||||
|
||||
## Verification Checklist
|
||||
- [ ] `/ready` reports all identity providers ready.
|
||||
- [ ] OAuth flows issue tokens signed by the restored keys.
|
||||
- [ ] `PluginRegistrationSummary` logs expected providers on startup.
|
||||
- [ ] Revocation manifest export (`dotnet run --project src/Authority/StellaOps.Authority`) succeeds.
|
||||
- [ ] Monitoring dashboards show metrics resuming (see OPS5 deliverables).
|
||||
|
||||
- **Mongo version:** keep dump/restore images pinned to the deployment version (compose uses `mongo:7`). Driver 3.5.0 requires MongoDB **4.2+**—clusters still on 4.0 must be upgraded before restore, and future driver releases will drop 4.0 entirely. citeturn1open1
|
||||
|
||||
## Verification Checklist
|
||||
- [ ] `/ready` reports all identity providers ready.
|
||||
- [ ] OAuth flows issue tokens signed by the restored keys.
|
||||
- [ ] `PluginRegistrationSummary` logs expected providers on startup.
|
||||
- [ ] Revocation manifest export (`dotnet run --project src/Authority/StellaOps.Authority`) succeeds.
|
||||
- [ ] Monitoring dashboards show metrics resuming (see OPS5 deliverables).
|
||||
|
||||
|
||||
@@ -1,174 +1,174 @@
|
||||
{
|
||||
"title": "StellaOps Authority - Token & Access Monitoring",
|
||||
"uid": "authority-token-monitoring",
|
||||
"schemaVersion": 38,
|
||||
"version": 1,
|
||||
"editable": true,
|
||||
"timezone": "",
|
||||
"graphTooltip": 0,
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "datasource",
|
||||
"type": "datasource",
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"hide": 0,
|
||||
"current": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Token Requests – Success vs Failure",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "req/s",
|
||||
"displayName": "{{grant_type}} ({{status}})"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (grant_type, status) (rate(http_server_duration_seconds_count{service_name=\"stellaops-authority\", http_route=\"/token\"}[5m]))",
|
||||
"legendFormat": "{{grant_type}} {{status}}"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Rate Limiter Rejections",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "req/s",
|
||||
"displayName": "{{limiter}}"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (limiter) (rate(aspnetcore_rate_limiting_rejections_total{service_name=\"stellaops-authority\"}[5m]))",
|
||||
"legendFormat": "{{limiter}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Bypass Events (5m)",
|
||||
"type": "stat",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 1 },
|
||||
{ "color": "red", "value": 5 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(log_messages_total{message_template=\"Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}.\"}[5m]))"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["last"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "horizontal",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Lockout Events (15m)",
|
||||
"type": "stat",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 5 },
|
||||
{ "color": "red", "value": 10 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(log_messages_total{message_template=\"Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter}).\"}[15m]))"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["last"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "horizontal",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Trace Explorer Shortcut",
|
||||
"type": "text",
|
||||
"options": {
|
||||
"mode": "markdown",
|
||||
"content": "[Open Trace Explorer](#/explore?left={\"datasource\":\"tempo\",\"queries\":[{\"query\":\"{service.name=\\\"stellaops-authority\\\", span_name=~\\\"authority.token.*\\\"}\",\"refId\":\"A\"}]})"
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
{
|
||||
"title": "StellaOps Authority - Token & Access Monitoring",
|
||||
"uid": "authority-token-monitoring",
|
||||
"schemaVersion": 38,
|
||||
"version": 1,
|
||||
"editable": true,
|
||||
"timezone": "",
|
||||
"graphTooltip": 0,
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "datasource",
|
||||
"type": "datasource",
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"hide": 0,
|
||||
"current": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Token Requests – Success vs Failure",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "req/s",
|
||||
"displayName": "{{grant_type}} ({{status}})"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (grant_type, status) (rate(http_server_duration_seconds_count{service_name=\"stellaops-authority\", http_route=\"/token\"}[5m]))",
|
||||
"legendFormat": "{{grant_type}} {{status}}"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Rate Limiter Rejections",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "req/s",
|
||||
"displayName": "{{limiter}}"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (limiter) (rate(aspnetcore_rate_limiting_rejections_total{service_name=\"stellaops-authority\"}[5m]))",
|
||||
"legendFormat": "{{limiter}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Bypass Events (5m)",
|
||||
"type": "stat",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 1 },
|
||||
{ "color": "red", "value": 5 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(log_messages_total{message_template=\"Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}.\"}[5m]))"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["last"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "horizontal",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Lockout Events (15m)",
|
||||
"type": "stat",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 5 },
|
||||
{ "color": "red", "value": 10 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(log_messages_total{message_template=\"Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter}).\"}[15m]))"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["last"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "horizontal",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Trace Explorer Shortcut",
|
||||
"type": "text",
|
||||
"options": {
|
||||
"mode": "markdown",
|
||||
"content": "[Open Trace Explorer](#/explore?left={\"datasource\":\"tempo\",\"queries\":[{\"query\":\"{service.name=\\\"stellaops-authority\\\", span_name=~\\\"authority.token.*\\\"}\",\"refId\":\"A\"}]})"
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
|
||||
@@ -1,94 +1,94 @@
|
||||
# Authority Signing Key Rotation Playbook
|
||||
|
||||
> **Status:** Authored 2025-10-12 as part of OPS3.KEY-ROTATION rollout.
|
||||
> Use together with `docs/11_AUTHORITY.md` (Authority service guide) and the automation shipped under `ops/authority/`.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Authority publishes JWKS and revocation bundles signed with ES256 keys. To rotate those keys without downtime we now provide:
|
||||
|
||||
- **Automation script:** `ops/authority/key-rotation.sh`
|
||||
Shell helper that POSTS to `/internal/signing/rotate`, supports metadata, dry-run, and confirms JWKS afterwards.
|
||||
- **CI workflow:** `.gitea/workflows/authority-key-rotation.yml`
|
||||
Manual dispatch workflow that pulls environment-specific secrets, runs the script, and records the result. Works across staging/production by passing the `environment` input.
|
||||
|
||||
This playbook documents the repeatable sequence for all environments.
|
||||
|
||||
## 2. Pre-requisites
|
||||
|
||||
1. **Generate a new PEM key (per environment)**
|
||||
```bash
|
||||
openssl ecparam -name prime256v1 -genkey -noout \
|
||||
-out certificates/authority-signing-<env>-<year>.pem
|
||||
chmod 600 certificates/authority-signing-<env>-<year>.pem
|
||||
```
|
||||
2. **Stash the previous key** under the same volume so it can be referenced in `signing.additionalKeys` after rotation.
|
||||
3. **Ensure secrets/vars exist in Gitea**
|
||||
- `<ENV>_AUTHORITY_BOOTSTRAP_KEY`
|
||||
- `<ENV>_AUTHORITY_URL`
|
||||
- Optional shared defaults `AUTHORITY_BOOTSTRAP_KEY`, `AUTHORITY_URL`.
|
||||
|
||||
## 3. Executing the rotation
|
||||
|
||||
### Option A – via CI workflow (recommended)
|
||||
|
||||
1. Navigate to **Actions → Authority Key Rotation**.
|
||||
2. Provide inputs:
|
||||
- `environment`: `staging`, `production`, etc.
|
||||
- `key_id`: new `kid` (e.g. `authority-signing-2025-dev`).
|
||||
- `key_path`: path as seen by the Authority service (e.g. `../certificates/authority-signing-2025-dev.pem`).
|
||||
- Optional `metadata`: comma-separated `key=value` pairs (for audit trails).
|
||||
3. Trigger. The workflow:
|
||||
- Reads the bootstrap key/URL from secrets.
|
||||
- Runs `ops/authority/key-rotation.sh`.
|
||||
- Prints the JWKS response for verification.
|
||||
|
||||
### Option B – manual shell invocation
|
||||
|
||||
```bash
|
||||
AUTHORITY_BOOTSTRAP_KEY=$(cat /secure/authority-bootstrap.key) \
|
||||
./ops/authority/key-rotation.sh \
|
||||
--authority-url https://authority.example.com \
|
||||
--key-id authority-signing-2025-dev \
|
||||
--key-path ../certificates/authority-signing-2025-dev.pem \
|
||||
--meta rotatedBy=ops --meta changeTicket=OPS-1234
|
||||
```
|
||||
|
||||
Use `--dry-run` to inspect the payload before execution.
|
||||
|
||||
## 4. Post-rotation checklist
|
||||
|
||||
1. Update `authority.yaml` (or environment-specific overrides):
|
||||
- Set `signing.activeKeyId` to the new key.
|
||||
- Set `signing.keyPath` to the new PEM.
|
||||
- Append the previous key into `signing.additionalKeys`.
|
||||
- Ensure `keySource`/`provider` match the values passed to the script.
|
||||
2. Run `stellaops-cli auth revoke export` so revocation bundles are re-signed with the new key.
|
||||
3. Confirm `/jwks` lists the new `kid` with `status: "active"` and the previous one as `retired`.
|
||||
4. Archive the old key securely; keep it available until all tokens/bundles signed with it have expired.
|
||||
|
||||
## 5. Development key state
|
||||
|
||||
For the sample configuration (`etc/authority.yaml.sample`) we minted a placeholder dev key:
|
||||
|
||||
- Active: `authority-signing-2025-dev` (`certificates/authority-signing-2025-dev.pem`)
|
||||
- Retired: `authority-signing-dev`
|
||||
|
||||
Treat these as examples; real environments must maintain their own PEM material.
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/11_AUTHORITY.md` – Architecture and rotation SOP (Section 5).
|
||||
- `docs/modules/authority/operations/backup-restore.md` – Recovery flow referencing this playbook.
|
||||
- `ops/authority/README.md` – CLI usage and examples.
|
||||
- `scripts/rotate-policy-cli-secret.sh` – Helper to mint new `policy-cli` shared secrets when policy scope bundles change.
|
||||
|
||||
## 7. Appendix — Policy CLI secret rotation
|
||||
|
||||
Scope migrations such as AUTH-POLICY-23-004 require issuing fresh credentials for the `policy-cli` client. Use the helper script committed with the repo to keep secrets deterministic across environments.
|
||||
|
||||
```bash
|
||||
./scripts/rotate-policy-cli-secret.sh --output etc/secrets/policy-cli.secret
|
||||
```
|
||||
|
||||
The script writes a timestamped header and a random secret into the target file. Use `--dry-run` when generating material for external secret stores. After updating secrets in staging/production, recycle the Authority pods and confirm the new client credentials work before the next release freeze.
|
||||
# Authority Signing Key Rotation Playbook
|
||||
|
||||
> **Status:** Authored 2025-10-12 as part of OPS3.KEY-ROTATION rollout.
|
||||
> Use together with `docs/11_AUTHORITY.md` (Authority service guide) and the automation shipped under `ops/authority/`.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Authority publishes JWKS and revocation bundles signed with ES256 keys. To rotate those keys without downtime we now provide:
|
||||
|
||||
- **Automation script:** `ops/authority/key-rotation.sh`
|
||||
Shell helper that POSTS to `/internal/signing/rotate`, supports metadata, dry-run, and confirms JWKS afterwards.
|
||||
- **CI workflow:** `.gitea/workflows/authority-key-rotation.yml`
|
||||
Manual dispatch workflow that pulls environment-specific secrets, runs the script, and records the result. Works across staging/production by passing the `environment` input.
|
||||
|
||||
This playbook documents the repeatable sequence for all environments.
|
||||
|
||||
## 2. Pre-requisites
|
||||
|
||||
1. **Generate a new PEM key (per environment)**
|
||||
```bash
|
||||
openssl ecparam -name prime256v1 -genkey -noout \
|
||||
-out certificates/authority-signing-<env>-<year>.pem
|
||||
chmod 600 certificates/authority-signing-<env>-<year>.pem
|
||||
```
|
||||
2. **Stash the previous key** under the same volume so it can be referenced in `signing.additionalKeys` after rotation.
|
||||
3. **Ensure secrets/vars exist in Gitea**
|
||||
- `<ENV>_AUTHORITY_BOOTSTRAP_KEY`
|
||||
- `<ENV>_AUTHORITY_URL`
|
||||
- Optional shared defaults `AUTHORITY_BOOTSTRAP_KEY`, `AUTHORITY_URL`.
|
||||
|
||||
## 3. Executing the rotation
|
||||
|
||||
### Option A – via CI workflow (recommended)
|
||||
|
||||
1. Navigate to **Actions → Authority Key Rotation**.
|
||||
2. Provide inputs:
|
||||
- `environment`: `staging`, `production`, etc.
|
||||
- `key_id`: new `kid` (e.g. `authority-signing-2025-dev`).
|
||||
- `key_path`: path as seen by the Authority service (e.g. `../certificates/authority-signing-2025-dev.pem`).
|
||||
- Optional `metadata`: comma-separated `key=value` pairs (for audit trails).
|
||||
3. Trigger. The workflow:
|
||||
- Reads the bootstrap key/URL from secrets.
|
||||
- Runs `ops/authority/key-rotation.sh`.
|
||||
- Prints the JWKS response for verification.
|
||||
|
||||
### Option B – manual shell invocation
|
||||
|
||||
```bash
|
||||
AUTHORITY_BOOTSTRAP_KEY=$(cat /secure/authority-bootstrap.key) \
|
||||
./ops/authority/key-rotation.sh \
|
||||
--authority-url https://authority.example.com \
|
||||
--key-id authority-signing-2025-dev \
|
||||
--key-path ../certificates/authority-signing-2025-dev.pem \
|
||||
--meta rotatedBy=ops --meta changeTicket=OPS-1234
|
||||
```
|
||||
|
||||
Use `--dry-run` to inspect the payload before execution.
|
||||
|
||||
## 4. Post-rotation checklist
|
||||
|
||||
1. Update `authority.yaml` (or environment-specific overrides):
|
||||
- Set `signing.activeKeyId` to the new key.
|
||||
- Set `signing.keyPath` to the new PEM.
|
||||
- Append the previous key into `signing.additionalKeys`.
|
||||
- Ensure `keySource`/`provider` match the values passed to the script.
|
||||
2. Run `stellaops-cli auth revoke export` so revocation bundles are re-signed with the new key.
|
||||
3. Confirm `/jwks` lists the new `kid` with `status: "active"` and the previous one as `retired`.
|
||||
4. Archive the old key securely; keep it available until all tokens/bundles signed with it have expired.
|
||||
|
||||
## 5. Development key state
|
||||
|
||||
For the sample configuration (`etc/authority.yaml.sample`) we minted a placeholder dev key:
|
||||
|
||||
- Active: `authority-signing-2025-dev` (`certificates/authority-signing-2025-dev.pem`)
|
||||
- Retired: `authority-signing-dev`
|
||||
|
||||
Treat these as examples; real environments must maintain their own PEM material.
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/11_AUTHORITY.md` – Architecture and rotation SOP (Section 5).
|
||||
- `docs/modules/authority/operations/backup-restore.md` – Recovery flow referencing this playbook.
|
||||
- `ops/authority/README.md` – CLI usage and examples.
|
||||
- `scripts/rotate-policy-cli-secret.sh` – Helper to mint new `policy-cli` shared secrets when policy scope bundles change.
|
||||
|
||||
## 7. Appendix — Policy CLI secret rotation
|
||||
|
||||
Scope migrations such as AUTH-POLICY-23-004 require issuing fresh credentials for the `policy-cli` client. Use the helper script committed with the repo to keep secrets deterministic across environments.
|
||||
|
||||
```bash
|
||||
./scripts/rotate-policy-cli-secret.sh --output etc/secrets/policy-cli.secret
|
||||
```
|
||||
|
||||
The script writes a timestamped header and a random secret into the target file. Use `--dry-run` when generating material for external secret stores. After updating secrets in staging/production, recycle the Authority pods and confirm the new client credentials work before the next release freeze.
|
||||
|
||||
@@ -1,83 +1,83 @@
|
||||
# Authority Monitoring & Alerting Playbook
|
||||
|
||||
## Telemetry Sources
|
||||
- **Traces:** Activity source `StellaOps.Authority` emits spans for every token flow (`authority.token.validate_*`, `authority.token.handle_*`, `authority.token.validate_access`). Key tags include `authority.endpoint`, `authority.grant_type`, `authority.username`, `authority.client_id`, and `authority.identity_provider`.
|
||||
- **Metrics:** OpenTelemetry instrumentation (`AddAspNetCoreInstrumentation`, `AddHttpClientInstrumentation`, custom meter `StellaOps.Authority`) exports:
|
||||
- `http.server.request.duration` histogram (`http_route`, `http_status_code`, `authority.endpoint` tag via `aspnetcore` enrichment).
|
||||
- `process.runtime.gc.*`, `process.runtime.dotnet.*` (from `AddRuntimeInstrumentation`).
|
||||
- **Logs:** Serilog writes structured events to stdout. Notable templates:
|
||||
- `"Password grant verification failed ..."` and `"Plugin {PluginName} denied access ... due to lockout"` (lockout spike detector).
|
||||
- `"Password grant validation failed for {Username}: provider '{Provider}' does not support MFA required for exception approvals."` (identifies users attempting `exceptions:approve` without MFA support; tie to fresh-auth errors).
|
||||
- `"Client credentials validation failed for {ClientId}: exception scopes require tenant assignment."` (signals misconfigured exception service identities).
|
||||
- `"Granting StellaOps bypass for remote {RemoteIp}"` (bypass usage).
|
||||
- `"Rate limit exceeded for path {Path} from {RemoteIp}"` (limiter alerts).
|
||||
|
||||
## Prometheus Metrics to Collect
|
||||
| Metric | Query | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `token_requests_total` | `sum by (grant_type, status) (rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token"}[5m]))` | Token issuance volume per grant type (`grant_type` comes via `authority.grant_type` span attribute → Exemplars in Grafana). |
|
||||
| `token_failure_ratio` | `sum(rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token", http_status_code=~"4..|5.."}[5m])) / sum(rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token"}[5m]))` | Alert when > 5 % for 10 min. |
|
||||
| `authorize_rate_limit_hits` | `sum(rate(aspnetcore_rate_limiting_rejections_total{service_name="stellaops-authority", limiter="authority-token"}[5m]))` | Detect rate limiting saturations (requires OTEL ASP.NET rate limiter exporter). |
|
||||
| `lockout_events` | `sum by (plugin) (rate(log_messages_total{app="stellaops-authority", level="Warning", message_template="Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter})."}[5m]))` | Derived from Loki/Promtail log counter. |
|
||||
| `bypass_usage_total` | `sum(rate(log_messages_total{app="stellaops-authority", level="Information", message_template="Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}."}[5m]))` | Track trusted bypass invocations. |
|
||||
|
||||
> **Exporter note:** Enable `aspnetcore` meters (`dotnet-counters` name `Microsoft.AspNetCore.Hosting`), or configure the OpenTelemetry Collector `metrics` pipeline with `metric_statements` to remap histogram counts into the shown series.
|
||||
|
||||
## Alert Rules
|
||||
1. **Token Failure Surge**
|
||||
- _Expression_: `token_failure_ratio > 0.05`
|
||||
- _For_: `10m`
|
||||
- _Labels_: `severity="critical"`
|
||||
- _Annotations_: Include `topk(5, sum by (authority_identity_provider) (increase(authority_token_rejections_total[10m])))` as diagnostic hint (requires span → metric transformation).
|
||||
2. **Lockout Spike**
|
||||
- _Expression_: `sum(rate(log_messages_total{message_template="Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter})."}[15m])) > 10`
|
||||
- _For_: `15m`
|
||||
- Investigate credential stuffing; consider temporarily tightening `RateLimiting.Token`.
|
||||
3. **Bypass Threshold**
|
||||
- _Expression_: `sum(rate(log_messages_total{message_template="Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}."}[5m])) > 1`
|
||||
- _For_: `5m`
|
||||
- Alert severity `warning` — verify the calling host list.
|
||||
4. **Rate Limiter Saturation**
|
||||
- _Expression_: `sum(rate(aspnetcore_rate_limiting_rejections_total{service_name="stellaops-authority"}[5m])) > 0`
|
||||
- Escalate if sustained for 5 min; confirm trusted clients aren’t misconfigured.
|
||||
|
||||
## Grafana Dashboard
|
||||
- Import `docs/modules/authority/operations/grafana-dashboard.json` to provision baseline panels:
|
||||
- **Token Success vs Failure** – stacked rate visualization split by grant type.
|
||||
- **Rate Limiter Hits** – bar chart showing `authority-token` and `authority-authorize`.
|
||||
- **Bypass & Lockout Events** – dual-stat panel using Loki-derived counters.
|
||||
- **Trace Explorer Link** – panel links to `StellaOps.Authority` span search pre-filtered by `authority.grant_type`.
|
||||
|
||||
## Collector Configuration Snippets
|
||||
```yaml
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:9464"
|
||||
processors:
|
||||
batch:
|
||||
attributes/token_grant:
|
||||
actions:
|
||||
- key: grant_type
|
||||
action: upsert
|
||||
from_attribute: authority.grant_type
|
||||
service:
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [attributes/token_grant, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [batch]
|
||||
exporters: [loki]
|
||||
```
|
||||
|
||||
## Operational Checklist
|
||||
- [ ] Confirm `STELLAOPS_AUTHORITY__OBSERVABILITY__EXPORTERS` enables OTLP in production builds.
|
||||
- [ ] Ensure Promtail captures container stdout with Serilog structured formatting.
|
||||
- [ ] Periodically validate alert noise by running load tests that trigger the rate limiter.
|
||||
- [ ] Include dashboard JSON in Offline Kit for air-gapped clusters; update version header when metrics change.
|
||||
# Authority Monitoring & Alerting Playbook
|
||||
|
||||
## Telemetry Sources
|
||||
- **Traces:** Activity source `StellaOps.Authority` emits spans for every token flow (`authority.token.validate_*`, `authority.token.handle_*`, `authority.token.validate_access`). Key tags include `authority.endpoint`, `authority.grant_type`, `authority.username`, `authority.client_id`, and `authority.identity_provider`.
|
||||
- **Metrics:** OpenTelemetry instrumentation (`AddAspNetCoreInstrumentation`, `AddHttpClientInstrumentation`, custom meter `StellaOps.Authority`) exports:
|
||||
- `http.server.request.duration` histogram (`http_route`, `http_status_code`, `authority.endpoint` tag via `aspnetcore` enrichment).
|
||||
- `process.runtime.gc.*`, `process.runtime.dotnet.*` (from `AddRuntimeInstrumentation`).
|
||||
- **Logs:** Serilog writes structured events to stdout. Notable templates:
|
||||
- `"Password grant verification failed ..."` and `"Plugin {PluginName} denied access ... due to lockout"` (lockout spike detector).
|
||||
- `"Password grant validation failed for {Username}: provider '{Provider}' does not support MFA required for exception approvals."` (identifies users attempting `exceptions:approve` without MFA support; tie to fresh-auth errors).
|
||||
- `"Client credentials validation failed for {ClientId}: exception scopes require tenant assignment."` (signals misconfigured exception service identities).
|
||||
- `"Granting StellaOps bypass for remote {RemoteIp}"` (bypass usage).
|
||||
- `"Rate limit exceeded for path {Path} from {RemoteIp}"` (limiter alerts).
|
||||
|
||||
## Prometheus Metrics to Collect
|
||||
| Metric | Query | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `token_requests_total` | `sum by (grant_type, status) (rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token"}[5m]))` | Token issuance volume per grant type (`grant_type` comes via `authority.grant_type` span attribute → Exemplars in Grafana). |
|
||||
| `token_failure_ratio` | `sum(rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token", http_status_code=~"4..|5.."}[5m])) / sum(rate(http_server_duration_seconds_count{service_name="stellaops-authority", http_route="/token"}[5m]))` | Alert when > 5 % for 10 min. |
|
||||
| `authorize_rate_limit_hits` | `sum(rate(aspnetcore_rate_limiting_rejections_total{service_name="stellaops-authority", limiter="authority-token"}[5m]))` | Detect rate limiting saturations (requires OTEL ASP.NET rate limiter exporter). |
|
||||
| `lockout_events` | `sum by (plugin) (rate(log_messages_total{app="stellaops-authority", level="Warning", message_template="Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter})."}[5m]))` | Derived from Loki/Promtail log counter. |
|
||||
| `bypass_usage_total` | `sum(rate(log_messages_total{app="stellaops-authority", level="Information", message_template="Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}."}[5m]))` | Track trusted bypass invocations. |
|
||||
|
||||
> **Exporter note:** Enable `aspnetcore` meters (`dotnet-counters` name `Microsoft.AspNetCore.Hosting`), or configure the OpenTelemetry Collector `metrics` pipeline with `metric_statements` to remap histogram counts into the shown series.
|
||||
|
||||
## Alert Rules
|
||||
1. **Token Failure Surge**
|
||||
- _Expression_: `token_failure_ratio > 0.05`
|
||||
- _For_: `10m`
|
||||
- _Labels_: `severity="critical"`
|
||||
- _Annotations_: Include `topk(5, sum by (authority_identity_provider) (increase(authority_token_rejections_total[10m])))` as diagnostic hint (requires span → metric transformation).
|
||||
2. **Lockout Spike**
|
||||
- _Expression_: `sum(rate(log_messages_total{message_template="Plugin {PluginName} denied access for {Username} due to lockout (retry after {RetryAfter})."}[15m])) > 10`
|
||||
- _For_: `15m`
|
||||
- Investigate credential stuffing; consider temporarily tightening `RateLimiting.Token`.
|
||||
3. **Bypass Threshold**
|
||||
- _Expression_: `sum(rate(log_messages_total{message_template="Granting StellaOps bypass for remote {RemoteIp}; required scopes {RequiredScopes}."}[5m])) > 1`
|
||||
- _For_: `5m`
|
||||
- Alert severity `warning` — verify the calling host list.
|
||||
4. **Rate Limiter Saturation**
|
||||
- _Expression_: `sum(rate(aspnetcore_rate_limiting_rejections_total{service_name="stellaops-authority"}[5m])) > 0`
|
||||
- Escalate if sustained for 5 min; confirm trusted clients aren’t misconfigured.
|
||||
|
||||
## Grafana Dashboard
|
||||
- Import `docs/modules/authority/operations/grafana-dashboard.json` to provision baseline panels:
|
||||
- **Token Success vs Failure** – stacked rate visualization split by grant type.
|
||||
- **Rate Limiter Hits** – bar chart showing `authority-token` and `authority-authorize`.
|
||||
- **Bypass & Lockout Events** – dual-stat panel using Loki-derived counters.
|
||||
- **Trace Explorer Link** – panel links to `StellaOps.Authority` span search pre-filtered by `authority.grant_type`.
|
||||
|
||||
## Collector Configuration Snippets
|
||||
```yaml
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:9464"
|
||||
processors:
|
||||
batch:
|
||||
attributes/token_grant:
|
||||
actions:
|
||||
- key: grant_type
|
||||
action: upsert
|
||||
from_attribute: authority.grant_type
|
||||
service:
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [attributes/token_grant, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [batch]
|
||||
exporters: [loki]
|
||||
```
|
||||
|
||||
## Operational Checklist
|
||||
- [ ] Confirm `STELLAOPS_AUTHORITY__OBSERVABILITY__EXPORTERS` enables OTLP in production builds.
|
||||
- [ ] Ensure Promtail captures container stdout with Serilog structured formatting.
|
||||
- [ ] Periodically validate alert noise by running load tests that trigger the rate limiter.
|
||||
- [ ] Include dashboard JSON in Offline Kit for air-gapped clusters; update version header when metrics change.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# CI Recipes agent guide
|
||||
|
||||
## Mission
|
||||
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# CI Recipes agent guide
|
||||
|
||||
## Mission
|
||||
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,29 +1,29 @@
|
||||
# StellaOps CI Recipes
|
||||
|
||||
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
|
||||
|
||||
## Responsibilities
|
||||
- Provide ready-to-use pipeline snippets for ingestion, scanning, policy evaluation, and exports.
|
||||
- Document required secrets/scopes and deterministic build knobs.
|
||||
- Highlight offline-compatible workflows and cache strategies.
|
||||
|
||||
## Key components
|
||||
- Recipe catalogue in ./recipes.md.
|
||||
|
||||
## Integrations & dependencies
|
||||
- DevOps release workflows.
|
||||
- Module-specific test suites referenced in recipes.
|
||||
|
||||
## Operational notes
|
||||
- Encourage reuse through templated YAML/JSON fragments.
|
||||
|
||||
## Related resources
|
||||
- ./recipes.md
|
||||
|
||||
## Backlog references
|
||||
- CI recipes refresh tracked in ../../TASKS.md under DOCS-CI stories.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** bake ingestion/verifier guardrails into CI recipes.
|
||||
- **Epic 10 – Export Center:** provide pipeline snippets for export packaging, signing, and Offline Kit publication.
|
||||
- **Epic 11 – Notifications Studio:** offer CI hooks for notification previews/tests where relevant.
|
||||
# StellaOps CI Recipes
|
||||
|
||||
CI module collects reproducible pipeline recipes for builds, tests, and release promotion across supported platforms.
|
||||
|
||||
## Responsibilities
|
||||
- Provide ready-to-use pipeline snippets for ingestion, scanning, policy evaluation, and exports.
|
||||
- Document required secrets/scopes and deterministic build knobs.
|
||||
- Highlight offline-compatible workflows and cache strategies.
|
||||
|
||||
## Key components
|
||||
- Recipe catalogue in ./recipes.md.
|
||||
|
||||
## Integrations & dependencies
|
||||
- DevOps release workflows.
|
||||
- Module-specific test suites referenced in recipes.
|
||||
|
||||
## Operational notes
|
||||
- Encourage reuse through templated YAML/JSON fragments.
|
||||
|
||||
## Related resources
|
||||
- ./recipes.md
|
||||
|
||||
## Backlog references
|
||||
- CI recipes refresh tracked in ../../TASKS.md under DOCS-CI stories.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** bake ingestion/verifier guardrails into CI recipes.
|
||||
- **Epic 10 – Export Center:** provide pipeline snippets for export packaging, signing, and Offline Kit publication.
|
||||
- **Epic 11 – Notifications Studio:** offer CI hooks for notification previews/tests where relevant.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — CI Recipes
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CI RECIPES-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CI RECIPES-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CI RECIPES-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — CI Recipes
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CI RECIPES-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CI RECIPES-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CI RECIPES-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# CI Recipes architecture
|
||||
|
||||
> Reference the AOC guardrails, export workflows, and notification patterns documented in the Authority, Export Center, and Notify module guides when designing CI templates.
|
||||
|
||||
This placeholder summarises the planned architecture for CI Recipes. Consolidate design details from implementation plans and upcoming epics before coding.
|
||||
|
||||
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.
|
||||
# CI Recipes architecture
|
||||
|
||||
> Reference the AOC guardrails, export workflows, and notification patterns documented in the Authority, Export Center, and Notify module guides when designing CI templates.
|
||||
|
||||
This placeholder summarises the planned architecture for CI Recipes. Consolidate design details from implementation plans and upcoming epics before coding.
|
||||
|
||||
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
# Implementation plan — CI Recipes
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** ensure pipelines enforce schemas, provenance, and verifier jobs.
|
||||
- **Epic 10 – Export Center:** add export/signing/Offline Kit automation templates.
|
||||
- **Epic 11 – Notifications Studio:** document CI hooks for notification previews/tests.
|
||||
- Track DOCS-CI stories in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — CI Recipes
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** ensure pipelines enforce schemas, provenance, and verifier jobs.
|
||||
- **Epic 10 – Export Center:** add export/signing/Offline Kit automation templates.
|
||||
- **Epic 11 – Notifications Studio:** document CI hooks for notification previews/tests.
|
||||
- Track DOCS-CI stories in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,353 +1,353 @@
|
||||
# Stella Ops CI Recipes — (2025‑08‑04)
|
||||
|
||||
## 0 · Key variables (export these once)
|
||||
|
||||
| Variable | Meaning | Typical value |
|
||||
| ------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
|
||||
| `STELLA_URL` | Host that: ① stores the **CLI** & **SBOM‑builder** images under `/registry` **and** ② receives API calls at `https://$STELLA_URL` | `stella-ops.ci.acme.example` |
|
||||
| `DOCKER_HOST` | How containers reach your Docker daemon (because we no longer mount `/var/run/docker.sock`) | `tcp://docker:2375` |
|
||||
| `WORKSPACE` | Directory where the pipeline stores artefacts (SBOM file) | `$(pwd)` |
|
||||
| `IMAGE` | The image you are building & scanning | `acme/backend:sha-${COMMIT_SHA}` |
|
||||
| `SBOM_FILE` | Immutable SBOM name – `<image-ref>‑YYYYMMDDThhmmssZ.sbom.json` | `acme_backend_sha‑abc123‑20250804T153050Z.sbom.json` |
|
||||
|
||||
> **Authority graph scopes note (2025-10-27):** CI stages that spin up the Authority compose profile now rely on the checked-in `etc/authority.yaml`. Before running integration smoke jobs, inject real secrets for every `etc/secrets/*.secret` file (Cartographer, Graph API, Policy Engine, Concelier, Excititor). The repository defaults contain `*-change-me` placeholders and Authority will reject tokens if those secrets are not overridden. Reissue CI tokens that previously used `policy:write`/`policy:submit`/`policy:edit` scopes—new bundles must request `policy:read`, `policy:author`, `policy:review`, `policy:simulate`, and (`policy:approve`/`policy:operate`/`policy:activate` when pipelines promote policies).
|
||||
|
||||
```bash
|
||||
export STELLA_URL="stella-ops.ci.acme.example"
|
||||
export DOCKER_HOST="tcp://docker:2375" # Jenkins/Circle often expose it like this
|
||||
export WORKSPACE="$(pwd)"
|
||||
export IMAGE="acme/backend:sha-${COMMIT_SHA}"
|
||||
export SBOM_FILE="$(echo "${IMAGE}" | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1 · SBOM creation strategies
|
||||
|
||||
### Option A – **Buildx attested SBOM** (preferred if you can use BuildKit)
|
||||
|
||||
You pass **two build args** so the Dockerfile can run the builder and copy the result out of the build context.
|
||||
|
||||
```bash
|
||||
docker buildx build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--provenance=true --sbom=true \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
```
|
||||
|
||||
**If you **cannot** use Buildx, use Option B below.** The older “run a builder stage inside the Dockerfile” pattern is unreliable for producing an SBOM of the final image.
|
||||
|
||||
```Dockerfile
|
||||
|
||||
ARG STELLA_SBOM_BUILDER
|
||||
ARG SBOM_FILE
|
||||
|
||||
FROM $STELLA_SBOM_BUILDER as sbom
|
||||
ARG IMAGE
|
||||
ARG SBOM_FILE
|
||||
RUN $STELLA_SBOM_BUILDER build --image $IMAGE --output /out/$SBOM_FILE
|
||||
|
||||
# ---- actual build stages … ----
|
||||
FROM alpine:3.20
|
||||
COPY --from=sbom /out/$SBOM_FILE / # (optional) keep or discard
|
||||
|
||||
# (rest of your Dockerfile)
|
||||
```
|
||||
|
||||
### Option B – **External builder step** (works everywhere; recommended baseline if Buildx isn’t available)
|
||||
|
||||
*(keep this block if your pipeline already has an image‑build step that you can’t modify)*
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-e DOCKER_HOST="$DOCKER_HOST" \ # let builder reach the daemon remotely
|
||||
-v "$WORKSPACE:/workspace" \ # place SBOM beside the source code
|
||||
"$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Scan the image & upload results
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-e DOCKER_HOST="$DOCKER_HOST" \ # remote‑daemon pointer
|
||||
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \ # mount SBOM under same name at container root
|
||||
-e STELLA_OPS_URL="https://${STELLA_URL}" \ # where the CLI posts findings
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
```
|
||||
|
||||
The CLI returns **exit 0** if policies pass, **>0** if blocked — perfect for failing the job.
|
||||
|
||||
---
|
||||
|
||||
## 3 · CI templates
|
||||
|
||||
Below are minimal, cut‑and‑paste snippets.
|
||||
**Feel free to delete Option B** if you adopt Option A.
|
||||
|
||||
### 3.1 Jenkins (Declarative Pipeline)
|
||||
|
||||
```groovy
|
||||
pipeline {
|
||||
agent { docker { image 'docker:25' args '--privileged' } } // gives us /usr/bin/docker
|
||||
environment {
|
||||
STELLA_URL = 'stella-ops.ci.acme.example'
|
||||
DOCKER_HOST = 'tcp://docker:2375'
|
||||
IMAGE = "acme/backend:${env.BUILD_NUMBER}"
|
||||
SBOM_FILE = "acme_backend_${env.BUILD_NUMBER}-${new Date().format('yyyyMMdd\'T\'HHmmss\'Z\'', TimeZone.getTimeZone('UTC'))}.sbom.json"
|
||||
}
|
||||
stages {
|
||||
stage('Build image + SBOM (Option A)') {
|
||||
steps {
|
||||
sh '''
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
'''
|
||||
}
|
||||
}
|
||||
/* ---------- Option B fallback (when you must keep the existing build step as‑is) ----------
|
||||
stage('SBOM builder (Option B)') {
|
||||
steps {
|
||||
sh '''
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$WORKSPACE:/workspace" \
|
||||
"$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
'''
|
||||
}
|
||||
}
|
||||
------------------------------------------------------------------------------------------ */
|
||||
stage('Scan & upload') {
|
||||
steps {
|
||||
sh '''
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \
|
||||
-e STELLA_OPS_URL="https://$STELLA_URL" \
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
'''
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.2 CircleCI `.circleci/config.yml`
|
||||
|
||||
```yaml
|
||||
version: 2.1
|
||||
jobs:
|
||||
stella_scan:
|
||||
docker:
|
||||
- image: cimg/base:stable # baremetal image with Docker CLI
|
||||
environment:
|
||||
STELLA_URL: stella-ops.ci.acme.example
|
||||
DOCKER_HOST: tcp://docker:2375 # Circle’s “remote Docker” socket
|
||||
steps:
|
||||
- checkout
|
||||
|
||||
- run:
|
||||
name: Compute vars
|
||||
command: |
|
||||
echo 'export IMAGE="acme/backend:${CIRCLE_SHA1}"' >> $BASH_ENV
|
||||
echo 'export SBOM_FILE="$(echo acme/backend:${CIRCLE_SHA1} | tr "/:+" "__")-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"' >> $BASH_ENV
|
||||
- run:
|
||||
name: Build image + SBOM (Option A)
|
||||
command: |
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
# --- Option B fallback (when you must keep the existing build step as‑is) ---
|
||||
#- run:
|
||||
# name: SBOM builder (Option B)
|
||||
# command: |
|
||||
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
# -v "$PWD:/workspace" \
|
||||
# "$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
# build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
- run:
|
||||
name: Scan
|
||||
command: |
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$PWD/${SBOM_FILE}:/${SBOM_FILE}:ro" \
|
||||
-e STELLA_OPS_URL="https://$STELLA_URL" \
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
workflows:
|
||||
stella:
|
||||
jobs: [stella_scan]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Gitea Actions `.gitea/workflows/stella.yml`
|
||||
|
||||
*(Gitea 1.22+ ships native Actions compatible with GitHub syntax)*
|
||||
|
||||
```yaml
|
||||
name: Stella Scan
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
stella:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
STELLA_URL: ${{ secrets.STELLA_URL }}
|
||||
DOCKER_HOST: tcp://docker:2375 # provided by the docker:dind service
|
||||
services:
|
||||
docker:
|
||||
image: docker:dind
|
||||
options: >-
|
||||
--privileged
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Compute vars
|
||||
id: vars
|
||||
run: |
|
||||
echo "IMAGE=ghcr.io/${{ gitea.repository }}:${{ gitea.sha }}" >> $GITEA_OUTPUT
|
||||
echo "SBOM_FILE=$(echo ghcr.io/${{ gitea.repository }}:${{ gitea.sha }} | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json" >> $GITEA_OUTPUT
|
||||
|
||||
- name: Build image + SBOM (Option A)
|
||||
run: |
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="${STELLA_URL}/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="${{ steps.vars.outputs.SBOM_FILE }}" \
|
||||
-t "${{ steps.vars.outputs.IMAGE }}" .
|
||||
|
||||
# --- Option B fallback (when you must keep the existing build step as‑is) ---
|
||||
#- name: SBOM builder (Option B)
|
||||
# run: |
|
||||
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
# -v "$(pwd):/workspace" \
|
||||
# "${STELLA_URL}/registry/stella-sbom-builder:latest" \
|
||||
# build --image "${{ steps.vars.outputs.IMAGE }}" --output "/workspace/${{ steps.vars.outputs.SBOM_FILE }}"
|
||||
|
||||
- name: Scan
|
||||
run: |
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$(pwd)/${{ steps.vars.outputs.SBOM_FILE }}:/${{ steps.vars.outputs.SBOM_FILE }}:ro" \
|
||||
-e STELLA_OPS_URL="https://${STELLA_URL}" \
|
||||
"${STELLA_URL}/registry/stella-cli:latest" \
|
||||
scan --sbom "/${{ steps.vars.outputs.SBOM_FILE }}" "${{ steps.vars.outputs.IMAGE }}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Docs CI (Gitea Actions & Offline Mirror)
|
||||
|
||||
StellaOps ships a dedicated Docs workflow at `.gitea/workflows/docs.yml`. When mirroring the pipeline offline or running it locally, install the same toolchain so markdown linting, schema validation, and HTML preview stay deterministic.
|
||||
|
||||
### 4.1 Toolchain bootstrap
|
||||
|
||||
```bash
|
||||
# Node.js 20.x is required; install once per runner
|
||||
npm install --no-save \
|
||||
markdown-link-check \
|
||||
remark-cli \
|
||||
remark-preset-lint-recommended \
|
||||
ajv \
|
||||
ajv-cli \
|
||||
ajv-formats
|
||||
|
||||
# Python 3.11+ powers the preview renderer
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install markdown pygments
|
||||
```
|
||||
|
||||
> **No `pip` available?** Some hardened Python builds (including the repo’s `tmp/docenv`
|
||||
> interpreter) ship without `pip`/`ensurepip`. In that case download the pure‑Python
|
||||
> sdists (e.g. `Markdown-3.x.tar.gz`, `pygments-2.x.tar.gz`) and extract their
|
||||
> packages directly into the virtualenv’s `lib/python*/site-packages/` folder.
|
||||
> This keeps the renderer working even when package managers are disabled.
|
||||
|
||||
**Offline tip.** Add the packages above to your artifact mirror (for example `ops/devops/offline-kit.json`) so runners can install them via `npm --offline` / `pip --no-index`.
|
||||
|
||||
### 4.2 Schema validation step
|
||||
|
||||
Ajv compiles every event schema to guard against syntax or format regressions. The workflow uses `ajv-formats` for UUID/date-time support.
|
||||
|
||||
```bash
|
||||
for schema in docs/events/*.json; do
|
||||
npx ajv compile -c ajv-formats -s "$schema"
|
||||
done
|
||||
```
|
||||
|
||||
Run this loop before committing schema changes. For new references, append `-r additional-file.json` so CI and local runs stay aligned.
|
||||
|
||||
### 4.3 Preview build
|
||||
|
||||
```bash
|
||||
python scripts/render_docs.py --source docs --output artifacts/docs-preview --clean
|
||||
```
|
||||
|
||||
Host the resulting bundle via any static file server for review (for example `python -m http.server`).
|
||||
|
||||
### 4.4 Publishing checklist
|
||||
|
||||
- [ ] Toolchain installs succeed without hitting the public internet (mirror or cached tarballs).
|
||||
- [ ] Ajv validation passes for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`.
|
||||
- [ ] Markdown link check (`npx markdown-link-check`) reports no broken references.
|
||||
- [ ] Preview bundle archived (or attached) for stakeholders.
|
||||
|
||||
### 4.5 Policy DSL lint stage
|
||||
|
||||
Policy Engine v2 pipelines now fail fast if policy documents are malformed. After checkout and dotnet restore, run:
|
||||
|
||||
```bash
|
||||
dotnet run \
|
||||
--project src/Tools/PolicyDslValidator/PolicyDslValidator.csproj \
|
||||
-- \
|
||||
--strict docs/examples/policies/*.yaml
|
||||
```
|
||||
|
||||
- `--strict` treats warnings as errors so missing metadata doesn’t slip through.
|
||||
- The validator accepts globs, so you can point it at tenant policy directories later (`policies/**/*.yaml`).
|
||||
- Exit codes follow UNIX conventions: `0` success, `1` parse/errors, `2` warnings when `--strict` is set, `64` usage mistakes.
|
||||
|
||||
Capture the validator output as part of your build logs; Support uses it when triaging policy rollout issues.
|
||||
|
||||
### 4.6 Policy simulation smoke
|
||||
|
||||
Catch unexpected policy regressions by exercising a small set of golden SBOM findings via the simulation smoke tool:
|
||||
|
||||
```bash
|
||||
dotnet run \
|
||||
--project src/Tools/PolicySimulationSmoke/PolicySimulationSmoke.csproj \
|
||||
-- \
|
||||
--scenario-root samples/policy/simulations \
|
||||
--output artifacts/policy-simulations
|
||||
```
|
||||
|
||||
- The tool loads each `scenario.json` under `samples/policy/simulations`, evaluates the referenced policy, and fails the build if projected verdicts change.
|
||||
- In CI the command runs twice (to `run1/` and `run2/`) and `diff -u` compares the summaries—any mismatch signals a determinism regression.
|
||||
- Artifacts land in `artifacts/policy-simulations/policy-simulation-summary.json`; upload them for later inspection (see CI workflow).
|
||||
- Expand scenarios by copying real-world findings into the samples directory—ensure expected statuses are recorded so regressions trip the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting cheat‑sheet
|
||||
|
||||
| Symptom | Root cause | First things to try |
|
||||
| ------------------------------------- | --------------------------- | --------------------------------------------------------------- |
|
||||
| `no such host $STELLA_URL` | DNS typo or VPN outage | `ping $STELLA_URL` from runner |
|
||||
| `connection refused` when CLI uploads | Port 443 blocked | open firewall / check ingress |
|
||||
| `failed to stat /<sbom>.json` | SBOM wasn’t produced | Did Option A actually run builder? If not, enable Option B |
|
||||
| `registry unauthorized` | Runner lacks registry creds | `docker login $STELLA_URL/registry` (store creds in CI secrets) |
|
||||
| Non‑zero scan exit | Blocking vuln/licence | Open project in Ops UI → triage or waive |
|
||||
|
||||
---
|
||||
|
||||
### Change log
|
||||
|
||||
* **2025‑10‑18** – Documented Docs CI toolchain (Ajv validation, static preview) and offline checklist.
|
||||
* **2025‑08‑04** – Variable clean‑up, removed Docker‑socket & cache mounts, added Jenkins / CircleCI / Gitea examples, clarified Option B comment.
|
||||
# Stella Ops CI Recipes — (2025‑08‑04)
|
||||
|
||||
## 0 · Key variables (export these once)
|
||||
|
||||
| Variable | Meaning | Typical value |
|
||||
| ------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
|
||||
| `STELLA_URL` | Host that: ① stores the **CLI** & **SBOM‑builder** images under `/registry` **and** ② receives API calls at `https://$STELLA_URL` | `stella-ops.ci.acme.example` |
|
||||
| `DOCKER_HOST` | How containers reach your Docker daemon (because we no longer mount `/var/run/docker.sock`) | `tcp://docker:2375` |
|
||||
| `WORKSPACE` | Directory where the pipeline stores artefacts (SBOM file) | `$(pwd)` |
|
||||
| `IMAGE` | The image you are building & scanning | `acme/backend:sha-${COMMIT_SHA}` |
|
||||
| `SBOM_FILE` | Immutable SBOM name – `<image-ref>‑YYYYMMDDThhmmssZ.sbom.json` | `acme_backend_sha‑abc123‑20250804T153050Z.sbom.json` |
|
||||
|
||||
> **Authority graph scopes note (2025-10-27):** CI stages that spin up the Authority compose profile now rely on the checked-in `etc/authority.yaml`. Before running integration smoke jobs, inject real secrets for every `etc/secrets/*.secret` file (Cartographer, Graph API, Policy Engine, Concelier, Excititor). The repository defaults contain `*-change-me` placeholders and Authority will reject tokens if those secrets are not overridden. Reissue CI tokens that previously used `policy:write`/`policy:submit`/`policy:edit` scopes—new bundles must request `policy:read`, `policy:author`, `policy:review`, `policy:simulate`, and (`policy:approve`/`policy:operate`/`policy:activate` when pipelines promote policies).
|
||||
|
||||
```bash
|
||||
export STELLA_URL="stella-ops.ci.acme.example"
|
||||
export DOCKER_HOST="tcp://docker:2375" # Jenkins/Circle often expose it like this
|
||||
export WORKSPACE="$(pwd)"
|
||||
export IMAGE="acme/backend:sha-${COMMIT_SHA}"
|
||||
export SBOM_FILE="$(echo "${IMAGE}" | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1 · SBOM creation strategies
|
||||
|
||||
### Option A – **Buildx attested SBOM** (preferred if you can use BuildKit)
|
||||
|
||||
You pass **two build args** so the Dockerfile can run the builder and copy the result out of the build context.
|
||||
|
||||
```bash
|
||||
docker buildx build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--provenance=true --sbom=true \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
```
|
||||
|
||||
**If you **cannot** use Buildx, use Option B below.** The older “run a builder stage inside the Dockerfile” pattern is unreliable for producing an SBOM of the final image.
|
||||
|
||||
```Dockerfile
|
||||
|
||||
ARG STELLA_SBOM_BUILDER
|
||||
ARG SBOM_FILE
|
||||
|
||||
FROM $STELLA_SBOM_BUILDER as sbom
|
||||
ARG IMAGE
|
||||
ARG SBOM_FILE
|
||||
RUN $STELLA_SBOM_BUILDER build --image $IMAGE --output /out/$SBOM_FILE
|
||||
|
||||
# ---- actual build stages … ----
|
||||
FROM alpine:3.20
|
||||
COPY --from=sbom /out/$SBOM_FILE / # (optional) keep or discard
|
||||
|
||||
# (rest of your Dockerfile)
|
||||
```
|
||||
|
||||
### Option B – **External builder step** (works everywhere; recommended baseline if Buildx isn’t available)
|
||||
|
||||
*(keep this block if your pipeline already has an image‑build step that you can’t modify)*
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-e DOCKER_HOST="$DOCKER_HOST" \ # let builder reach the daemon remotely
|
||||
-v "$WORKSPACE:/workspace" \ # place SBOM beside the source code
|
||||
"$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2 · Scan the image & upload results
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-e DOCKER_HOST="$DOCKER_HOST" \ # remote‑daemon pointer
|
||||
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \ # mount SBOM under same name at container root
|
||||
-e STELLA_OPS_URL="https://${STELLA_URL}" \ # where the CLI posts findings
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
```
|
||||
|
||||
The CLI returns **exit 0** if policies pass, **>0** if blocked — perfect for failing the job.
|
||||
|
||||
---
|
||||
|
||||
## 3 · CI templates
|
||||
|
||||
Below are minimal, cut‑and‑paste snippets.
|
||||
**Feel free to delete Option B** if you adopt Option A.
|
||||
|
||||
### 3.1 Jenkins (Declarative Pipeline)
|
||||
|
||||
```groovy
|
||||
pipeline {
|
||||
agent { docker { image 'docker:25' args '--privileged' } } // gives us /usr/bin/docker
|
||||
environment {
|
||||
STELLA_URL = 'stella-ops.ci.acme.example'
|
||||
DOCKER_HOST = 'tcp://docker:2375'
|
||||
IMAGE = "acme/backend:${env.BUILD_NUMBER}"
|
||||
SBOM_FILE = "acme_backend_${env.BUILD_NUMBER}-${new Date().format('yyyyMMdd\'T\'HHmmss\'Z\'', TimeZone.getTimeZone('UTC'))}.sbom.json"
|
||||
}
|
||||
stages {
|
||||
stage('Build image + SBOM (Option A)') {
|
||||
steps {
|
||||
sh '''
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
'''
|
||||
}
|
||||
}
|
||||
/* ---------- Option B fallback (when you must keep the existing build step as‑is) ----------
|
||||
stage('SBOM builder (Option B)') {
|
||||
steps {
|
||||
sh '''
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$WORKSPACE:/workspace" \
|
||||
"$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
'''
|
||||
}
|
||||
}
|
||||
------------------------------------------------------------------------------------------ */
|
||||
stage('Scan & upload') {
|
||||
steps {
|
||||
sh '''
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$WORKSPACE/${SBOM_FILE}:/${SBOM_FILE}:ro" \
|
||||
-e STELLA_OPS_URL="https://$STELLA_URL" \
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
'''
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.2 CircleCI `.circleci/config.yml`
|
||||
|
||||
```yaml
|
||||
version: 2.1
|
||||
jobs:
|
||||
stella_scan:
|
||||
docker:
|
||||
- image: cimg/base:stable # baremetal image with Docker CLI
|
||||
environment:
|
||||
STELLA_URL: stella-ops.ci.acme.example
|
||||
DOCKER_HOST: tcp://docker:2375 # Circle’s “remote Docker” socket
|
||||
steps:
|
||||
- checkout
|
||||
|
||||
- run:
|
||||
name: Compute vars
|
||||
command: |
|
||||
echo 'export IMAGE="acme/backend:${CIRCLE_SHA1}"' >> $BASH_ENV
|
||||
echo 'export SBOM_FILE="$(echo acme/backend:${CIRCLE_SHA1} | tr "/:+" "__")-$(date -u +%Y%m%dT%H%M%SZ).sbom.json"' >> $BASH_ENV
|
||||
- run:
|
||||
name: Build image + SBOM (Option A)
|
||||
command: |
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="$SBOM_FILE" \
|
||||
-t "$IMAGE" .
|
||||
# --- Option B fallback (when you must keep the existing build step as‑is) ---
|
||||
#- run:
|
||||
# name: SBOM builder (Option B)
|
||||
# command: |
|
||||
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
# -v "$PWD:/workspace" \
|
||||
# "$STELLA_URL/registry/stella-sbom-builder:latest" \
|
||||
# build --image "$IMAGE" --output "/workspace/${SBOM_FILE}"
|
||||
- run:
|
||||
name: Scan
|
||||
command: |
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$PWD/${SBOM_FILE}:/${SBOM_FILE}:ro" \
|
||||
-e STELLA_OPS_URL="https://$STELLA_URL" \
|
||||
"$STELLA_URL/registry/stella-cli:latest" \
|
||||
scan --sbom "/${SBOM_FILE}" "$IMAGE"
|
||||
workflows:
|
||||
stella:
|
||||
jobs: [stella_scan]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Gitea Actions `.gitea/workflows/stella.yml`
|
||||
|
||||
*(Gitea 1.22+ ships native Actions compatible with GitHub syntax)*
|
||||
|
||||
```yaml
|
||||
name: Stella Scan
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
stella:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
STELLA_URL: ${{ secrets.STELLA_URL }}
|
||||
DOCKER_HOST: tcp://docker:2375 # provided by the docker:dind service
|
||||
services:
|
||||
docker:
|
||||
image: docker:dind
|
||||
options: >-
|
||||
--privileged
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Compute vars
|
||||
id: vars
|
||||
run: |
|
||||
echo "IMAGE=ghcr.io/${{ gitea.repository }}:${{ gitea.sha }}" >> $GITEA_OUTPUT
|
||||
echo "SBOM_FILE=$(echo ghcr.io/${{ gitea.repository }}:${{ gitea.sha }} | tr '/:+' '__')-$(date -u +%Y%m%dT%H%M%SZ).sbom.json" >> $GITEA_OUTPUT
|
||||
|
||||
- name: Build image + SBOM (Option A)
|
||||
run: |
|
||||
docker build \
|
||||
--build-arg STELLA_SBOM_BUILDER="${STELLA_URL}/registry/stella-sbom-builder:latest" \
|
||||
--build-arg SBOM_FILE="${{ steps.vars.outputs.SBOM_FILE }}" \
|
||||
-t "${{ steps.vars.outputs.IMAGE }}" .
|
||||
|
||||
# --- Option B fallback (when you must keep the existing build step as‑is) ---
|
||||
#- name: SBOM builder (Option B)
|
||||
# run: |
|
||||
# docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
# -v "$(pwd):/workspace" \
|
||||
# "${STELLA_URL}/registry/stella-sbom-builder:latest" \
|
||||
# build --image "${{ steps.vars.outputs.IMAGE }}" --output "/workspace/${{ steps.vars.outputs.SBOM_FILE }}"
|
||||
|
||||
- name: Scan
|
||||
run: |
|
||||
docker run --rm -e DOCKER_HOST="$DOCKER_HOST" \
|
||||
-v "$(pwd)/${{ steps.vars.outputs.SBOM_FILE }}:/${{ steps.vars.outputs.SBOM_FILE }}:ro" \
|
||||
-e STELLA_OPS_URL="https://${STELLA_URL}" \
|
||||
"${STELLA_URL}/registry/stella-cli:latest" \
|
||||
scan --sbom "/${{ steps.vars.outputs.SBOM_FILE }}" "${{ steps.vars.outputs.IMAGE }}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Docs CI (Gitea Actions & Offline Mirror)
|
||||
|
||||
StellaOps ships a dedicated Docs workflow at `.gitea/workflows/docs.yml`. When mirroring the pipeline offline or running it locally, install the same toolchain so markdown linting, schema validation, and HTML preview stay deterministic.
|
||||
|
||||
### 4.1 Toolchain bootstrap
|
||||
|
||||
```bash
|
||||
# Node.js 20.x is required; install once per runner
|
||||
npm install --no-save \
|
||||
markdown-link-check \
|
||||
remark-cli \
|
||||
remark-preset-lint-recommended \
|
||||
ajv \
|
||||
ajv-cli \
|
||||
ajv-formats
|
||||
|
||||
# Python 3.11+ powers the preview renderer
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install markdown pygments
|
||||
```
|
||||
|
||||
> **No `pip` available?** Some hardened Python builds (including the repo’s `tmp/docenv`
|
||||
> interpreter) ship without `pip`/`ensurepip`. In that case download the pure‑Python
|
||||
> sdists (e.g. `Markdown-3.x.tar.gz`, `pygments-2.x.tar.gz`) and extract their
|
||||
> packages directly into the virtualenv’s `lib/python*/site-packages/` folder.
|
||||
> This keeps the renderer working even when package managers are disabled.
|
||||
|
||||
**Offline tip.** Add the packages above to your artifact mirror (for example `ops/devops/offline-kit.json`) so runners can install them via `npm --offline` / `pip --no-index`.
|
||||
|
||||
### 4.2 Schema validation step
|
||||
|
||||
Ajv compiles every event schema to guard against syntax or format regressions. The workflow uses `ajv-formats` for UUID/date-time support.
|
||||
|
||||
```bash
|
||||
for schema in docs/events/*.json; do
|
||||
npx ajv compile -c ajv-formats -s "$schema"
|
||||
done
|
||||
```
|
||||
|
||||
Run this loop before committing schema changes. For new references, append `-r additional-file.json` so CI and local runs stay aligned.
|
||||
|
||||
### 4.3 Preview build
|
||||
|
||||
```bash
|
||||
python scripts/render_docs.py --source docs --output artifacts/docs-preview --clean
|
||||
```
|
||||
|
||||
Host the resulting bundle via any static file server for review (for example `python -m http.server`).
|
||||
|
||||
### 4.4 Publishing checklist
|
||||
|
||||
- [ ] Toolchain installs succeed without hitting the public internet (mirror or cached tarballs).
|
||||
- [ ] Ajv validation passes for `scanner.report.ready@1`, `scheduler.rescan.delta@1`, `attestor.logged@1`.
|
||||
- [ ] Markdown link check (`npx markdown-link-check`) reports no broken references.
|
||||
- [ ] Preview bundle archived (or attached) for stakeholders.
|
||||
|
||||
### 4.5 Policy DSL lint stage
|
||||
|
||||
Policy Engine v2 pipelines now fail fast if policy documents are malformed. After checkout and dotnet restore, run:
|
||||
|
||||
```bash
|
||||
dotnet run \
|
||||
--project src/Tools/PolicyDslValidator/PolicyDslValidator.csproj \
|
||||
-- \
|
||||
--strict docs/examples/policies/*.yaml
|
||||
```
|
||||
|
||||
- `--strict` treats warnings as errors so missing metadata doesn’t slip through.
|
||||
- The validator accepts globs, so you can point it at tenant policy directories later (`policies/**/*.yaml`).
|
||||
- Exit codes follow UNIX conventions: `0` success, `1` parse/errors, `2` warnings when `--strict` is set, `64` usage mistakes.
|
||||
|
||||
Capture the validator output as part of your build logs; Support uses it when triaging policy rollout issues.
|
||||
|
||||
### 4.6 Policy simulation smoke
|
||||
|
||||
Catch unexpected policy regressions by exercising a small set of golden SBOM findings via the simulation smoke tool:
|
||||
|
||||
```bash
|
||||
dotnet run \
|
||||
--project src/Tools/PolicySimulationSmoke/PolicySimulationSmoke.csproj \
|
||||
-- \
|
||||
--scenario-root samples/policy/simulations \
|
||||
--output artifacts/policy-simulations
|
||||
```
|
||||
|
||||
- The tool loads each `scenario.json` under `samples/policy/simulations`, evaluates the referenced policy, and fails the build if projected verdicts change.
|
||||
- In CI the command runs twice (to `run1/` and `run2/`) and `diff -u` compares the summaries—any mismatch signals a determinism regression.
|
||||
- Artifacts land in `artifacts/policy-simulations/policy-simulation-summary.json`; upload them for later inspection (see CI workflow).
|
||||
- Expand scenarios by copying real-world findings into the samples directory—ensure expected statuses are recorded so regressions trip the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Troubleshooting cheat‑sheet
|
||||
|
||||
| Symptom | Root cause | First things to try |
|
||||
| ------------------------------------- | --------------------------- | --------------------------------------------------------------- |
|
||||
| `no such host $STELLA_URL` | DNS typo or VPN outage | `ping $STELLA_URL` from runner |
|
||||
| `connection refused` when CLI uploads | Port 443 blocked | open firewall / check ingress |
|
||||
| `failed to stat /<sbom>.json` | SBOM wasn’t produced | Did Option A actually run builder? If not, enable Option B |
|
||||
| `registry unauthorized` | Runner lacks registry creds | `docker login $STELLA_URL/registry` (store creds in CI secrets) |
|
||||
| Non‑zero scan exit | Blocking vuln/licence | Open project in Ops UI → triage or waive |
|
||||
|
||||
---
|
||||
|
||||
### Change log
|
||||
|
||||
* **2025‑10‑18** – Documented Docs CI toolchain (Ajv validation, static preview) and offline checklist.
|
||||
* **2025‑08‑04** – Variable clean‑up, removed Docker‑socket & cache mounts, added Jenkins / CircleCI / Gitea examples, clarified Option B comment.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# CLI agent guide
|
||||
|
||||
## Mission
|
||||
The `stella` CLI is the operator-facing Swiss army knife for scans, exports, policy management, offline kit operations, and automation scripting.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# CLI agent guide
|
||||
|
||||
## Mission
|
||||
The `stella` CLI is the operator-facing Swiss army knife for scans, exports, policy management, offline kit operations, and automation scripting.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,40 +1,40 @@
|
||||
# StellaOps CLI
|
||||
|
||||
The `stella` CLI is the operator-facing Swiss army knife for scans, exports, policy management, offline kit operations, and automation scripting.
|
||||
|
||||
## Responsibilities
|
||||
- Deliver deterministic verbs for scan, diff, export, policy, and observability operations.
|
||||
- Handle interactive and non-interactive authentication via Authority (device code, client credentials).
|
||||
- Support offline kit workflows including bundle verification and seed installation.
|
||||
- Expose JSON outputs suitable for CI parity and golden tests.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Cli` native AOT host.
|
||||
- Shared helpers in `StellaOps.Cli.Core`.
|
||||
- Restart-time plug-ins under `StellaOps.Cli.Plugins.*`.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Authority for token exchange.
|
||||
- Backend APIs (Scanner, Policy, Export Center, Notify).
|
||||
- Offline kit bundles and local keychain/DPoP storage.
|
||||
|
||||
## Operational notes
|
||||
- Deterministic output fixtures under `src/Cli/StellaOps.Cli.Tests`.
|
||||
- Versioned command docs in `docs/modules/cli/guides`.
|
||||
- Plugin catalogue in `plugins/cli/**` (restart-only).
|
||||
|
||||
## Related resources
|
||||
- ./guides/20_REFERENCE.md
|
||||
- ./guides/cli-reference.md
|
||||
- ./guides/policy.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-CLI-OBS-52-001 / DOCS-CLI-FORENSICS-53-001 in ../../TASKS.md.
|
||||
- CLI-CORE-41-001 epic in `src/Cli/StellaOps.Cli/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 2 – Policy Engine & Editor:** deliver deterministic policy authoring, simulation, and explain verbs.
|
||||
- **Epic 4 – Policy Studio:** integrate registry/promotion workflows, approvals, and lint tooling.
|
||||
- **Epic 6 – Vulnerability Explorer:** surface triage and ledger operations.
|
||||
- **Epic 10 – Export Center:** orchestrate export requests, verification, and Offline Kit automation.
|
||||
- **Epic 11 – Notifications Studio:** manage notification authoring/previews from the command line.
|
||||
# StellaOps CLI
|
||||
|
||||
The `stella` CLI is the operator-facing Swiss army knife for scans, exports, policy management, offline kit operations, and automation scripting.
|
||||
|
||||
## Responsibilities
|
||||
- Deliver deterministic verbs for scan, diff, export, policy, and observability operations.
|
||||
- Handle interactive and non-interactive authentication via Authority (device code, client credentials).
|
||||
- Support offline kit workflows including bundle verification and seed installation.
|
||||
- Expose JSON outputs suitable for CI parity and golden tests.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Cli` native AOT host.
|
||||
- Shared helpers in `StellaOps.Cli.Core`.
|
||||
- Restart-time plug-ins under `StellaOps.Cli.Plugins.*`.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Authority for token exchange.
|
||||
- Backend APIs (Scanner, Policy, Export Center, Notify).
|
||||
- Offline kit bundles and local keychain/DPoP storage.
|
||||
|
||||
## Operational notes
|
||||
- Deterministic output fixtures under `src/Cli/StellaOps.Cli.Tests`.
|
||||
- Versioned command docs in `docs/modules/cli/guides`.
|
||||
- Plugin catalogue in `plugins/cli/**` (restart-only).
|
||||
|
||||
## Related resources
|
||||
- ./guides/20_REFERENCE.md
|
||||
- ./guides/cli-reference.md
|
||||
- ./guides/policy.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-CLI-OBS-52-001 / DOCS-CLI-FORENSICS-53-001 in ../../TASKS.md.
|
||||
- CLI-CORE-41-001 epic in `src/Cli/StellaOps.Cli/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 2 – Policy Engine & Editor:** deliver deterministic policy authoring, simulation, and explain verbs.
|
||||
- **Epic 4 – Policy Studio:** integrate registry/promotion workflows, approvals, and lint tooling.
|
||||
- **Epic 6 – Vulnerability Explorer:** surface triage and ledger operations.
|
||||
- **Epic 10 – Export Center:** orchestrate export requests, verification, and Offline Kit automation.
|
||||
- **Epic 11 – Notifications Studio:** manage notification authoring/previews from the command line.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — CLI
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CLI-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CLI-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CLI-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — CLI
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CLI-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CLI-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CLI-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# CLI Reference (`stella --help`)
|
||||
|
||||
> **Auto‑generated file — do not edit manually.**
|
||||
> On every tagged release the CI pipeline runs
|
||||
> `stella --help --markdown > docs/modules/cli/guides/20_REFERENCE.md`
|
||||
> ensuring this document always matches the shipped binary.
|
||||
|
||||
*(The reference will appear after the first public α release.)*
|
||||
# CLI Reference (`stella --help`)
|
||||
|
||||
> **Auto‑generated file — do not edit manually.**
|
||||
> On every tagged release the CI pipeline runs
|
||||
> `stella --help --markdown > docs/modules/cli/guides/20_REFERENCE.md`
|
||||
> ensuring this document always matches the shipped binary.
|
||||
|
||||
*(The reference will appear after the first public α release.)*
|
||||
|
||||
@@ -1,316 +1,316 @@
|
||||
# CLI AOC Commands Reference
|
||||
|
||||
> **Audience:** DevEx engineers, operators, and CI authors integrating the `stella` CLI with Aggregation-Only Contract (AOC) workflows.
|
||||
> **Scope:** Command synopsis, options, exit codes, and offline considerations for `stella sources ingest --dry-run` and `stella aoc verify` as introduced in Sprint 19.
|
||||
|
||||
# CLI AOC Commands Reference
|
||||
|
||||
> **Audience:** DevEx engineers, operators, and CI authors integrating the `stella` CLI with Aggregation-Only Contract (AOC) workflows.
|
||||
> **Scope:** Command synopsis, options, exit codes, and offline considerations for `stella sources ingest --dry-run` and `stella aoc verify` as introduced in Sprint 19.
|
||||
|
||||
Both commands are designed to enforce the AOC guardrails documented in the [aggregation-only reference](../../../ingestion/aggregation-only-contract.md) and the [architecture overview](../architecture.md). They consume Authority-issued tokens with tenant scopes and never mutate ingestion stores.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites
|
||||
|
||||
- CLI version: `stella` ≥ 0.19.0 (AOC feature gate enabled).
|
||||
- Required scopes (DPoP-bound):
|
||||
- `advisory:read` for Concelier sources.
|
||||
- `vex:read` for Excititor sources (optional but required for VEX checks).
|
||||
- `aoc:verify` to invoke guard verification endpoints.
|
||||
- `tenant:select` if your deployment uses tenant switching.
|
||||
- Connectivity: direct access to Concelier/Excititor APIs or Offline Kit snapshot (see § 4).
|
||||
- Environment: set `STELLA_AUTHORITY_URL`, `STELLA_TENANT`, and export a valid OpTok via `stella auth login` or existing token cache.
|
||||
|
||||
---
|
||||
|
||||
## 2 · `stella sources ingest --dry-run`
|
||||
|
||||
### 2.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source <source-key> \
|
||||
--input <path-or-uri> \
|
||||
[--tenant <tenant-id>] \
|
||||
[--format json|table] \
|
||||
[--no-color] \
|
||||
[--output <file>]
|
||||
```
|
||||
|
||||
### 2.2 Description
|
||||
|
||||
Previews an ingestion write without touching MongoDB. The command loads an upstream advisory or VEX document, computes the would-write payload, runs it through the `AOCWriteGuard`, and reports any forbidden fields, provenance gaps, or idempotency issues. Use it during connector development, CI validation, or while triaging incidents.
|
||||
|
||||
### 2.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--source <source-key>` | Logical source name (`redhat`, `ubuntu`, `osv`, etc.). Mirrors connector configuration. |
|
||||
| `--input <path-or-uri>` | Path to local CSAF/OSV/VEX file or HTTPS URI. CLI normalises transport (gzip/base64) before guard evaluation. |
|
||||
| `--tenant <tenant-id>` | Overrides default tenant for multi-tenant deployments. Mandatory when `STELLA_TENANT` is not set. |
|
||||
| `--format json|table` | Output format. `table` (default) prints summary with highlighted violations; `json` emits machine-readable report (see below). |
|
||||
| `--no-color` | Disables ANSI colour output for CI logs. |
|
||||
| `--output <file>` | Writes the JSON report to file while still printing human-readable summary to stdout. |
|
||||
|
||||
### 2.4 Output schema (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "redhat",
|
||||
"tenant": "default",
|
||||
"guardVersion": "1.0.0",
|
||||
"status": "ok",
|
||||
"document": {
|
||||
"contentHash": "sha256:…",
|
||||
"supersedes": null,
|
||||
"provenance": {
|
||||
"signature": { "format": "pgp", "present": true }
|
||||
}
|
||||
},
|
||||
"violations": []
|
||||
}
|
||||
```
|
||||
|
||||
When violations exist, `status` becomes `error` and `violations` contains entries with `code` (`ERR_AOC_00x`), a short `message`, and JSON Pointer `path` values indicating offending fields.
|
||||
|
||||
### 2.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Guard passed; would-write payload is AOC compliant. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field (`severity`, `cvss`, etc.) detected. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt (multiple upstream sources fused). |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation (duplicate without supersedes). |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance fields. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings present (Policy-only data). |
|
||||
| `17` | `ERR_AOC_007` – Unknown top-level fields / schema violation. |
|
||||
| `70` | Transport error (network, auth, malformed input). |
|
||||
|
||||
> Exit codes map directly to the `ERR_AOC_00x` table for scripting consistency. Multiple violations yield the highest-priority code (e.g., 11 takes precedence over 14).
|
||||
|
||||
### 2.6 Examples
|
||||
|
||||
Dry-run a local CSAF file:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-1234.json
|
||||
```
|
||||
|
||||
Stream from HTTPS and emit JSON for CI:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source osv \
|
||||
--input https://osv.dev/vulnerability/GHSA-aaaa-bbbb \
|
||||
--format json \
|
||||
--output artifacts/osv-dry-run.json
|
||||
|
||||
cat artifacts/osv-dry-run.json | jq '.violations'
|
||||
```
|
||||
|
||||
### 2.7 Offline notes
|
||||
|
||||
When operating in sealed/offline mode:
|
||||
|
||||
- Use `--input` paths pointing to Offline Kit snapshots (`offline-kit/advisories/*.json`).
|
||||
- Provide `--tenant` explicitly if the offline bundle contains multiple tenants.
|
||||
- The command does not attempt network access when given a file path.
|
||||
- Store reports with `--output` to include in transfer packages for policy review.
|
||||
|
||||
---
|
||||
|
||||
## 3 · `stella aoc verify`
|
||||
|
||||
### 3.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
[--since <iso8601|duration>] \
|
||||
[--limit <count>] \
|
||||
[--sources <list>] \
|
||||
[--codes <ERR_AOC_00x,...>] \
|
||||
[--format table|json] \
|
||||
[--export <file>] \
|
||||
[--tenant <tenant-id>] \
|
||||
[--no-color]
|
||||
```
|
||||
|
||||
### 3.2 Description
|
||||
|
||||
Replays the AOC guard against stored raw documents. By default it checks all advisories and VEX statements ingested in the last 24 hours for the active tenant, reporting totals, top violation codes, and sample documents. Use it in CI pipelines, scheduled verifications, or during incident response.
|
||||
|
||||
### 3.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--since <value>` | Verification window. Accepts ISO 8601 timestamp (`2025-10-25T12:00:00Z`) or duration (`48h`, `7d`). Defaults to `24h`. |
|
||||
| `--limit <count>` | Maximum number of violations to display (per code). `0` means show all. Defaults to `20`. |
|
||||
| `--sources <list>` | Comma-separated list of sources (`redhat,ubuntu,osv`). Filters both advisories and VEX entries. |
|
||||
| `--codes <list>` | Restricts output to specific `ERR_AOC_00x` codes. Useful for regression tracking. |
|
||||
| `--format table|json` | `table` (default) prints summary plus top violations; `json` outputs machine-readable report identical to the `/aoc/verify` API. |
|
||||
| `--export <file>` | Writes the JSON report to disk (useful for audits/offline uploads). |
|
||||
| `--tenant <tenant-id>` | Overrides tenant context. Required for cross-tenant verifications when run by platform operators. |
|
||||
| `--no-color` | Disables ANSI colours. |
|
||||
|
||||
`table` mode prints a summary showing the active tenant, evaluated window, counts of checked advisories/VEX statements, the active limit, total writes/violations, and whether the page was truncated. Status is colour-coded as `ok`, `violations`, or `truncated`. When violations exist the detail table lists the code, total occurrences, first sample document (`source` + `documentId` + `contentHash`), and JSON pointer path.
|
||||
|
||||
### 3.4 Report structure (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"tenant": "default",
|
||||
"window": {
|
||||
"from": "2025-10-25T12:00:00Z",
|
||||
"to": "2025-10-26T12:00:00Z"
|
||||
},
|
||||
"checked": {
|
||||
"advisories": 482,
|
||||
"vex": 75
|
||||
},
|
||||
"violations": [
|
||||
{
|
||||
"code": "ERR_AOC_001",
|
||||
"count": 2,
|
||||
"examples": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"documentId": "advisory_raw:redhat:RHSA-2025:1",
|
||||
"contentHash": "sha256:…",
|
||||
"path": "/content/raw/cvss"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"metrics": {
|
||||
"ingestion_write_total": 557,
|
||||
"aoc_violation_total": 2
|
||||
},
|
||||
"truncated": false
|
||||
}
|
||||
```
|
||||
|
||||
### 3.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Verification succeeded with zero violations. |
|
||||
| `11…17` | Same mapping as § 2.5 when violations are detected. Highest-priority code returned. |
|
||||
| `18` | Verification ran but results truncated (limit reached) – treat as warning; rerun with higher `--limit`. |
|
||||
| `70` | Transport/authentication error. |
|
||||
| `71` | CLI misconfiguration (missing tenant, invalid `--since`, etc.). |
|
||||
|
||||
### 3.6 Examples
|
||||
|
||||
Daily verification across all sources:
|
||||
|
||||
```bash
|
||||
stella aoc verify --since 24h --format table
|
||||
```
|
||||
|
||||
CI pipeline focusing on errant sources and exporting evidence:
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--sources redhat,ubuntu \
|
||||
--codes ERR_AOC_001,ERR_AOC_004 \
|
||||
--format json \
|
||||
--limit 100 \
|
||||
--export artifacts/aoc-verify.json
|
||||
|
||||
jq '.violations[] | {code, count}' artifacts/aoc-verify.json
|
||||
```
|
||||
|
||||
Air-gapped verification using Offline Kit snapshot (example script):
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--since 7d \
|
||||
--format json \
|
||||
--export /mnt/offline/aoc-verify-$(date +%F).json
|
||||
|
||||
sha256sum /mnt/offline/aoc-verify-*.json > /mnt/offline/checksums.txt
|
||||
```
|
||||
|
||||
### 3.7 Automation tips
|
||||
|
||||
- Schedule with `cron` or platform scheduler and fail the job when exit code ≥ 11.
|
||||
- Pair with `stella sources ingest --dry-run` for pre-flight validation before re-enabling a paused source.
|
||||
- Push JSON exports to observability pipelines for historical tracking of violation counts.
|
||||
|
||||
### 3.8 Offline notes
|
||||
|
||||
- Works against Offline Kit Mongo snapshots when CLI is pointed at the local API gateway included in the bundle.
|
||||
- When fully disconnected, run against exported `aoc verify` reports generated on production and replay them using `--format json --export` (automation recipe above).
|
||||
- Include verification output in compliance packages alongside Offline Kit manifests.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Global exit-code reference
|
||||
|
||||
| Code | Summary |
|
||||
|------|---------|
|
||||
| `0` | Success / no violations. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field present. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt detected. |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation. |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance/signature metadata. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings in ingestion payload. |
|
||||
| `17` | `ERR_AOC_007` – Schema violation / unknown fields. |
|
||||
| `18` | Partial verification (limit reached). |
|
||||
| `70` | Transport or HTTP failure. |
|
||||
| `71` | CLI usage error (invalid arguments, missing tenant). |
|
||||
|
||||
Use these codes in CI to map outcomes to build statuses or alert severities.
|
||||
|
||||
---
|
||||
|
||||
## 4 · `stella vuln observations` (Overlay paging)
|
||||
|
||||
`stella vuln observations` lists raw advisory observations for downstream overlays (Graph Explorer, Policy simulations, Console). Large tenants can now page through results deterministically.
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--limit <count>` | Caps the number of observations returned in a single call. Defaults to `200`; values above `500` are clamped server-side. |
|
||||
| `--cursor <token>` | Opaque continuation token produced by the previous page (`nextCursor` in JSON output). Pass it back to resume iteration. |
|
||||
|
||||
Additional notes:
|
||||
|
||||
- Table mode prints a hint when `hasMore` is `true`:
|
||||
`[yellow]More observations available. Continue with --cursor <token>[/]`.
|
||||
- JSON mode returns `nextCursor` and `hasMore` alongside the observation list so automation can loop until `hasMore` is `false`.
|
||||
- Supplying a non-positive limit falls back to the default (`200`). Invalid/expired cursors yield `400 Bad Request`; restart without `--cursor` to begin a fresh iteration.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Related references
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 1 · Prerequisites
|
||||
|
||||
- CLI version: `stella` ≥ 0.19.0 (AOC feature gate enabled).
|
||||
- Required scopes (DPoP-bound):
|
||||
- `advisory:read` for Concelier sources.
|
||||
- `vex:read` for Excititor sources (optional but required for VEX checks).
|
||||
- `aoc:verify` to invoke guard verification endpoints.
|
||||
- `tenant:select` if your deployment uses tenant switching.
|
||||
- Connectivity: direct access to Concelier/Excititor APIs or Offline Kit snapshot (see § 4).
|
||||
- Environment: set `STELLA_AUTHORITY_URL`, `STELLA_TENANT`, and export a valid OpTok via `stella auth login` or existing token cache.
|
||||
|
||||
---
|
||||
|
||||
## 2 · `stella sources ingest --dry-run`
|
||||
|
||||
### 2.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source <source-key> \
|
||||
--input <path-or-uri> \
|
||||
[--tenant <tenant-id>] \
|
||||
[--format json|table] \
|
||||
[--no-color] \
|
||||
[--output <file>]
|
||||
```
|
||||
|
||||
### 2.2 Description
|
||||
|
||||
Previews an ingestion write without touching MongoDB. The command loads an upstream advisory or VEX document, computes the would-write payload, runs it through the `AOCWriteGuard`, and reports any forbidden fields, provenance gaps, or idempotency issues. Use it during connector development, CI validation, or while triaging incidents.
|
||||
|
||||
### 2.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--source <source-key>` | Logical source name (`redhat`, `ubuntu`, `osv`, etc.). Mirrors connector configuration. |
|
||||
| `--input <path-or-uri>` | Path to local CSAF/OSV/VEX file or HTTPS URI. CLI normalises transport (gzip/base64) before guard evaluation. |
|
||||
| `--tenant <tenant-id>` | Overrides default tenant for multi-tenant deployments. Mandatory when `STELLA_TENANT` is not set. |
|
||||
| `--format json|table` | Output format. `table` (default) prints summary with highlighted violations; `json` emits machine-readable report (see below). |
|
||||
| `--no-color` | Disables ANSI colour output for CI logs. |
|
||||
| `--output <file>` | Writes the JSON report to file while still printing human-readable summary to stdout. |
|
||||
|
||||
### 2.4 Output schema (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "redhat",
|
||||
"tenant": "default",
|
||||
"guardVersion": "1.0.0",
|
||||
"status": "ok",
|
||||
"document": {
|
||||
"contentHash": "sha256:…",
|
||||
"supersedes": null,
|
||||
"provenance": {
|
||||
"signature": { "format": "pgp", "present": true }
|
||||
}
|
||||
},
|
||||
"violations": []
|
||||
}
|
||||
```
|
||||
|
||||
When violations exist, `status` becomes `error` and `violations` contains entries with `code` (`ERR_AOC_00x`), a short `message`, and JSON Pointer `path` values indicating offending fields.
|
||||
|
||||
### 2.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Guard passed; would-write payload is AOC compliant. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field (`severity`, `cvss`, etc.) detected. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt (multiple upstream sources fused). |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation (duplicate without supersedes). |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance fields. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings present (Policy-only data). |
|
||||
| `17` | `ERR_AOC_007` – Unknown top-level fields / schema violation. |
|
||||
| `70` | Transport error (network, auth, malformed input). |
|
||||
|
||||
> Exit codes map directly to the `ERR_AOC_00x` table for scripting consistency. Multiple violations yield the highest-priority code (e.g., 11 takes precedence over 14).
|
||||
|
||||
### 2.6 Examples
|
||||
|
||||
Dry-run a local CSAF file:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source redhat \
|
||||
--input ./fixtures/redhat/RHSA-2025-1234.json
|
||||
```
|
||||
|
||||
Stream from HTTPS and emit JSON for CI:
|
||||
|
||||
```bash
|
||||
stella sources ingest --dry-run \
|
||||
--source osv \
|
||||
--input https://osv.dev/vulnerability/GHSA-aaaa-bbbb \
|
||||
--format json \
|
||||
--output artifacts/osv-dry-run.json
|
||||
|
||||
cat artifacts/osv-dry-run.json | jq '.violations'
|
||||
```
|
||||
|
||||
### 2.7 Offline notes
|
||||
|
||||
When operating in sealed/offline mode:
|
||||
|
||||
- Use `--input` paths pointing to Offline Kit snapshots (`offline-kit/advisories/*.json`).
|
||||
- Provide `--tenant` explicitly if the offline bundle contains multiple tenants.
|
||||
- The command does not attempt network access when given a file path.
|
||||
- Store reports with `--output` to include in transfer packages for policy review.
|
||||
|
||||
---
|
||||
|
||||
## 3 · `stella aoc verify`
|
||||
|
||||
### 3.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
[--since <iso8601|duration>] \
|
||||
[--limit <count>] \
|
||||
[--sources <list>] \
|
||||
[--codes <ERR_AOC_00x,...>] \
|
||||
[--format table|json] \
|
||||
[--export <file>] \
|
||||
[--tenant <tenant-id>] \
|
||||
[--no-color]
|
||||
```
|
||||
|
||||
### 3.2 Description
|
||||
|
||||
Replays the AOC guard against stored raw documents. By default it checks all advisories and VEX statements ingested in the last 24 hours for the active tenant, reporting totals, top violation codes, and sample documents. Use it in CI pipelines, scheduled verifications, or during incident response.
|
||||
|
||||
### 3.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--since <value>` | Verification window. Accepts ISO 8601 timestamp (`2025-10-25T12:00:00Z`) or duration (`48h`, `7d`). Defaults to `24h`. |
|
||||
| `--limit <count>` | Maximum number of violations to display (per code). `0` means show all. Defaults to `20`. |
|
||||
| `--sources <list>` | Comma-separated list of sources (`redhat,ubuntu,osv`). Filters both advisories and VEX entries. |
|
||||
| `--codes <list>` | Restricts output to specific `ERR_AOC_00x` codes. Useful for regression tracking. |
|
||||
| `--format table|json` | `table` (default) prints summary plus top violations; `json` outputs machine-readable report identical to the `/aoc/verify` API. |
|
||||
| `--export <file>` | Writes the JSON report to disk (useful for audits/offline uploads). |
|
||||
| `--tenant <tenant-id>` | Overrides tenant context. Required for cross-tenant verifications when run by platform operators. |
|
||||
| `--no-color` | Disables ANSI colours. |
|
||||
|
||||
`table` mode prints a summary showing the active tenant, evaluated window, counts of checked advisories/VEX statements, the active limit, total writes/violations, and whether the page was truncated. Status is colour-coded as `ok`, `violations`, or `truncated`. When violations exist the detail table lists the code, total occurrences, first sample document (`source` + `documentId` + `contentHash`), and JSON pointer path.
|
||||
|
||||
### 3.4 Report structure (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"tenant": "default",
|
||||
"window": {
|
||||
"from": "2025-10-25T12:00:00Z",
|
||||
"to": "2025-10-26T12:00:00Z"
|
||||
},
|
||||
"checked": {
|
||||
"advisories": 482,
|
||||
"vex": 75
|
||||
},
|
||||
"violations": [
|
||||
{
|
||||
"code": "ERR_AOC_001",
|
||||
"count": 2,
|
||||
"examples": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"documentId": "advisory_raw:redhat:RHSA-2025:1",
|
||||
"contentHash": "sha256:…",
|
||||
"path": "/content/raw/cvss"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"metrics": {
|
||||
"ingestion_write_total": 557,
|
||||
"aoc_violation_total": 2
|
||||
},
|
||||
"truncated": false
|
||||
}
|
||||
```
|
||||
|
||||
### 3.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Verification succeeded with zero violations. |
|
||||
| `11…17` | Same mapping as § 2.5 when violations are detected. Highest-priority code returned. |
|
||||
| `18` | Verification ran but results truncated (limit reached) – treat as warning; rerun with higher `--limit`. |
|
||||
| `70` | Transport/authentication error. |
|
||||
| `71` | CLI misconfiguration (missing tenant, invalid `--since`, etc.). |
|
||||
|
||||
### 3.6 Examples
|
||||
|
||||
Daily verification across all sources:
|
||||
|
||||
```bash
|
||||
stella aoc verify --since 24h --format table
|
||||
```
|
||||
|
||||
CI pipeline focusing on errant sources and exporting evidence:
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--sources redhat,ubuntu \
|
||||
--codes ERR_AOC_001,ERR_AOC_004 \
|
||||
--format json \
|
||||
--limit 100 \
|
||||
--export artifacts/aoc-verify.json
|
||||
|
||||
jq '.violations[] | {code, count}' artifacts/aoc-verify.json
|
||||
```
|
||||
|
||||
Air-gapped verification using Offline Kit snapshot (example script):
|
||||
|
||||
```bash
|
||||
stella aoc verify \
|
||||
--since 7d \
|
||||
--format json \
|
||||
--export /mnt/offline/aoc-verify-$(date +%F).json
|
||||
|
||||
sha256sum /mnt/offline/aoc-verify-*.json > /mnt/offline/checksums.txt
|
||||
```
|
||||
|
||||
### 3.7 Automation tips
|
||||
|
||||
- Schedule with `cron` or platform scheduler and fail the job when exit code ≥ 11.
|
||||
- Pair with `stella sources ingest --dry-run` for pre-flight validation before re-enabling a paused source.
|
||||
- Push JSON exports to observability pipelines for historical tracking of violation counts.
|
||||
|
||||
### 3.8 Offline notes
|
||||
|
||||
- Works against Offline Kit Mongo snapshots when CLI is pointed at the local API gateway included in the bundle.
|
||||
- When fully disconnected, run against exported `aoc verify` reports generated on production and replay them using `--format json --export` (automation recipe above).
|
||||
- Include verification output in compliance packages alongside Offline Kit manifests.
|
||||
|
||||
---
|
||||
|
||||
## 4 · Global exit-code reference
|
||||
|
||||
| Code | Summary |
|
||||
|------|---------|
|
||||
| `0` | Success / no violations. |
|
||||
| `11` | `ERR_AOC_001` – Forbidden field present. |
|
||||
| `12` | `ERR_AOC_002` – Merge attempt detected. |
|
||||
| `13` | `ERR_AOC_003` – Idempotency violation. |
|
||||
| `14` | `ERR_AOC_004` – Missing provenance/signature metadata. |
|
||||
| `15` | `ERR_AOC_005` – Signature/checksum mismatch. |
|
||||
| `16` | `ERR_AOC_006` – Effective findings in ingestion payload. |
|
||||
| `17` | `ERR_AOC_007` – Schema violation / unknown fields. |
|
||||
| `18` | Partial verification (limit reached). |
|
||||
| `70` | Transport or HTTP failure. |
|
||||
| `71` | CLI usage error (invalid arguments, missing tenant). |
|
||||
|
||||
Use these codes in CI to map outcomes to build statuses or alert severities.
|
||||
|
||||
---
|
||||
|
||||
## 4 · `stella vuln observations` (Overlay paging)
|
||||
|
||||
`stella vuln observations` lists raw advisory observations for downstream overlays (Graph Explorer, Policy simulations, Console). Large tenants can now page through results deterministically.
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--limit <count>` | Caps the number of observations returned in a single call. Defaults to `200`; values above `500` are clamped server-side. |
|
||||
| `--cursor <token>` | Opaque continuation token produced by the previous page (`nextCursor` in JSON output). Pass it back to resume iteration. |
|
||||
|
||||
Additional notes:
|
||||
|
||||
- Table mode prints a hint when `hasMore` is `true`:
|
||||
`[yellow]More observations available. Continue with --cursor <token>[/]`.
|
||||
- JSON mode returns `nextCursor` and `hasMore` alongside the observation list so automation can loop until `hasMore` is `false`.
|
||||
- Supplying a non-positive limit falls back to the default (`200`). Invalid/expired cursors yield `400 Bad Request`; restart without `--cursor` to begin a fresh iteration.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Related references
|
||||
|
||||
- [Aggregation-Only Contract reference](../../../ingestion/aggregation-only-contract.md)
|
||||
- [Architecture overview](../../platform/architecture-overview.md)
|
||||
- [Architecture overview](../../platform/architecture-overview.md)
|
||||
- [Console AOC dashboard](../../../ui/console.md)
|
||||
- [Authority scopes](../../authority/architecture.md)
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Usage documented for both table and JSON formats.
|
||||
- [ ] Exit-code mapping matches `ERR_AOC_00x` definitions and automation guidance.
|
||||
- [ ] Offline/air-gap workflow captured for both commands.
|
||||
- [ ] References to AOC architecture and console docs included.
|
||||
- [ ] Examples validated against current CLI syntax (update post-implementation).
|
||||
- [ ] Docs guild screenshot/narrative placeholder logged for release notes (pending CLI team capture).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-29 (Sprint 24).*
|
||||
|
||||
## 13. Authority configuration quick reference
|
||||
|
||||
| Setting | Purpose | How to set |
|
||||
|---------|---------|------------|
|
||||
| `StellaOps:Authority:OperatorReason` | Incident/change description recorded with `orch:operate` tokens. | CLI flag `--Authority:OperatorReason=...` or env `STELLAOPS_ORCH_REASON`. |
|
||||
| `StellaOps:Authority:OperatorTicket` | Change/incident ticket reference paired with orchestrator control actions. | CLI flag `--Authority:OperatorTicket=...` or env `STELLAOPS_ORCH_TICKET`. |
|
||||
|
||||
> Tokens requesting `orch:operate` will fail with `invalid_request` unless both values are present. Choose concise strings (≤256 chars for reason, ≤128 chars for ticket) and avoid sensitive data.
|
||||
|
||||
- [Authority scopes](../../authority/architecture.md)
|
||||
|
||||
---
|
||||
|
||||
## 6 · Compliance checklist
|
||||
|
||||
- [ ] Usage documented for both table and JSON formats.
|
||||
- [ ] Exit-code mapping matches `ERR_AOC_00x` definitions and automation guidance.
|
||||
- [ ] Offline/air-gap workflow captured for both commands.
|
||||
- [ ] References to AOC architecture and console docs included.
|
||||
- [ ] Examples validated against current CLI syntax (update post-implementation).
|
||||
- [ ] Docs guild screenshot/narrative placeholder logged for release notes (pending CLI team capture).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-29 (Sprint 24).*
|
||||
|
||||
## 13. Authority configuration quick reference
|
||||
|
||||
| Setting | Purpose | How to set |
|
||||
|---------|---------|------------|
|
||||
| `StellaOps:Authority:OperatorReason` | Incident/change description recorded with `orch:operate` tokens. | CLI flag `--Authority:OperatorReason=...` or env `STELLAOPS_ORCH_REASON`. |
|
||||
| `StellaOps:Authority:OperatorTicket` | Change/incident ticket reference paired with orchestrator control actions. | CLI flag `--Authority:OperatorTicket=...` or env `STELLAOPS_ORCH_TICKET`. |
|
||||
|
||||
> Tokens requesting `orch:operate` will fail with `invalid_request` unless both values are present. Choose concise strings (≤256 chars for reason, ≤128 chars for ticket) and avoid sensitive data.
|
||||
|
||||
|
||||
@@ -1,318 +1,318 @@
|
||||
# Stella CLI — Policy Commands
|
||||
|
||||
> **Audience:** Policy authors, reviewers, operators, and CI engineers using the `stella` CLI to interact with Policy Engine.
|
||||
> **Supported from:** `stella` CLI ≥ 0.20.0 (Policy Engine v2 sprint line).
|
||||
> **Prerequisites:** Authority-issued bearer token with the scopes noted per command (export `STELLA_TOKEN` or pass `--token`).
|
||||
> **2025-10-27 scope update:** CLI/CI tokens issued prior to Sprint 23 (AUTH-POLICY-23-001) must drop `policy:write`/`policy:submit`/`policy:edit` and instead request `policy:read`, `policy:author`, `policy:review`, and `policy:simulate` (plus `policy:approve`/`policy:operate`/`policy:activate` for promotion pipelines).
|
||||
|
||||
---
|
||||
|
||||
## 1 · Global Options & Output Modes
|
||||
|
||||
All `stella policy *` commands honour the common CLI options:
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--server <url>` | `https://stella.local` | Policy Engine gateway root. |
|
||||
| `--tenant <id>` | token default | Override tenant for multi-tenant installs. |
|
||||
| `--format <table\|json\|yaml>` | `table` for TTY, `json` otherwise | Output format for listings/diffs. |
|
||||
| `--output <file>` | stdout | Write full JSON payload to file. |
|
||||
| `--sealed` | false | Force sealed-mode behaviour (no outbound fetch). |
|
||||
| `--trace` | false | Emit verbose timing/log correlation info. |
|
||||
|
||||
> **Tip:** Set `STELLA_PROFILE=policy` in CI to load saved defaults from `~/.stella/profiles/policy.toml`.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Authoring & Drafting Commands
|
||||
|
||||
### 2.1 `stella policy new`
|
||||
|
||||
Create a draft policy from a template or scratch.
|
||||
|
||||
```
|
||||
stella policy new --policy-id P-7 --name "Default Org Policy" \
|
||||
--template baseline --output-path policies/P-7.stella
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--policy-id` *(required)* | Stable identifier (e.g., `P-7`). |
|
||||
| `--name` | Friendly display name. |
|
||||
| `--template` | `baseline`, `serverless`, `blank`. |
|
||||
| `--from` | Start from existing version (`policyId@version`). |
|
||||
| `--open` | Launches `$EDITOR` after creation. |
|
||||
|
||||
Writes DSL to local file and registers draft version (`status=draft`). Requires `policy:write`.
|
||||
|
||||
### 2.2 `stella policy edit`
|
||||
|
||||
Open an existing draft in the local editor.
|
||||
|
||||
```
|
||||
stella policy edit P-7 --version 4
|
||||
```
|
||||
|
||||
- Auto-checks out latest draft if `--version` omitted.
|
||||
- Saves to temp file, uploads on editor exit (unless `--no-upload`).
|
||||
- Use `--watch` to keep command alive and re-upload on every save.
|
||||
|
||||
### 2.3 `stella policy lint`
|
||||
|
||||
Static validation without submitting.
|
||||
|
||||
```
|
||||
stella policy lint policies/P-7.stella --format json
|
||||
```
|
||||
|
||||
Outputs diagnostics (line/column, code, message). Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | No lint errors. |
|
||||
| `10` | Syntax/compile errors (`ERR_POL_001`). |
|
||||
| `11` | Unsupported syntax version. |
|
||||
|
||||
### 2.4 `stella policy compile`
|
||||
|
||||
Emits IR digest and rule summary.
|
||||
|
||||
```
|
||||
stella policy compile P-7 --version 4
|
||||
```
|
||||
|
||||
Returns JSON with `digest`, `rules.count`, action counts. Exit `0` success, `10` on compile errors.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Lifecycle Workflow
|
||||
|
||||
### 3.1 Submit
|
||||
|
||||
```
|
||||
stella policy submit P-7 --version 4 \
|
||||
--reviewer user:kay --reviewer group:sec-reviewers \
|
||||
--note "Simulated against golden SBOM set" \
|
||||
--attach sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Requires `policy:submit`. CLI validates that lint/compile run within 24 h and bundle attachments exist.
|
||||
|
||||
### 3.2 Review
|
||||
|
||||
```
|
||||
stella policy review P-7 --version 4 --approve \
|
||||
--note "Looks good; ensure incident playbook updated."
|
||||
```
|
||||
|
||||
- `--approve`, `--request-changes`, or `--comment`.
|
||||
- Provide `--blocking` to mark comment as blocking.
|
||||
- Requires `policy:review`.
|
||||
|
||||
### 3.3 Approve
|
||||
|
||||
```
|
||||
stella policy approve P-7 --version 4 \
|
||||
--note "Determinism CI green; simulation diff attached." \
|
||||
--attach sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Prompts for confirmation; refuses if approver == submitter. Requires `policy:approve`.
|
||||
|
||||
### 3.4 Activate
|
||||
|
||||
```
|
||||
stella policy activate P-7 --version 4 --run-now --priority high
|
||||
```
|
||||
|
||||
- Optional `--scheduled-at 2025-10-27T02:00:00Z`.
|
||||
- Requires `policy:activate` and `policy:run`.
|
||||
|
||||
**Options**
|
||||
|
||||
- `--version <number>` (required) – target revision to promote.
|
||||
- `--note <text>` – record an activation note alongside the approval.
|
||||
- `--run-now` – enqueue an immediate full run after activation.
|
||||
- `--scheduled-at <timestamp>` – schedule activation for a specific UTC time (ISO-8601 format).
|
||||
- `--priority <label>` – optional scheduling priority hint (`low`, `standard`, `high`).
|
||||
- `--rollback` – mark the activation as a rollback of a previously active version.
|
||||
- `--incident <id>` – associate the activation with an incident identifier.
|
||||
|
||||
**Exit codes**
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Activation completed (or policy already active). |
|
||||
| `75` | Activation recorded but awaiting a second approver. |
|
||||
|
||||
### 3.5 Archive / Rollback
|
||||
|
||||
```
|
||||
stella policy archive P-7 --version 3 --reason "Superseded by v4"
|
||||
stella policy activate P-7 --version 3 --rollback --incident INC-2025-104
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Simulation & Runs
|
||||
|
||||
### 4.1 Simulate
|
||||
|
||||
```
|
||||
stella policy simulate P-7 \
|
||||
--base 3 --candidate 4 \
|
||||
--sbom sbom:S-42 --sbom sbom:S-318 \
|
||||
--env exposure=internet --env sealed=false \
|
||||
--format json --output sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Output fields (JSON):
|
||||
|
||||
```json
|
||||
{
|
||||
"diff": {
|
||||
"added": 12,
|
||||
"removed": 8,
|
||||
"unchanged": 657,
|
||||
"bySeverity": {
|
||||
"Critical": {"up": 1, "down": 0},
|
||||
"High": {"up": 3, "down": 4}
|
||||
}
|
||||
},
|
||||
"explainUri": "blob://policy/P-7/simulations/2025-10-26.json"
|
||||
}
|
||||
```
|
||||
|
||||
> Schema reminder: CLI commands surface objects defined in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`; use the samples in `samples/api/scheduler/` for contract validation when extending output parsing.
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Simulation succeeded; diffs informational. |
|
||||
| `20` | Blocking delta (`--fail-on-diff` triggered). |
|
||||
| `21` | Simulation input missing (`ERR_POL_003`). |
|
||||
| `22` | Determinism guard (`ERR_POL_004`). |
|
||||
| `23` | API/permission error (`ERR_POL_002`, `ERR_POL_005`). |
|
||||
|
||||
### 4.2 Run
|
||||
|
||||
```
|
||||
stella policy run P-7 --mode full \
|
||||
--sbom sbom:S-42 --env exposure=internal-only \
|
||||
--wait --watch
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--mode` | `full` or `incremental` (default incremental). |
|
||||
| `--sbom` | Explicit SBOM IDs (optional). |
|
||||
| `--priority` | `normal`, `high`, `emergency`. |
|
||||
| `--wait` | Poll run status until completion. |
|
||||
| `--watch` | Stream progress events (requires TTY). |
|
||||
|
||||
`stella policy run status <runId>` retrieves run metadata.
|
||||
`stella policy run list --status failed --limit 20` returns recent runs.
|
||||
|
||||
### 4.3 Replay & Cancel
|
||||
|
||||
```
|
||||
stella policy run replay run:P-7:2025-10-26:auto --output bundles/replay.tgz
|
||||
stella policy run cancel run:P-7:2025-10-26:auto
|
||||
```
|
||||
|
||||
Replay downloads sealed bundle for deterministic verification.
|
||||
|
||||
### 4.4 Schema artefacts for CLI validation
|
||||
|
||||
- CI publishes canonical JSON Schema exports for `PolicyRunRequest`, `PolicyRunStatus`, `PolicyDiffSummary`, and `PolicyExplainTrace` as the `policy-schema-exports` artifact (see `.gitea/workflows/build-test-deploy.yml`).
|
||||
- Each run writes the files to `artifacts/policy-schemas/<commit>/` and stores a unified diff (`policy-schema-diff.patch`) comparing them with the tracked baseline in `docs/schemas/`.
|
||||
- Schema changes trigger an alert in Slack `#policy-engine` via the `POLICY_ENGINE_SCHEMA_WEBHOOK` secret so CLI maintainers know to refresh fixtures or validation rules.
|
||||
- Consume these artefacts in CLI tests to keep payload validation aligned without committing generated files into the repo.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Findings & Explainability
|
||||
|
||||
### 5.1 List Findings
|
||||
|
||||
```
|
||||
stella findings ls --policy P-7 \
|
||||
--sbom sbom:S-42 \
|
||||
--status affected --severity High,Critical \
|
||||
--since 2025-10-01T00:00:00Z \
|
||||
--page 2 --page-size 100 \
|
||||
--format table
|
||||
```
|
||||
|
||||
Common flags:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--sbom` | Repeatable filter for SBOM identifiers. |
|
||||
| `--status` | Repeatable filter (`affected`, `quieted`, `mitigated`, `not_affected`, etc.). |
|
||||
| `--severity` | Repeatable filter using normalized labels (`Critical`, `High`, `Medium`, `Low`, `Unknown`). |
|
||||
| `--since` | Return findings updated on/after the ISO-8601 timestamp. |
|
||||
| `--cursor` | Resume listing using the opaque token from a prior page. |
|
||||
| `--page`, `--page-size` | Page-based pagination (page >=1, size <=500; falls back to backend defaults). |
|
||||
| `--output` | Persist JSON payload to disk (implied JSON rendering). |
|
||||
| `--format` | `table` (default for TTY) or `json`. |
|
||||
|
||||
### 5.2 Fetch Explain
|
||||
|
||||
```
|
||||
stella findings explain --policy P-7 \
|
||||
P-7:S-42:pkg:npm/lodash@4.17.21:CVE-2021-23337 \
|
||||
--mode verbose \
|
||||
--format json --output explains/lodash.json
|
||||
```
|
||||
|
||||
Outputs ordered rule hits, inputs, evidence snapshots, and sealed-mode hints. Supported `--mode` values mirror API contracts (for example `summary`, `verbose`); omit to use backend default.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Exit Codes Summary
|
||||
|
||||
| Exit code | Description | Typical ERR codes |
|
||||
|-----------|-------------|-------------------|
|
||||
| `0` | Success (command completed, warnings only). | — |
|
||||
| `10` | DSL syntax/compile failure. | `ERR_POL_001` |
|
||||
| `11` | Unsupported DSL version / schema mismatch. | `ERR_POL_001` |
|
||||
| `12` | Approval/rbac failure. | `ERR_POL_002`, `ERR_POL_005` |
|
||||
| `20` | Simulation diff exceeded thresholds (`--fail-on-diff`). | — |
|
||||
| `21` | Required inputs missing (SBOM/advisory/VEX). | `ERR_POL_003` |
|
||||
| `22` | Determinism guard triggered. | `ERR_POL_004` |
|
||||
| `23` | Run canceled or timed out. | `ERR_POL_006` |
|
||||
| `30` | Network/transport error (non-HTTP success). | — |
|
||||
| `64` | CLI usage error (invalid flag/argument). | — |
|
||||
|
||||
All non-zero exits emit structured error envelope on stderr when `--format json` or `STELLA_JSON_ERRORS=1`.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Offline & Air-Gap Usage
|
||||
|
||||
- Use `--sealed` to ensure commands avoid outbound calls; required for sealed enclaves.
|
||||
- `stella policy bundle export --policy P-7 --version 4 --output bundles/policy-P-7-v4.bundle` pairs with Offline Kit import.
|
||||
- Replay bundles (`run replay`) are DSSE-signed; verify with `stella offline verify`.
|
||||
- Store credentials in `~/.stella/offline.toml` for non-interactive air-gapped pipelines.
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance Checklist
|
||||
|
||||
- [ ] **Help text synced:** `stella policy --help` matches documented flags/examples (update during release pipeline).
|
||||
- [ ] **Exit codes mapped:** Table above reflects CLI implementation and CI asserts mapping for `ERR_POL_*`.
|
||||
- [ ] **JSON schemas verified:** Example payloads validated against OpenAPI/SDK contracts before publishing. (_CI now exports canonical schemas as `policy-schema-exports`; wire tests to consume them._)
|
||||
- [ ] **Scope guidance present:** Each command lists required Authority scopes.
|
||||
- [ ] **Offline guidance included:** Sealed-mode steps and bundle workflows documented.
|
||||
- [ ] **Cross-links tested:** Links to DSL, lifecycle, runs, and API docs resolve locally (`yarn docs:lint`).
|
||||
- [ ] **Examples no-op safe:** Command examples either read-only or use placeholders (no destructive defaults).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 20).*
|
||||
# Stella CLI — Policy Commands
|
||||
|
||||
> **Audience:** Policy authors, reviewers, operators, and CI engineers using the `stella` CLI to interact with Policy Engine.
|
||||
> **Supported from:** `stella` CLI ≥ 0.20.0 (Policy Engine v2 sprint line).
|
||||
> **Prerequisites:** Authority-issued bearer token with the scopes noted per command (export `STELLA_TOKEN` or pass `--token`).
|
||||
> **2025-10-27 scope update:** CLI/CI tokens issued prior to Sprint 23 (AUTH-POLICY-23-001) must drop `policy:write`/`policy:submit`/`policy:edit` and instead request `policy:read`, `policy:author`, `policy:review`, and `policy:simulate` (plus `policy:approve`/`policy:operate`/`policy:activate` for promotion pipelines).
|
||||
|
||||
---
|
||||
|
||||
## 1 · Global Options & Output Modes
|
||||
|
||||
All `stella policy *` commands honour the common CLI options:
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--server <url>` | `https://stella.local` | Policy Engine gateway root. |
|
||||
| `--tenant <id>` | token default | Override tenant for multi-tenant installs. |
|
||||
| `--format <table\|json\|yaml>` | `table` for TTY, `json` otherwise | Output format for listings/diffs. |
|
||||
| `--output <file>` | stdout | Write full JSON payload to file. |
|
||||
| `--sealed` | false | Force sealed-mode behaviour (no outbound fetch). |
|
||||
| `--trace` | false | Emit verbose timing/log correlation info. |
|
||||
|
||||
> **Tip:** Set `STELLA_PROFILE=policy` in CI to load saved defaults from `~/.stella/profiles/policy.toml`.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Authoring & Drafting Commands
|
||||
|
||||
### 2.1 `stella policy new`
|
||||
|
||||
Create a draft policy from a template or scratch.
|
||||
|
||||
```
|
||||
stella policy new --policy-id P-7 --name "Default Org Policy" \
|
||||
--template baseline --output-path policies/P-7.stella
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--policy-id` *(required)* | Stable identifier (e.g., `P-7`). |
|
||||
| `--name` | Friendly display name. |
|
||||
| `--template` | `baseline`, `serverless`, `blank`. |
|
||||
| `--from` | Start from existing version (`policyId@version`). |
|
||||
| `--open` | Launches `$EDITOR` after creation. |
|
||||
|
||||
Writes DSL to local file and registers draft version (`status=draft`). Requires `policy:write`.
|
||||
|
||||
### 2.2 `stella policy edit`
|
||||
|
||||
Open an existing draft in the local editor.
|
||||
|
||||
```
|
||||
stella policy edit P-7 --version 4
|
||||
```
|
||||
|
||||
- Auto-checks out latest draft if `--version` omitted.
|
||||
- Saves to temp file, uploads on editor exit (unless `--no-upload`).
|
||||
- Use `--watch` to keep command alive and re-upload on every save.
|
||||
|
||||
### 2.3 `stella policy lint`
|
||||
|
||||
Static validation without submitting.
|
||||
|
||||
```
|
||||
stella policy lint policies/P-7.stella --format json
|
||||
```
|
||||
|
||||
Outputs diagnostics (line/column, code, message). Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | No lint errors. |
|
||||
| `10` | Syntax/compile errors (`ERR_POL_001`). |
|
||||
| `11` | Unsupported syntax version. |
|
||||
|
||||
### 2.4 `stella policy compile`
|
||||
|
||||
Emits IR digest and rule summary.
|
||||
|
||||
```
|
||||
stella policy compile P-7 --version 4
|
||||
```
|
||||
|
||||
Returns JSON with `digest`, `rules.count`, action counts. Exit `0` success, `10` on compile errors.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Lifecycle Workflow
|
||||
|
||||
### 3.1 Submit
|
||||
|
||||
```
|
||||
stella policy submit P-7 --version 4 \
|
||||
--reviewer user:kay --reviewer group:sec-reviewers \
|
||||
--note "Simulated against golden SBOM set" \
|
||||
--attach sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Requires `policy:submit`. CLI validates that lint/compile run within 24 h and bundle attachments exist.
|
||||
|
||||
### 3.2 Review
|
||||
|
||||
```
|
||||
stella policy review P-7 --version 4 --approve \
|
||||
--note "Looks good; ensure incident playbook updated."
|
||||
```
|
||||
|
||||
- `--approve`, `--request-changes`, or `--comment`.
|
||||
- Provide `--blocking` to mark comment as blocking.
|
||||
- Requires `policy:review`.
|
||||
|
||||
### 3.3 Approve
|
||||
|
||||
```
|
||||
stella policy approve P-7 --version 4 \
|
||||
--note "Determinism CI green; simulation diff attached." \
|
||||
--attach sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Prompts for confirmation; refuses if approver == submitter. Requires `policy:approve`.
|
||||
|
||||
### 3.4 Activate
|
||||
|
||||
```
|
||||
stella policy activate P-7 --version 4 --run-now --priority high
|
||||
```
|
||||
|
||||
- Optional `--scheduled-at 2025-10-27T02:00:00Z`.
|
||||
- Requires `policy:activate` and `policy:run`.
|
||||
|
||||
**Options**
|
||||
|
||||
- `--version <number>` (required) – target revision to promote.
|
||||
- `--note <text>` – record an activation note alongside the approval.
|
||||
- `--run-now` – enqueue an immediate full run after activation.
|
||||
- `--scheduled-at <timestamp>` – schedule activation for a specific UTC time (ISO-8601 format).
|
||||
- `--priority <label>` – optional scheduling priority hint (`low`, `standard`, `high`).
|
||||
- `--rollback` – mark the activation as a rollback of a previously active version.
|
||||
- `--incident <id>` – associate the activation with an incident identifier.
|
||||
|
||||
**Exit codes**
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Activation completed (or policy already active). |
|
||||
| `75` | Activation recorded but awaiting a second approver. |
|
||||
|
||||
### 3.5 Archive / Rollback
|
||||
|
||||
```
|
||||
stella policy archive P-7 --version 3 --reason "Superseded by v4"
|
||||
stella policy activate P-7 --version 3 --rollback --incident INC-2025-104
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · Simulation & Runs
|
||||
|
||||
### 4.1 Simulate
|
||||
|
||||
```
|
||||
stella policy simulate P-7 \
|
||||
--base 3 --candidate 4 \
|
||||
--sbom sbom:S-42 --sbom sbom:S-318 \
|
||||
--env exposure=internet --env sealed=false \
|
||||
--format json --output sims/P-7-v4-vs-v3.json
|
||||
```
|
||||
|
||||
Output fields (JSON):
|
||||
|
||||
```json
|
||||
{
|
||||
"diff": {
|
||||
"added": 12,
|
||||
"removed": 8,
|
||||
"unchanged": 657,
|
||||
"bySeverity": {
|
||||
"Critical": {"up": 1, "down": 0},
|
||||
"High": {"up": 3, "down": 4}
|
||||
}
|
||||
},
|
||||
"explainUri": "blob://policy/P-7/simulations/2025-10-26.json"
|
||||
}
|
||||
```
|
||||
|
||||
> Schema reminder: CLI commands surface objects defined in `src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`; use the samples in `samples/api/scheduler/` for contract validation when extending output parsing.
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Simulation succeeded; diffs informational. |
|
||||
| `20` | Blocking delta (`--fail-on-diff` triggered). |
|
||||
| `21` | Simulation input missing (`ERR_POL_003`). |
|
||||
| `22` | Determinism guard (`ERR_POL_004`). |
|
||||
| `23` | API/permission error (`ERR_POL_002`, `ERR_POL_005`). |
|
||||
|
||||
### 4.2 Run
|
||||
|
||||
```
|
||||
stella policy run P-7 --mode full \
|
||||
--sbom sbom:S-42 --env exposure=internal-only \
|
||||
--wait --watch
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--mode` | `full` or `incremental` (default incremental). |
|
||||
| `--sbom` | Explicit SBOM IDs (optional). |
|
||||
| `--priority` | `normal`, `high`, `emergency`. |
|
||||
| `--wait` | Poll run status until completion. |
|
||||
| `--watch` | Stream progress events (requires TTY). |
|
||||
|
||||
`stella policy run status <runId>` retrieves run metadata.
|
||||
`stella policy run list --status failed --limit 20` returns recent runs.
|
||||
|
||||
### 4.3 Replay & Cancel
|
||||
|
||||
```
|
||||
stella policy run replay run:P-7:2025-10-26:auto --output bundles/replay.tgz
|
||||
stella policy run cancel run:P-7:2025-10-26:auto
|
||||
```
|
||||
|
||||
Replay downloads sealed bundle for deterministic verification.
|
||||
|
||||
### 4.4 Schema artefacts for CLI validation
|
||||
|
||||
- CI publishes canonical JSON Schema exports for `PolicyRunRequest`, `PolicyRunStatus`, `PolicyDiffSummary`, and `PolicyExplainTrace` as the `policy-schema-exports` artifact (see `.gitea/workflows/build-test-deploy.yml`).
|
||||
- Each run writes the files to `artifacts/policy-schemas/<commit>/` and stores a unified diff (`policy-schema-diff.patch`) comparing them with the tracked baseline in `docs/schemas/`.
|
||||
- Schema changes trigger an alert in Slack `#policy-engine` via the `POLICY_ENGINE_SCHEMA_WEBHOOK` secret so CLI maintainers know to refresh fixtures or validation rules.
|
||||
- Consume these artefacts in CLI tests to keep payload validation aligned without committing generated files into the repo.
|
||||
|
||||
---
|
||||
|
||||
## 5 · Findings & Explainability
|
||||
|
||||
### 5.1 List Findings
|
||||
|
||||
```
|
||||
stella findings ls --policy P-7 \
|
||||
--sbom sbom:S-42 \
|
||||
--status affected --severity High,Critical \
|
||||
--since 2025-10-01T00:00:00Z \
|
||||
--page 2 --page-size 100 \
|
||||
--format table
|
||||
```
|
||||
|
||||
Common flags:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--sbom` | Repeatable filter for SBOM identifiers. |
|
||||
| `--status` | Repeatable filter (`affected`, `quieted`, `mitigated`, `not_affected`, etc.). |
|
||||
| `--severity` | Repeatable filter using normalized labels (`Critical`, `High`, `Medium`, `Low`, `Unknown`). |
|
||||
| `--since` | Return findings updated on/after the ISO-8601 timestamp. |
|
||||
| `--cursor` | Resume listing using the opaque token from a prior page. |
|
||||
| `--page`, `--page-size` | Page-based pagination (page >=1, size <=500; falls back to backend defaults). |
|
||||
| `--output` | Persist JSON payload to disk (implied JSON rendering). |
|
||||
| `--format` | `table` (default for TTY) or `json`. |
|
||||
|
||||
### 5.2 Fetch Explain
|
||||
|
||||
```
|
||||
stella findings explain --policy P-7 \
|
||||
P-7:S-42:pkg:npm/lodash@4.17.21:CVE-2021-23337 \
|
||||
--mode verbose \
|
||||
--format json --output explains/lodash.json
|
||||
```
|
||||
|
||||
Outputs ordered rule hits, inputs, evidence snapshots, and sealed-mode hints. Supported `--mode` values mirror API contracts (for example `summary`, `verbose`); omit to use backend default.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Exit Codes Summary
|
||||
|
||||
| Exit code | Description | Typical ERR codes |
|
||||
|-----------|-------------|-------------------|
|
||||
| `0` | Success (command completed, warnings only). | — |
|
||||
| `10` | DSL syntax/compile failure. | `ERR_POL_001` |
|
||||
| `11` | Unsupported DSL version / schema mismatch. | `ERR_POL_001` |
|
||||
| `12` | Approval/rbac failure. | `ERR_POL_002`, `ERR_POL_005` |
|
||||
| `20` | Simulation diff exceeded thresholds (`--fail-on-diff`). | — |
|
||||
| `21` | Required inputs missing (SBOM/advisory/VEX). | `ERR_POL_003` |
|
||||
| `22` | Determinism guard triggered. | `ERR_POL_004` |
|
||||
| `23` | Run canceled or timed out. | `ERR_POL_006` |
|
||||
| `30` | Network/transport error (non-HTTP success). | — |
|
||||
| `64` | CLI usage error (invalid flag/argument). | — |
|
||||
|
||||
All non-zero exits emit structured error envelope on stderr when `--format json` or `STELLA_JSON_ERRORS=1`.
|
||||
|
||||
---
|
||||
|
||||
## 7 · Offline & Air-Gap Usage
|
||||
|
||||
- Use `--sealed` to ensure commands avoid outbound calls; required for sealed enclaves.
|
||||
- `stella policy bundle export --policy P-7 --version 4 --output bundles/policy-P-7-v4.bundle` pairs with Offline Kit import.
|
||||
- Replay bundles (`run replay`) are DSSE-signed; verify with `stella offline verify`.
|
||||
- Store credentials in `~/.stella/offline.toml` for non-interactive air-gapped pipelines.
|
||||
|
||||
---
|
||||
|
||||
## 8 · Compliance Checklist
|
||||
|
||||
- [ ] **Help text synced:** `stella policy --help` matches documented flags/examples (update during release pipeline).
|
||||
- [ ] **Exit codes mapped:** Table above reflects CLI implementation and CI asserts mapping for `ERR_POL_*`.
|
||||
- [ ] **JSON schemas verified:** Example payloads validated against OpenAPI/SDK contracts before publishing. (_CI now exports canonical schemas as `policy-schema-exports`; wire tests to consume them._)
|
||||
- [ ] **Scope guidance present:** Each command lists required Authority scopes.
|
||||
- [ ] **Offline guidance included:** Sealed-mode steps and bundle workflows documented.
|
||||
- [ ] **Cross-links tested:** Links to DSL, lifecycle, runs, and API docs resolve locally (`yarn docs:lint`).
|
||||
- [ ] **Examples no-op safe:** Command examples either read-only or use placeholders (no destructive defaults).
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 20).*
|
||||
|
||||
@@ -1,23 +1,23 @@
|
||||
# Implementation plan — CLI
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 2 – Policy Engine & Editor:** deliver deterministic policy verbs, simulation, and explain outputs.
|
||||
- **Epic 4 – Policy Studio:** add registry/promotion workflows, lint tooling, and approvals UX.
|
||||
- **Epic 6 – Vulnerability Explorer:** integrate ledger/triage operations.
|
||||
- **Epic 10 – Export Center:** automate export verification and Offline Kit flows.
|
||||
- **Epic 11 – Notifications Studio:** manage rule/channel authoring and previews via CLI.
|
||||
- Track CLI-specific work (e.g., CLI-CORE-41-001, DOCS-CLI-OBS-52-001) in ../../TASKS.md and src/Cli/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — CLI
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 2 – Policy Engine & Editor:** deliver deterministic policy verbs, simulation, and explain outputs.
|
||||
- **Epic 4 – Policy Studio:** add registry/promotion workflows, lint tooling, and approvals UX.
|
||||
- **Epic 6 – Vulnerability Explorer:** integrate ledger/triage operations.
|
||||
- **Epic 10 – Export Center:** automate export verification and Offline Kit flows.
|
||||
- **Epic 11 – Notifications Studio:** manage rule/channel authoring and previews via CLI.
|
||||
- Track CLI-specific work (e.g., CLI-CORE-41-001, DOCS-CLI-OBS-52-001) in ../../TASKS.md and src/Cli/**/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,134 +1,134 @@
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
# CLI Release & Packaging Runbook
|
||||
|
||||
This runbook describes how to build, sign, package, and distribute the StellaOps CLI with Task Pack support. It covers connected and air-gapped workflows, SBOM generation, parity gating, and distribution artifacts required by Sprint 43 (`DEVOPS-CLI-43-001`, `DEPLOY-PACKS-43-001`).
|
||||
|
||||
---
|
||||
|
||||
## 1 · Release Artifacts
|
||||
|
||||
| Artifact | Description | Notes |
|
||||
|----------|-------------|-------|
|
||||
| `stella-<version>-linux-x64.tar.gz` | Linux binary + completions | Includes man pages, localization files. |
|
||||
| `stella-<version>-macos-universal.tar.gz` | macOS universal binary | Signed/notarized where applicable. |
|
||||
| `stella-<version>-windows-x64.zip` | Windows binary + PowerShell modules | Code-signed. |
|
||||
| `stella-cli-container:<version>` | OCI image with CLI + pack runtime | Deterministic rootfs (scratch/distroless). |
|
||||
| SBOM (`.cdx.json`) | CycloneDX SBOM per artifact | Generated via `stella sbom generate` or `syft`. |
|
||||
| Checksums (`SHA256SUMS`) | Aggregated digest list | Signed with cosign. |
|
||||
| Provenance (`.intoto.jsonl`) | DSSE attestation (SLSA L2) | Contains build metadata. |
|
||||
| Release notes | Markdown summary | Links to task packs docs, parity matrix. |
|
||||
|
||||
---
|
||||
|
||||
## 2 · Build Pipeline
|
||||
|
||||
1. **Source checkout** – pinned commit, reproducible environment (Docker).
|
||||
2. **Dependency lock** – `dotnet restore`, `npm ci` (for CLI frontends), ensure deterministic build flags.
|
||||
3. **Build binaries** – cross-platform targets with reproducible timestamps.
|
||||
4. **Run tests** – unit + integration; include `stella pack` commands (plan/run/verify) in CI.
|
||||
5. **Generate SBOM** – `syft packages dist/stella-linux-x64 --output cyclonedx-json`.
|
||||
6. **Bundle** – compress artifacts, include completions (`bash`, `zsh`, `fish`, PowerShell).
|
||||
7. **Sign** – cosign signatures for binaries, checksums, container image.
|
||||
8. **Publish** – upload to `downloads.stella-ops.org`, container registry, Packs Registry (for CLI container).
|
||||
9. **Parity gating** – run CLI parity matrix tests vs Console features (automation in `DEVOPS-CLI-43-001`).
|
||||
|
||||
CI must run in isolated environment (no network beyond allowlist). Cache dependencies for offline bundling.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Versioning & Channels
|
||||
|
||||
- Semantic versioning (`YYYY.MM.patch`), e.g., `2025.10.0`.
|
||||
- Channels:
|
||||
- `edge` – nightly builds, limited support.
|
||||
- `beta` – pre-release candidates.
|
||||
- `stable` – production-ready, after parity gating.
|
||||
- Release promotions mirror Task Pack channels; update downloads manifest (`deploy/downloads/manifest.json`).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Signing & Verification
|
||||
|
||||
- Binaries signed with cosign (`cosign sign-blob`).
|
||||
- Container image signed (`cosign sign stella-cli-container:<version>`).
|
||||
- DSSE provenance includes:
|
||||
- Build pipeline ID.
|
||||
- Source commit and repo.
|
||||
- Dependencies SBOM digest.
|
||||
- Test results summary.
|
||||
- Verification command for operators:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--certificate-identity https://ci.stella-ops.org \
|
||||
--certificate-oidc-issuer https://fulcio.sigstore.dev \
|
||||
--signature stella-2025.10.0-linux-x64.sig \
|
||||
stella-2025.10.0-linux-x64.tar.gz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5 · Distribution
|
||||
|
||||
### 5.1 Online
|
||||
|
||||
- Publish artifacts to Downloads service; update manifest with digests, SBOM URLs, attestations.
|
||||
- Update CLI parity docs (`docs/cli-vs-ui-parity.md`) and release notes.
|
||||
- Push container image to registry with SBOM + attestations referenced as OCI referrers.
|
||||
- Notify stakeholders via `#release-cli` channel and release mailing list.
|
||||
|
||||
### 5.2 Offline / Air-Gap
|
||||
|
||||
- Bundle CLI artifacts, Task Pack samples, and registry mirror:
|
||||
|
||||
```bash
|
||||
stella pack bundle export \
|
||||
--packs "sbom-remediation:1.3.0" \
|
||||
--output offline/packs-bundle-2025.10.0.tgz
|
||||
|
||||
stella cli bundle export \
|
||||
--output offline/cli-2025.10.0.tgz \
|
||||
--include-container \
|
||||
--include-sbom
|
||||
```
|
||||
|
||||
- Update Offline Kit manifest with new CLI version and pack bundle entries.
|
||||
- Provide import scripts (`ouk import`) for sealed sites.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Parity Gating
|
||||
|
||||
- `stella cli parity check` compares CLI commands vs parity matrix.
|
||||
- CI fails release if any required command flagged `🟥` or `🟡` with severity > threshold.
|
||||
- Parity report uploaded to Downloads workspace and linked in docs.
|
||||
- Manual review required for new commands (ensure `man` pages and help text localized).
|
||||
|
||||
---
|
||||
|
||||
## 7 · Localization & Documentation
|
||||
|
||||
- CLI includes localization bundles; ensure `i18n.txz` packaged.
|
||||
- Update man pages (`man/stella-pack.1`) and HTML docs.
|
||||
- Sync docs: `docs/modules/cli/guides/overview.md`, pack authoring guide, release notes.
|
||||
- Document new flags/commands in `docs/modules/cli/guides/commands/pack.md` (tracked in Sprint 42 tasks).
|
||||
|
||||
---
|
||||
|
||||
## 8 · Release Checklist
|
||||
|
||||
- [ ] All binaries built reproducibly (CI logs archived).
|
||||
- [ ] Tests + parity matrix passing.
|
||||
- [ ] SBOM + provenance generated and published.
|
||||
- [ ] Cosign signatures created and verified.
|
||||
- [ ] Downloads manifest updated (edge/beta/stable).
|
||||
- [ ] Offline bundle exported and validated.
|
||||
- [ ] Release notes + documentation updates merged.
|
||||
- [ ] Notifications sent (chat/email).
|
||||
- [ ] Imposed rule reminder present at top of document.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 43).*
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
# CLI Release & Packaging Runbook
|
||||
|
||||
This runbook describes how to build, sign, package, and distribute the StellaOps CLI with Task Pack support. It covers connected and air-gapped workflows, SBOM generation, parity gating, and distribution artifacts required by Sprint 43 (`DEVOPS-CLI-43-001`, `DEPLOY-PACKS-43-001`).
|
||||
|
||||
---
|
||||
|
||||
## 1 · Release Artifacts
|
||||
|
||||
| Artifact | Description | Notes |
|
||||
|----------|-------------|-------|
|
||||
| `stella-<version>-linux-x64.tar.gz` | Linux binary + completions | Includes man pages, localization files. |
|
||||
| `stella-<version>-macos-universal.tar.gz` | macOS universal binary | Signed/notarized where applicable. |
|
||||
| `stella-<version>-windows-x64.zip` | Windows binary + PowerShell modules | Code-signed. |
|
||||
| `stella-cli-container:<version>` | OCI image with CLI + pack runtime | Deterministic rootfs (scratch/distroless). |
|
||||
| SBOM (`.cdx.json`) | CycloneDX SBOM per artifact | Generated via `stella sbom generate` or `syft`. |
|
||||
| Checksums (`SHA256SUMS`) | Aggregated digest list | Signed with cosign. |
|
||||
| Provenance (`.intoto.jsonl`) | DSSE attestation (SLSA L2) | Contains build metadata. |
|
||||
| Release notes | Markdown summary | Links to task packs docs, parity matrix. |
|
||||
|
||||
---
|
||||
|
||||
## 2 · Build Pipeline
|
||||
|
||||
1. **Source checkout** – pinned commit, reproducible environment (Docker).
|
||||
2. **Dependency lock** – `dotnet restore`, `npm ci` (for CLI frontends), ensure deterministic build flags.
|
||||
3. **Build binaries** – cross-platform targets with reproducible timestamps.
|
||||
4. **Run tests** – unit + integration; include `stella pack` commands (plan/run/verify) in CI.
|
||||
5. **Generate SBOM** – `syft packages dist/stella-linux-x64 --output cyclonedx-json`.
|
||||
6. **Bundle** – compress artifacts, include completions (`bash`, `zsh`, `fish`, PowerShell).
|
||||
7. **Sign** – cosign signatures for binaries, checksums, container image.
|
||||
8. **Publish** – upload to `downloads.stella-ops.org`, container registry, Packs Registry (for CLI container).
|
||||
9. **Parity gating** – run CLI parity matrix tests vs Console features (automation in `DEVOPS-CLI-43-001`).
|
||||
|
||||
CI must run in isolated environment (no network beyond allowlist). Cache dependencies for offline bundling.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Versioning & Channels
|
||||
|
||||
- Semantic versioning (`YYYY.MM.patch`), e.g., `2025.10.0`.
|
||||
- Channels:
|
||||
- `edge` – nightly builds, limited support.
|
||||
- `beta` – pre-release candidates.
|
||||
- `stable` – production-ready, after parity gating.
|
||||
- Release promotions mirror Task Pack channels; update downloads manifest (`deploy/downloads/manifest.json`).
|
||||
|
||||
---
|
||||
|
||||
## 4 · Signing & Verification
|
||||
|
||||
- Binaries signed with cosign (`cosign sign-blob`).
|
||||
- Container image signed (`cosign sign stella-cli-container:<version>`).
|
||||
- DSSE provenance includes:
|
||||
- Build pipeline ID.
|
||||
- Source commit and repo.
|
||||
- Dependencies SBOM digest.
|
||||
- Test results summary.
|
||||
- Verification command for operators:
|
||||
|
||||
```bash
|
||||
cosign verify-blob \
|
||||
--certificate-identity https://ci.stella-ops.org \
|
||||
--certificate-oidc-issuer https://fulcio.sigstore.dev \
|
||||
--signature stella-2025.10.0-linux-x64.sig \
|
||||
stella-2025.10.0-linux-x64.tar.gz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5 · Distribution
|
||||
|
||||
### 5.1 Online
|
||||
|
||||
- Publish artifacts to Downloads service; update manifest with digests, SBOM URLs, attestations.
|
||||
- Update CLI parity docs (`docs/cli-vs-ui-parity.md`) and release notes.
|
||||
- Push container image to registry with SBOM + attestations referenced as OCI referrers.
|
||||
- Notify stakeholders via `#release-cli` channel and release mailing list.
|
||||
|
||||
### 5.2 Offline / Air-Gap
|
||||
|
||||
- Bundle CLI artifacts, Task Pack samples, and registry mirror:
|
||||
|
||||
```bash
|
||||
stella pack bundle export \
|
||||
--packs "sbom-remediation:1.3.0" \
|
||||
--output offline/packs-bundle-2025.10.0.tgz
|
||||
|
||||
stella cli bundle export \
|
||||
--output offline/cli-2025.10.0.tgz \
|
||||
--include-container \
|
||||
--include-sbom
|
||||
```
|
||||
|
||||
- Update Offline Kit manifest with new CLI version and pack bundle entries.
|
||||
- Provide import scripts (`ouk import`) for sealed sites.
|
||||
|
||||
---
|
||||
|
||||
## 6 · Parity Gating
|
||||
|
||||
- `stella cli parity check` compares CLI commands vs parity matrix.
|
||||
- CI fails release if any required command flagged `🟥` or `🟡` with severity > threshold.
|
||||
- Parity report uploaded to Downloads workspace and linked in docs.
|
||||
- Manual review required for new commands (ensure `man` pages and help text localized).
|
||||
|
||||
---
|
||||
|
||||
## 7 · Localization & Documentation
|
||||
|
||||
- CLI includes localization bundles; ensure `i18n.txz` packaged.
|
||||
- Update man pages (`man/stella-pack.1`) and HTML docs.
|
||||
- Sync docs: `docs/modules/cli/guides/overview.md`, pack authoring guide, release notes.
|
||||
- Document new flags/commands in `docs/modules/cli/guides/commands/pack.md` (tracked in Sprint 42 tasks).
|
||||
|
||||
---
|
||||
|
||||
## 8 · Release Checklist
|
||||
|
||||
- [ ] All binaries built reproducibly (CI logs archived).
|
||||
- [ ] Tests + parity matrix passing.
|
||||
- [ ] SBOM + provenance generated and published.
|
||||
- [ ] Cosign signatures created and verified.
|
||||
- [ ] Downloads manifest updated (edge/beta/stable).
|
||||
- [ ] Offline bundle exported and validated.
|
||||
- [ ] Release notes + documentation updates merged.
|
||||
- [ ] Notifications sent (chat/email).
|
||||
- [ ] Imposed rule reminder present at top of document.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2025-10-27 (Sprint 43).*
|
||||
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Concelier agent guide
|
||||
|
||||
## Mission
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# Concelier agent guide
|
||||
|
||||
## Mission
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,36 +1,36 @@
|
||||
# StellaOps Concelier
|
||||
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Responsibilities
|
||||
- Fetch and normalise vulnerability advisories via restart-time connectors.
|
||||
- Persist observations and correlation linksets without precedence decisions.
|
||||
- Emit deterministic exports (JSON, Trivy DB) for downstream policy evaluation.
|
||||
- Coordinate offline/air-gap updates via Offline Kit bundles.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Concelier.WebService` orchestration host.
|
||||
- Connector libraries under `StellaOps.Concelier.Connector.*`.
|
||||
- Exporter packages (`StellaOps.Concelier.Exporter.*`).
|
||||
|
||||
## Integrations & dependencies
|
||||
- MongoDB for canonical observations and schedules.
|
||||
- Policy Engine / Export Center / CLI for evidence consumption.
|
||||
- Notify and UI for advisory deltas.
|
||||
|
||||
## Operational notes
|
||||
- Connector runbooks in ./operations/connectors/.
|
||||
- Mirror operations for Offline Kit parity.
|
||||
- Grafana dashboards for connector health.
|
||||
|
||||
## Related resources
|
||||
- ./operations/conflict-resolution.md
|
||||
- ./operations/mirror.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-LNM-22-001, DOCS-LNM-22-007 in ../../TASKS.md.
|
||||
- Connector-specific TODOs in `src/Concelier/**/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
|
||||
- **Epic 10 – Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.
|
||||
# StellaOps Concelier
|
||||
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Responsibilities
|
||||
- Fetch and normalise vulnerability advisories via restart-time connectors.
|
||||
- Persist observations and correlation linksets without precedence decisions.
|
||||
- Emit deterministic exports (JSON, Trivy DB) for downstream policy evaluation.
|
||||
- Coordinate offline/air-gap updates via Offline Kit bundles.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Concelier.WebService` orchestration host.
|
||||
- Connector libraries under `StellaOps.Concelier.Connector.*`.
|
||||
- Exporter packages (`StellaOps.Concelier.Exporter.*`).
|
||||
|
||||
## Integrations & dependencies
|
||||
- MongoDB for canonical observations and schedules.
|
||||
- Policy Engine / Export Center / CLI for evidence consumption.
|
||||
- Notify and UI for advisory deltas.
|
||||
|
||||
## Operational notes
|
||||
- Connector runbooks in ./operations/connectors/.
|
||||
- Mirror operations for Offline Kit parity.
|
||||
- Grafana dashboards for connector health.
|
||||
|
||||
## Related resources
|
||||
- ./operations/conflict-resolution.md
|
||||
- ./operations/mirror.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-LNM-22-001, DOCS-LNM-22-007 in ../../TASKS.md.
|
||||
- Connector-specific TODOs in `src/Concelier/**/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
|
||||
- **Epic 10 – Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — Concelier
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CONCELIER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CONCELIER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CONCELIER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — Concelier
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| CONCELIER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| CONCELIER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| CONCELIER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,67 +1,67 @@
|
||||
# Implementation plan — Concelier
|
||||
|
||||
## Delivery timeline
|
||||
- **Phase 1 — Guardrails & schema**
|
||||
Stand up Mongo JSON validators for `advisory_raw` and `vex_raw`, wire the `AOCWriteGuard` repository interceptor, and seed deterministic linkset builders. Freeze legacy normalisation paths and migrate callers to the new raw schema.
|
||||
- **Phase 2 — API & observability**
|
||||
Publish ingestion and verification endpoints (`POST /ingest/*`, `GET /advisories.raw`, `POST /aoc/verify`) with Authority scopes, expose telemetry (`aoc_violation_total`, guard spans, structured logs), and ensure Offline Kit packaging captures validator deployment steps.
|
||||
- **Phase 3 — Experience polish**
|
||||
Ship CLI/Console affordances (`stella sources ingest --dry-run`, dashboard tiles, violation drill-downs), finish Export Center hand-off metadata, and close out CI enforcement (`stella aoc verify` preflight, AST lint, seeded fixtures).
|
||||
|
||||
## Work breakdown by component
|
||||
- **Concelier WebService & worker**
|
||||
- Add Mongo validators and unique indexes over `(tenant, source.vendor, upstream.upstream_id, upstream.content_hash)`.
|
||||
- Implement write interceptors rejecting forbidden fields, missing provenance, or merge attempts.
|
||||
- Deterministically compute linksets and persist canonical JSON payloads.
|
||||
- Introduce `/ingest/advisory`, `/advisories/raw*`, and `/aoc/verify` surfaces guarded by `advisory:*` and `aoc:verify` scopes.
|
||||
- Emit guard metrics/traces and surface supersedes/violation audit logs.
|
||||
- **Excititor (shared ingestion contract)**
|
||||
- Mirror Concelier guard and schema changes for `vex_raw`.
|
||||
- Maintain restart-time plug-in determinism and linkset extraction parity.
|
||||
- **Shared libraries**
|
||||
- Publish `StellaOps.Ingestion.AOC` (forbidden key catalog, guard middleware, provenance helpers, signature verification).
|
||||
- Share error codes (`ERR_AOC_00x`) and deterministic hashing utilities.
|
||||
- **Policy Engine integration**
|
||||
- Enforce `effective_finding_*` write exclusivity.
|
||||
- Consume only raw documents + linksets, removing any implicit normalisation.
|
||||
- **Authority scopes**
|
||||
- Provision `advisory:ingest|read`, `vex:ingest|read`, `aoc:verify`; propagate tenant claims to ingestion services.
|
||||
- **CLI & Console**
|
||||
- Implement `stella sources ingest --dry-run` and `stella aoc verify` (with exit codes mapped to `ERR_AOC_00x`).
|
||||
- Surface AOC dashboards, violation drill-down, and verification shortcuts in the Console.
|
||||
- **CI/CD**
|
||||
- Add Roslyn analyzer / AST lint to block forbidden writes.
|
||||
- Seed fixtures and run `stella aoc verify` against snapshots in pipeline gating.
|
||||
|
||||
## Documentation deliverables
|
||||
- Update `docs/ingestion/aggregation-only-contract.md` with guard invariants, schemas, error codes, and migration guidance.
|
||||
- Refresh `docs/modules/concelier/operations/*.md` (mirror, conflict-resolution, authority audit) with validator rollouts and observability dashboards.
|
||||
- Cross-link Authority scope definitions, CLI reference, Console sources guide, and observability runbooks to the AOC guard changes.
|
||||
- Ensure Offline Kit documentation captures validator bootstrap and verify workflows.
|
||||
|
||||
## Acceptance criteria
|
||||
- Mongo validators and runtime guards reject forbidden fields and missing provenance with the documented `ERR_AOC_00x` codes.
|
||||
- Linksets and supersedes chains are deterministic; rerunning ingestion over identical payloads yields byte-identical documents.
|
||||
- CLI `stella aoc verify` exits non-zero on seeded violations and zero on clean datasets; Console dashboards show real-time guard status.
|
||||
- Export Center consumes advisory datasets without relying on legacy normalised fields.
|
||||
- CI fails if lint rules detect forbidden writes or if seeded guard tests regress.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Collector drift introduces new forbidden keys.** Mitigated by guard middleware + CI lint + schema validation; RFC required for linkset changes.
|
||||
- **Migration complexity from legacy normalisation.** Staged cutover with `_backup_*` copies and temporary views to keep Policy Engine parity.
|
||||
- **Performance overhead during ingest.** Guard remains O(number of keys); index review ensures insert latency stays within warm (<5 s) / cold (<30 s) targets.
|
||||
- **Tenancy leakage.** `tenant` required in schema, Authority-supplied claims enforced per request, observability alerts fire on missing tenant identifiers.
|
||||
|
||||
## Test strategy
|
||||
- **Unit**: guard rejection paths, provenance enforcement, idempotent insertions, linkset determinism.
|
||||
- **Property**: fuzz upstream payloads to guarantee no forbidden fields emerge.
|
||||
- **Integration**: batch ingest (50k advisories, mixed VEX fixtures), verifying zero guard violations and consistent supersedes.
|
||||
- **Contract**: Policy Engine consumers verify raw-only reads; Export Center consumes canonical datasets.
|
||||
- **End-to-end**: ingest/verify flow with CLI + Console actions to confirm observability and guard reporting.
|
||||
|
||||
## Definition of done
|
||||
- Validators deployed and verified in staging/offline environments.
|
||||
- Runtime guards, CLI/Console workflows, and CI linting all active.
|
||||
- Observability dashboards and runbooks updated; metrics visible.
|
||||
- Documentation updates merged; Offline Kit instructions published.
|
||||
- ./TASKS.md reflects status transitions; cross-module dependencies acknowledged in ../../TASKS.md.
|
||||
# Implementation plan — Concelier
|
||||
|
||||
## Delivery timeline
|
||||
- **Phase 1 — Guardrails & schema**
|
||||
Stand up Mongo JSON validators for `advisory_raw` and `vex_raw`, wire the `AOCWriteGuard` repository interceptor, and seed deterministic linkset builders. Freeze legacy normalisation paths and migrate callers to the new raw schema.
|
||||
- **Phase 2 — API & observability**
|
||||
Publish ingestion and verification endpoints (`POST /ingest/*`, `GET /advisories.raw`, `POST /aoc/verify`) with Authority scopes, expose telemetry (`aoc_violation_total`, guard spans, structured logs), and ensure Offline Kit packaging captures validator deployment steps.
|
||||
- **Phase 3 — Experience polish**
|
||||
Ship CLI/Console affordances (`stella sources ingest --dry-run`, dashboard tiles, violation drill-downs), finish Export Center hand-off metadata, and close out CI enforcement (`stella aoc verify` preflight, AST lint, seeded fixtures).
|
||||
|
||||
## Work breakdown by component
|
||||
- **Concelier WebService & worker**
|
||||
- Add Mongo validators and unique indexes over `(tenant, source.vendor, upstream.upstream_id, upstream.content_hash)`.
|
||||
- Implement write interceptors rejecting forbidden fields, missing provenance, or merge attempts.
|
||||
- Deterministically compute linksets and persist canonical JSON payloads.
|
||||
- Introduce `/ingest/advisory`, `/advisories/raw*`, and `/aoc/verify` surfaces guarded by `advisory:*` and `aoc:verify` scopes.
|
||||
- Emit guard metrics/traces and surface supersedes/violation audit logs.
|
||||
- **Excititor (shared ingestion contract)**
|
||||
- Mirror Concelier guard and schema changes for `vex_raw`.
|
||||
- Maintain restart-time plug-in determinism and linkset extraction parity.
|
||||
- **Shared libraries**
|
||||
- Publish `StellaOps.Ingestion.AOC` (forbidden key catalog, guard middleware, provenance helpers, signature verification).
|
||||
- Share error codes (`ERR_AOC_00x`) and deterministic hashing utilities.
|
||||
- **Policy Engine integration**
|
||||
- Enforce `effective_finding_*` write exclusivity.
|
||||
- Consume only raw documents + linksets, removing any implicit normalisation.
|
||||
- **Authority scopes**
|
||||
- Provision `advisory:ingest|read`, `vex:ingest|read`, `aoc:verify`; propagate tenant claims to ingestion services.
|
||||
- **CLI & Console**
|
||||
- Implement `stella sources ingest --dry-run` and `stella aoc verify` (with exit codes mapped to `ERR_AOC_00x`).
|
||||
- Surface AOC dashboards, violation drill-down, and verification shortcuts in the Console.
|
||||
- **CI/CD**
|
||||
- Add Roslyn analyzer / AST lint to block forbidden writes.
|
||||
- Seed fixtures and run `stella aoc verify` against snapshots in pipeline gating.
|
||||
|
||||
## Documentation deliverables
|
||||
- Update `docs/ingestion/aggregation-only-contract.md` with guard invariants, schemas, error codes, and migration guidance.
|
||||
- Refresh `docs/modules/concelier/operations/*.md` (mirror, conflict-resolution, authority audit) with validator rollouts and observability dashboards.
|
||||
- Cross-link Authority scope definitions, CLI reference, Console sources guide, and observability runbooks to the AOC guard changes.
|
||||
- Ensure Offline Kit documentation captures validator bootstrap and verify workflows.
|
||||
|
||||
## Acceptance criteria
|
||||
- Mongo validators and runtime guards reject forbidden fields and missing provenance with the documented `ERR_AOC_00x` codes.
|
||||
- Linksets and supersedes chains are deterministic; rerunning ingestion over identical payloads yields byte-identical documents.
|
||||
- CLI `stella aoc verify` exits non-zero on seeded violations and zero on clean datasets; Console dashboards show real-time guard status.
|
||||
- Export Center consumes advisory datasets without relying on legacy normalised fields.
|
||||
- CI fails if lint rules detect forbidden writes or if seeded guard tests regress.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Collector drift introduces new forbidden keys.** Mitigated by guard middleware + CI lint + schema validation; RFC required for linkset changes.
|
||||
- **Migration complexity from legacy normalisation.** Staged cutover with `_backup_*` copies and temporary views to keep Policy Engine parity.
|
||||
- **Performance overhead during ingest.** Guard remains O(number of keys); index review ensures insert latency stays within warm (<5 s) / cold (<30 s) targets.
|
||||
- **Tenancy leakage.** `tenant` required in schema, Authority-supplied claims enforced per request, observability alerts fire on missing tenant identifiers.
|
||||
|
||||
## Test strategy
|
||||
- **Unit**: guard rejection paths, provenance enforcement, idempotent insertions, linkset determinism.
|
||||
- **Property**: fuzz upstream payloads to guarantee no forbidden fields emerge.
|
||||
- **Integration**: batch ingest (50k advisories, mixed VEX fixtures), verifying zero guard violations and consistent supersedes.
|
||||
- **Contract**: Policy Engine consumers verify raw-only reads; Export Center consumes canonical datasets.
|
||||
- **End-to-end**: ingest/verify flow with CLI + Console actions to confirm observability and guard reporting.
|
||||
|
||||
## Definition of done
|
||||
- Validators deployed and verified in staging/offline environments.
|
||||
- Runtime guards, CLI/Console workflows, and CI linting all active.
|
||||
- Observability dashboards and runbooks updated; metrics visible.
|
||||
- Documentation updates merged; Offline Kit instructions published.
|
||||
- ./TASKS.md reflects status transitions; cross-module dependencies acknowledged in ../../TASKS.md.
|
||||
|
||||
@@ -1,159 +1,159 @@
|
||||
# Concelier Authority Audit Runbook
|
||||
|
||||
_Last updated: 2025-10-22_
|
||||
|
||||
This runbook helps operators verify and monitor the StellaOps Concelier ⇆ Authority integration. It focuses on the `/jobs*` surface, which now requires StellaOps Authority tokens, and the corresponding audit/metric signals that expose authentication and bypass activity.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Authority integration is enabled in `concelier.yaml` (or via `CONCELIER_AUTHORITY__*` environment variables) with a valid `clientId`, secret, audience, and required scopes.
|
||||
- OTLP metrics/log exporters are configured (`concelier.telemetry.*`) or container stdout is shipped to your SIEM.
|
||||
- Operators have access to the Concelier job trigger endpoints via CLI or REST for smoke tests.
|
||||
- The rollout table in `docs/10_CONCELIER_CLI_QUICKSTART.md` has been reviewed so stakeholders align on the staged → enforced toggle timeline.
|
||||
|
||||
### Configuration snippet
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
authority:
|
||||
enabled: true
|
||||
allowAnonymousFallback: false # keep true only during initial rollout
|
||||
issuer: "https://authority.internal"
|
||||
audiences:
|
||||
- "api://concelier"
|
||||
requiredScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
requiredTenants:
|
||||
- "tenant-default"
|
||||
bypassNetworks:
|
||||
- "127.0.0.1/32"
|
||||
- "::1/128"
|
||||
clientId: "concelier-jobs"
|
||||
clientSecretFile: "/run/secrets/concelier_authority_client"
|
||||
tokenClockSkewSeconds: 60
|
||||
resilience:
|
||||
enableRetries: true
|
||||
retryDelays:
|
||||
- "00:00:01"
|
||||
- "00:00:02"
|
||||
- "00:00:05"
|
||||
allowOfflineCacheFallback: true
|
||||
offlineCacheTolerance: "00:10:00"
|
||||
```
|
||||
|
||||
> Store secrets outside source control. Concelier reads `clientSecretFile` on startup; rotate by updating the mounted file and restarting the service.
|
||||
|
||||
### Resilience tuning
|
||||
|
||||
- **Connected sites:** keep the default 1 s / 2 s / 5 s retry ladder so Concelier retries transient Authority hiccups but still surfaces outages quickly. Leave `allowOfflineCacheFallback=true` so cached discovery/JWKS data can bridge short Pathfinder restarts.
|
||||
- **Air-gapped/Offline Kit installs:** extend `offlineCacheTolerance` (15–30 minutes) to keep the cached metadata valid between manual synchronisations. You can also disable retries (`enableRetries=false`) if infrastructure teams prefer to handle exponential backoff at the network layer; Concelier will fail fast but keep deterministic logs.
|
||||
- Concelier resolves these knobs through `IOptionsMonitor<StellaOpsAuthClientOptions>`. Edits to `concelier.yaml` are applied on configuration reload; restart the container if you change environment variables or do not have file-watch reloads enabled.
|
||||
|
||||
## 2. Key Signals
|
||||
|
||||
### 2.1 Audit log channel
|
||||
|
||||
Concelier emits structured audit entries via the `Concelier.Authorization.Audit` logger for every `/jobs*` request once Authority enforcement is active.
|
||||
|
||||
```
|
||||
Concelier authorization audit route=/jobs/definitions status=200 subject=ops@example.com clientId=concelier-cli scopes=concelier.jobs.trigger advisory:ingest bypass=False remote=10.1.4.7
|
||||
```
|
||||
|
||||
| Field | Sample value | Meaning |
|
||||
|--------------|-------------------------|------------------------------------------------------------------------------------------|
|
||||
| `route` | `/jobs/definitions` | Endpoint that processed the request. |
|
||||
| `status` | `200` / `401` / `409` | Final HTTP status code returned to the caller. |
|
||||
| `subject` | `ops@example.com` | User or service principal subject (falls back to `(anonymous)` when unauthenticated). |
|
||||
| `clientId` | `concelier-cli` | OAuth client ID provided by Authority (`(none)` if the token lacked the claim). |
|
||||
| `scopes` | `concelier.jobs.trigger advisory:ingest advisory:read` | Normalised scope list extracted from token claims; `(none)` if the token carried none. |
|
||||
| `tenant` | `tenant-default` | Tenant claim extracted from the Authority token (`(none)` when the token lacked it). |
|
||||
| `bypass` | `True` / `False` | Indicates whether the request succeeded because its source IP matched a bypass CIDR. |
|
||||
| `remote` | `10.1.4.7` | Remote IP recorded from the connection / forwarded header test hooks. |
|
||||
|
||||
Use your logging backend (e.g., Loki) to index the logger name and filter for suspicious combinations:
|
||||
|
||||
- `status=401 AND bypass=True` – bypass network accepted an unauthenticated call (should be temporary during rollout).
|
||||
- `status=202 AND scopes="(none)"` – a token without scopes triggered a job; tighten client configuration.
|
||||
- `status=202 AND NOT contains(scopes,"advisory:ingest")` – ingestion attempted without the new AOC scopes; confirm the Authority client registration matches the sample above.
|
||||
- `tenant!=(tenant-default)` – indicates a cross-tenant token was accepted. Ensure Concelier `requiredTenants` is aligned with Authority client registration.
|
||||
- Spike in `clientId="(none)"` – indicates upstream Authority is not issuing `client_id` claims or the CLI is outdated.
|
||||
|
||||
### 2.2 Metrics
|
||||
|
||||
Concelier publishes counters under the OTEL meter `StellaOps.Concelier.WebService.Jobs`. Tags: `job.kind`, `job.trigger`, `job.outcome`.
|
||||
|
||||
| Metric name | Description | PromQL example |
|
||||
|-------------------------------|----------------------------------------------------|----------------|
|
||||
| `web.jobs.triggered` | Accepted job trigger requests. | `sum by (job_kind) (rate(web_jobs_triggered_total[5m]))` |
|
||||
| `web.jobs.trigger.conflict` | Rejected triggers (already running, disabled…). | `sum(rate(web_jobs_trigger_conflict_total[5m]))` |
|
||||
| `web.jobs.trigger.failed` | Server-side job failures. | `sum(rate(web_jobs_trigger_failed_total[5m]))` |
|
||||
|
||||
> Prometheus/OTEL collectors typically surface counters with `_total` suffix. Adjust queries to match your pipeline’s generated metric names.
|
||||
|
||||
Correlate audit logs with the following global meter exported via `Concelier.SourceDiagnostics`:
|
||||
|
||||
- `concelier.source.http.requests_total{concelier_source="jobs-run"}` – ensures REST/manual triggers route through Authority.
|
||||
- If Grafana dashboards are deployed, extend the “Concelier Jobs” board with the above counters plus a table of recent audit log entries.
|
||||
|
||||
## 3. Alerting Guidance
|
||||
|
||||
1. **Unauthorized bypass attempt**
|
||||
- Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", status="401", bypass="True"}[5m])) > 0`
|
||||
- Action: verify `bypassNetworks` list; confirm expected maintenance windows; rotate credentials if suspicious.
|
||||
|
||||
2. **Missing scopes**
|
||||
- Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", scopes="(none)", status="200"}[5m])) > 0`
|
||||
- Action: audit Authority client registration; ensure `requiredScopes` includes `concelier.jobs.trigger`, `advisory:ingest`, and `advisory:read`.
|
||||
|
||||
3. **Trigger failure surge**
|
||||
- Query: `sum(rate(web_jobs_trigger_failed_total[10m])) > 0` with severity `warning` if sustained for 10 minutes.
|
||||
- Action: inspect correlated audit entries and `Concelier.Telemetry` traces for job execution errors.
|
||||
|
||||
4. **Conflict spike**
|
||||
- Query: `sum(rate(web_jobs_trigger_conflict_total[10m])) > 5` (tune threshold).
|
||||
- Action: downstream scheduling may be firing repetitive triggers; ensure precedence is configured properly.
|
||||
|
||||
5. **Authority offline**
|
||||
- Watch `Concelier.Authorization.Audit` logs for `status=503` or `status=500` along with `clientId="(none)"`. Investigate Authority availability before re-enabling anonymous fallback.
|
||||
|
||||
## 4. Rollout & Verification Procedure
|
||||
|
||||
1. **Pre-checks**
|
||||
- Align with the rollout phases documented in `docs/10_CONCELIER_CLI_QUICKSTART.md` (validation → rehearsal → enforced) and record the target dates in your change request.
|
||||
- Confirm `allowAnonymousFallback` is `false` in production; keep `true` only during staged validation.
|
||||
- Validate Authority issuer metadata is reachable from Concelier (`curl https://authority.internal/.well-known/openid-configuration` from the host).
|
||||
|
||||
2. **Smoke test with valid token**
|
||||
- Obtain a token via CLI: `stella auth login --scope "concelier.jobs.trigger advisory:ingest" --scope advisory:read`.
|
||||
- Trigger a read-only endpoint: `curl -H "Authorization: Bearer $TOKEN" https://concelier.internal/jobs/definitions`.
|
||||
- Expect HTTP 200/202 and an audit log with `bypass=False`, `scopes=concelier.jobs.trigger advisory:ingest advisory:read`, and `tenant=tenant-default`.
|
||||
|
||||
3. **Negative test without token**
|
||||
- Call the same endpoint without a token. Expect HTTP 401, `bypass=False`.
|
||||
- If the request succeeds, double-check `bypassNetworks` and ensure fallback is disabled.
|
||||
|
||||
4. **Bypass check (if applicable)**
|
||||
- From an allowed maintenance IP, call `/jobs/definitions` without a token. Confirm the audit log shows `bypass=True`. Review business justification and expiry date for such entries.
|
||||
|
||||
5. **Metrics validation**
|
||||
- Ensure `web.jobs.triggered` counter increments during accepted runs.
|
||||
- Exporters should show corresponding spans (`concelier.job.trigger`) if tracing is enabled.
|
||||
|
||||
## 5. Troubleshooting
|
||||
|
||||
| Symptom | Probable cause | Remediation |
|
||||
|---------|----------------|-------------|
|
||||
| Audit log shows `clientId=(none)` for all requests | Authority not issuing `client_id` claim or CLI outdated | Update StellaOps Authority configuration (`StellaOpsAuthorityOptions.Token.Claims.ClientId`), or upgrade the CLI token acquisition flow. |
|
||||
| Requests succeed with `bypass=True` unexpectedly | Local network added to `bypassNetworks` or fallback still enabled | Remove/adjust the CIDR list, disable anonymous fallback, restart Concelier. |
|
||||
| HTTP 401 with valid token | `requiredScopes` missing from client registration or token audience mismatch | Verify Authority client scopes (`concelier.jobs.trigger`) and ensure the token audience matches `audiences` config. |
|
||||
| Metrics missing from Prometheus | Telemetry exporters disabled or filter missing OTEL meter | Set `concelier.telemetry.enableMetrics=true`, ensure collector includes `StellaOps.Concelier.WebService.Jobs` meter. |
|
||||
| Sudden spike in `web.jobs.trigger.failed` | Downstream job failure or Authority timeout mid-request | Inspect Concelier job logs, re-run with tracing enabled, validate Authority latency. |
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/21_INSTALL_GUIDE.md` – Authority configuration quick start.
|
||||
- `docs/17_SECURITY_HARDENING_GUIDE.md` – Security guardrails and enforcement deadlines.
|
||||
- `docs/modules/authority/operations/monitoring.md` – Authority-side monitoring and alerting playbook.
|
||||
- `StellaOps.Concelier.WebService/Filters/JobAuthorizationAuditFilter.cs` – source of audit log fields.
|
||||
# Concelier Authority Audit Runbook
|
||||
|
||||
_Last updated: 2025-10-22_
|
||||
|
||||
This runbook helps operators verify and monitor the StellaOps Concelier ⇆ Authority integration. It focuses on the `/jobs*` surface, which now requires StellaOps Authority tokens, and the corresponding audit/metric signals that expose authentication and bypass activity.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Authority integration is enabled in `concelier.yaml` (or via `CONCELIER_AUTHORITY__*` environment variables) with a valid `clientId`, secret, audience, and required scopes.
|
||||
- OTLP metrics/log exporters are configured (`concelier.telemetry.*`) or container stdout is shipped to your SIEM.
|
||||
- Operators have access to the Concelier job trigger endpoints via CLI or REST for smoke tests.
|
||||
- The rollout table in `docs/10_CONCELIER_CLI_QUICKSTART.md` has been reviewed so stakeholders align on the staged → enforced toggle timeline.
|
||||
|
||||
### Configuration snippet
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
authority:
|
||||
enabled: true
|
||||
allowAnonymousFallback: false # keep true only during initial rollout
|
||||
issuer: "https://authority.internal"
|
||||
audiences:
|
||||
- "api://concelier"
|
||||
requiredScopes:
|
||||
- "concelier.jobs.trigger"
|
||||
- "advisory:read"
|
||||
- "advisory:ingest"
|
||||
requiredTenants:
|
||||
- "tenant-default"
|
||||
bypassNetworks:
|
||||
- "127.0.0.1/32"
|
||||
- "::1/128"
|
||||
clientId: "concelier-jobs"
|
||||
clientSecretFile: "/run/secrets/concelier_authority_client"
|
||||
tokenClockSkewSeconds: 60
|
||||
resilience:
|
||||
enableRetries: true
|
||||
retryDelays:
|
||||
- "00:00:01"
|
||||
- "00:00:02"
|
||||
- "00:00:05"
|
||||
allowOfflineCacheFallback: true
|
||||
offlineCacheTolerance: "00:10:00"
|
||||
```
|
||||
|
||||
> Store secrets outside source control. Concelier reads `clientSecretFile` on startup; rotate by updating the mounted file and restarting the service.
|
||||
|
||||
### Resilience tuning
|
||||
|
||||
- **Connected sites:** keep the default 1 s / 2 s / 5 s retry ladder so Concelier retries transient Authority hiccups but still surfaces outages quickly. Leave `allowOfflineCacheFallback=true` so cached discovery/JWKS data can bridge short Pathfinder restarts.
|
||||
- **Air-gapped/Offline Kit installs:** extend `offlineCacheTolerance` (15–30 minutes) to keep the cached metadata valid between manual synchronisations. You can also disable retries (`enableRetries=false`) if infrastructure teams prefer to handle exponential backoff at the network layer; Concelier will fail fast but keep deterministic logs.
|
||||
- Concelier resolves these knobs through `IOptionsMonitor<StellaOpsAuthClientOptions>`. Edits to `concelier.yaml` are applied on configuration reload; restart the container if you change environment variables or do not have file-watch reloads enabled.
|
||||
|
||||
## 2. Key Signals
|
||||
|
||||
### 2.1 Audit log channel
|
||||
|
||||
Concelier emits structured audit entries via the `Concelier.Authorization.Audit` logger for every `/jobs*` request once Authority enforcement is active.
|
||||
|
||||
```
|
||||
Concelier authorization audit route=/jobs/definitions status=200 subject=ops@example.com clientId=concelier-cli scopes=concelier.jobs.trigger advisory:ingest bypass=False remote=10.1.4.7
|
||||
```
|
||||
|
||||
| Field | Sample value | Meaning |
|
||||
|--------------|-------------------------|------------------------------------------------------------------------------------------|
|
||||
| `route` | `/jobs/definitions` | Endpoint that processed the request. |
|
||||
| `status` | `200` / `401` / `409` | Final HTTP status code returned to the caller. |
|
||||
| `subject` | `ops@example.com` | User or service principal subject (falls back to `(anonymous)` when unauthenticated). |
|
||||
| `clientId` | `concelier-cli` | OAuth client ID provided by Authority (`(none)` if the token lacked the claim). |
|
||||
| `scopes` | `concelier.jobs.trigger advisory:ingest advisory:read` | Normalised scope list extracted from token claims; `(none)` if the token carried none. |
|
||||
| `tenant` | `tenant-default` | Tenant claim extracted from the Authority token (`(none)` when the token lacked it). |
|
||||
| `bypass` | `True` / `False` | Indicates whether the request succeeded because its source IP matched a bypass CIDR. |
|
||||
| `remote` | `10.1.4.7` | Remote IP recorded from the connection / forwarded header test hooks. |
|
||||
|
||||
Use your logging backend (e.g., Loki) to index the logger name and filter for suspicious combinations:
|
||||
|
||||
- `status=401 AND bypass=True` – bypass network accepted an unauthenticated call (should be temporary during rollout).
|
||||
- `status=202 AND scopes="(none)"` – a token without scopes triggered a job; tighten client configuration.
|
||||
- `status=202 AND NOT contains(scopes,"advisory:ingest")` – ingestion attempted without the new AOC scopes; confirm the Authority client registration matches the sample above.
|
||||
- `tenant!=(tenant-default)` – indicates a cross-tenant token was accepted. Ensure Concelier `requiredTenants` is aligned with Authority client registration.
|
||||
- Spike in `clientId="(none)"` – indicates upstream Authority is not issuing `client_id` claims or the CLI is outdated.
|
||||
|
||||
### 2.2 Metrics
|
||||
|
||||
Concelier publishes counters under the OTEL meter `StellaOps.Concelier.WebService.Jobs`. Tags: `job.kind`, `job.trigger`, `job.outcome`.
|
||||
|
||||
| Metric name | Description | PromQL example |
|
||||
|-------------------------------|----------------------------------------------------|----------------|
|
||||
| `web.jobs.triggered` | Accepted job trigger requests. | `sum by (job_kind) (rate(web_jobs_triggered_total[5m]))` |
|
||||
| `web.jobs.trigger.conflict` | Rejected triggers (already running, disabled…). | `sum(rate(web_jobs_trigger_conflict_total[5m]))` |
|
||||
| `web.jobs.trigger.failed` | Server-side job failures. | `sum(rate(web_jobs_trigger_failed_total[5m]))` |
|
||||
|
||||
> Prometheus/OTEL collectors typically surface counters with `_total` suffix. Adjust queries to match your pipeline’s generated metric names.
|
||||
|
||||
Correlate audit logs with the following global meter exported via `Concelier.SourceDiagnostics`:
|
||||
|
||||
- `concelier.source.http.requests_total{concelier_source="jobs-run"}` – ensures REST/manual triggers route through Authority.
|
||||
- If Grafana dashboards are deployed, extend the “Concelier Jobs” board with the above counters plus a table of recent audit log entries.
|
||||
|
||||
## 3. Alerting Guidance
|
||||
|
||||
1. **Unauthorized bypass attempt**
|
||||
- Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", status="401", bypass="True"}[5m])) > 0`
|
||||
- Action: verify `bypassNetworks` list; confirm expected maintenance windows; rotate credentials if suspicious.
|
||||
|
||||
2. **Missing scopes**
|
||||
- Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", scopes="(none)", status="200"}[5m])) > 0`
|
||||
- Action: audit Authority client registration; ensure `requiredScopes` includes `concelier.jobs.trigger`, `advisory:ingest`, and `advisory:read`.
|
||||
|
||||
3. **Trigger failure surge**
|
||||
- Query: `sum(rate(web_jobs_trigger_failed_total[10m])) > 0` with severity `warning` if sustained for 10 minutes.
|
||||
- Action: inspect correlated audit entries and `Concelier.Telemetry` traces for job execution errors.
|
||||
|
||||
4. **Conflict spike**
|
||||
- Query: `sum(rate(web_jobs_trigger_conflict_total[10m])) > 5` (tune threshold).
|
||||
- Action: downstream scheduling may be firing repetitive triggers; ensure precedence is configured properly.
|
||||
|
||||
5. **Authority offline**
|
||||
- Watch `Concelier.Authorization.Audit` logs for `status=503` or `status=500` along with `clientId="(none)"`. Investigate Authority availability before re-enabling anonymous fallback.
|
||||
|
||||
## 4. Rollout & Verification Procedure
|
||||
|
||||
1. **Pre-checks**
|
||||
- Align with the rollout phases documented in `docs/10_CONCELIER_CLI_QUICKSTART.md` (validation → rehearsal → enforced) and record the target dates in your change request.
|
||||
- Confirm `allowAnonymousFallback` is `false` in production; keep `true` only during staged validation.
|
||||
- Validate Authority issuer metadata is reachable from Concelier (`curl https://authority.internal/.well-known/openid-configuration` from the host).
|
||||
|
||||
2. **Smoke test with valid token**
|
||||
- Obtain a token via CLI: `stella auth login --scope "concelier.jobs.trigger advisory:ingest" --scope advisory:read`.
|
||||
- Trigger a read-only endpoint: `curl -H "Authorization: Bearer $TOKEN" https://concelier.internal/jobs/definitions`.
|
||||
- Expect HTTP 200/202 and an audit log with `bypass=False`, `scopes=concelier.jobs.trigger advisory:ingest advisory:read`, and `tenant=tenant-default`.
|
||||
|
||||
3. **Negative test without token**
|
||||
- Call the same endpoint without a token. Expect HTTP 401, `bypass=False`.
|
||||
- If the request succeeds, double-check `bypassNetworks` and ensure fallback is disabled.
|
||||
|
||||
4. **Bypass check (if applicable)**
|
||||
- From an allowed maintenance IP, call `/jobs/definitions` without a token. Confirm the audit log shows `bypass=True`. Review business justification and expiry date for such entries.
|
||||
|
||||
5. **Metrics validation**
|
||||
- Ensure `web.jobs.triggered` counter increments during accepted runs.
|
||||
- Exporters should show corresponding spans (`concelier.job.trigger`) if tracing is enabled.
|
||||
|
||||
## 5. Troubleshooting
|
||||
|
||||
| Symptom | Probable cause | Remediation |
|
||||
|---------|----------------|-------------|
|
||||
| Audit log shows `clientId=(none)` for all requests | Authority not issuing `client_id` claim or CLI outdated | Update StellaOps Authority configuration (`StellaOpsAuthorityOptions.Token.Claims.ClientId`), or upgrade the CLI token acquisition flow. |
|
||||
| Requests succeed with `bypass=True` unexpectedly | Local network added to `bypassNetworks` or fallback still enabled | Remove/adjust the CIDR list, disable anonymous fallback, restart Concelier. |
|
||||
| HTTP 401 with valid token | `requiredScopes` missing from client registration or token audience mismatch | Verify Authority client scopes (`concelier.jobs.trigger`) and ensure the token audience matches `audiences` config. |
|
||||
| Metrics missing from Prometheus | Telemetry exporters disabled or filter missing OTEL meter | Set `concelier.telemetry.enableMetrics=true`, ensure collector includes `StellaOps.Concelier.WebService.Jobs` meter. |
|
||||
| Sudden spike in `web.jobs.trigger.failed` | Downstream job failure or Authority timeout mid-request | Inspect Concelier job logs, re-run with tracing enabled, validate Authority latency. |
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/21_INSTALL_GUIDE.md` – Authority configuration quick start.
|
||||
- `docs/17_SECURITY_HARDENING_GUIDE.md` – Security guardrails and enforcement deadlines.
|
||||
- `docs/modules/authority/operations/monitoring.md` – Authority-side monitoring and alerting playbook.
|
||||
- `StellaOps.Concelier.WebService/Filters/JobAuthorizationAuditFilter.cs` – source of audit log fields.
|
||||
|
||||
@@ -1,160 +1,160 @@
|
||||
# Concelier Conflict Resolution Runbook (Sprint 3)
|
||||
|
||||
This runbook equips Concelier operators to detect, triage, and resolve advisory conflicts now that the Sprint 3 merge engine landed (`AdvisoryPrecedenceMerger`, merge-event hashing, and telemetry counters). It builds on the canonical rules defined in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` and the metrics/logging instrumentation delivered this sprint.
|
||||
|
||||
---
|
||||
|
||||
## 1. Precedence Model (recap)
|
||||
|
||||
- **Default ranking:** `GHSA -> NVD -> OSV`, with distro/vendor PSIRTs outranking ecosystem feeds (`AdvisoryPrecedenceDefaults`). Use `concelier:merge:precedence:ranks` to override per source when incident response requires it.
|
||||
- **Freshness override:** if a lower-ranked source is >= 48 hours newer for a freshness-sensitive field (title, summary, affected ranges, references, credits), it wins. Every override stamps `provenance[].decisionReason = freshness`.
|
||||
- **Tie-breakers:** when precedence and freshness tie, the engine falls back to (1) primary source order, (2) shortest normalized text, (3) lowest stable hash. Merge-generated provenance records set `decisionReason = tie-breaker`.
|
||||
- **Audit trail:** each merged advisory receives a `merge` provenance entry listing the participating sources plus a `merge_event` record with canonical before/after SHA-256 hashes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Telemetry Shipped This Sprint
|
||||
|
||||
| Instrument | Type | Key Tags | Purpose |
|
||||
|------------|------|----------|---------|
|
||||
| `concelier.merge.operations` | Counter | `inputs` | Total precedence merges executed. |
|
||||
| `concelier.merge.overrides` | Counter | `primary_source`, `suppressed_source`, `primary_rank`, `suppressed_rank` | Field-level overrides chosen by precedence. |
|
||||
| `concelier.merge.range_overrides` | Counter | `advisory_key`, `package_type`, `primary_source`, `suppressed_source`, `primary_range_count`, `suppressed_range_count` | Package range overrides emitted by `AffectedPackagePrecedenceResolver`. |
|
||||
| `concelier.merge.conflicts` | Counter | `type` (`severity`, `precedence_tie`), `reason` (`mismatch`, `primary_missing`, `equal_rank`) | Conflicts requiring operator review. |
|
||||
| `concelier.merge.identity_conflicts` | Counter | `scheme`, `alias_value`, `advisory_count` | Alias collisions surfaced by the identity graph. |
|
||||
|
||||
### Structured logs
|
||||
|
||||
- `AdvisoryOverride` (EventId 1000) - logs merge suppressions with alias/provenance counts.
|
||||
- `PackageRangeOverride` (EventId 1001) - logs package-level precedence decisions.
|
||||
- `PrecedenceConflict` (EventId 1002) - logs mismatched severity or equal-rank scenarios.
|
||||
- `Alias collision ...` (no EventId) - emitted when `concelier.merge.identity_conflicts` increments.
|
||||
|
||||
Expect all logs at `Information`. Ensure OTEL exporters include the scope `StellaOps.Concelier.Merge`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Detection & Alerting
|
||||
|
||||
1. **Dashboard panels**
|
||||
- `concelier.merge.conflicts` - table grouped by `type/reason`. Alert when > 0 in a 15 minute window.
|
||||
- `concelier.merge.range_overrides` - stacked bar by `package_type`. Spikes highlight vendor PSIRT overrides over registry data.
|
||||
- `concelier.merge.overrides` with `primary_source|suppressed_source` - catches unexpected precedence flips (e.g., OSV overtaking GHSA).
|
||||
- `concelier.merge.identity_conflicts` - single-stat; alert when alias collisions occur more than once per day.
|
||||
2. **Log based alerts**
|
||||
- `eventId=1002` with `reason="equal_rank"` - indicates precedence table gaps; page merge owners.
|
||||
- `eventId=1002` with `reason="mismatch"` - severity disagreement; open connector bug if sustained.
|
||||
3. **Job health**
|
||||
- `stellaops-cli db merge` exit code `1` signifies unresolved conflicts. Pipe to automation that captures logs and notifies #concelier-ops.
|
||||
|
||||
### Threshold updates (2025-10-12)
|
||||
|
||||
- `concelier.merge.conflicts` – Page only when ≥ 2 events fire within 30 minutes; the synthetic conflict fixture run produces 0 conflicts, so the first event now routes to Slack for manual review instead of paging.
|
||||
- `concelier.merge.overrides` – Raise a warning when the 30-minute sum exceeds 10 (canonical triple yields exactly 1 summary override with `primary_source=osv`, `suppressed_source=ghsa`).
|
||||
- `concelier.merge.range_overrides` – Maintain the 15-minute alert at ≥ 3 but annotate dashboards that the regression triple emits a single `package_type=semver` override so ops can spot unexpected spikes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Triage Workflow
|
||||
|
||||
1. **Confirm job context**
|
||||
- `stellaops-cli db merge` (CLI) or `POST /jobs/merge:reconcile` (API) to rehydrate the merge job. Use `--verbose` to stream structured logs during triage.
|
||||
2. **Inspect metrics**
|
||||
- Correlate spikes in `concelier.merge.conflicts` with `primary_source`/`suppressed_source` tags from `concelier.merge.overrides`.
|
||||
3. **Pull structured logs**
|
||||
- Example (vector output):
|
||||
```
|
||||
jq 'select(.EventId.Name=="PrecedenceConflict") | {advisory: .State[0].Value, type: .ConflictType, reason: .Reason, primary: .PrimarySources, suppressed: .SuppressedSources}' stellaops-concelier.log
|
||||
```
|
||||
4. **Review merge events**
|
||||
- `mongosh`:
|
||||
```javascript
|
||||
use concelier;
|
||||
db.merge_event.find({ advisoryKey: "CVE-2025-1234" }).sort({ mergedAt: -1 }).limit(5);
|
||||
```
|
||||
- Compare `beforeHash` vs `afterHash` to confirm the merge actually changed canonical output.
|
||||
5. **Interrogate provenance**
|
||||
- `db.advisories.findOne({ advisoryKey: "CVE-2025-1234" }, { title: 1, severity: 1, provenance: 1, "affectedPackages.provenance": 1 })`
|
||||
- Check `provenance[].decisionReason` values (`precedence`, `freshness`, `tie-breaker`) to understand why the winning field was chosen.
|
||||
|
||||
---
|
||||
|
||||
## 5. Conflict Classification Matrix
|
||||
|
||||
| Signal | Likely Cause | Immediate Action |
|
||||
|--------|--------------|------------------|
|
||||
| `reason="mismatch"` with `type="severity"` | Upstream feeds disagree on CVSS vector/severity. | Verify which feed is freshest; if correctness is known, adjust connector mapping or precedence override. |
|
||||
| `reason="primary_missing"` | Higher-ranked source lacks the field entirely. | Backfill connector data or temporarily allow lower-ranked source via precedence override. |
|
||||
| `reason="equal_rank"` | Two feeds share the same precedence rank (custom config or missing entry). | Update `concelier:merge:precedence:ranks` to break the tie; restart merge job. |
|
||||
| Rising `concelier.merge.range_overrides` for a package type | Vendor PSIRT now supplies richer ranges. | Validate connectors emit `decisionReason="precedence"` and update dashboards to treat registry ranges as fallback. |
|
||||
| `concelier.merge.identity_conflicts` > 0 | Alias scheme mapping produced collisions (duplicate CVE <-> advisory pairs). | Inspect `Alias collision` log payload; reconcile the alias graph by adjusting connector alias output. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Resolution Playbook
|
||||
|
||||
1. **Connector data fix**
|
||||
- Re-run the offending connector stages (`stellaops-cli db fetch --source ghsa --stage map` etc.).
|
||||
- Once fixed, rerun merge and verify `decisionReason` reflects `freshness` or `precedence` as expected.
|
||||
2. **Temporary precedence override**
|
||||
- Edit `etc/concelier.yaml`:
|
||||
```yaml
|
||||
concelier:
|
||||
merge:
|
||||
precedence:
|
||||
ranks:
|
||||
osv: 1
|
||||
ghsa: 0
|
||||
```
|
||||
- Restart Concelier workers; confirm tags in `concelier.merge.overrides` show the new ranks.
|
||||
- Document the override with expiry in the change log.
|
||||
3. **Alias remediation**
|
||||
- Update connector mapping rules to weed out duplicate aliases (e.g., skip GHSA aliases that mirror CVE IDs).
|
||||
- Flush cached alias graphs if necessary (`db.alias_graph.drop()` is destructive-coordinate with Storage before issuing).
|
||||
4. **Escalation**
|
||||
- If override metrics spike due to upstream regression, open an incident with Security Guild, referencing merge logs and `merge_event` IDs.
|
||||
|
||||
---
|
||||
|
||||
## 7. Validation Checklist
|
||||
|
||||
- [ ] Merge job rerun returns exit code `0`.
|
||||
- [ ] `concelier.merge.conflicts` baseline returns to zero after corrective action.
|
||||
- [ ] Latest `merge_event` entry shows expected hash delta.
|
||||
- [ ] Affected advisory document shows updated `provenance[].decisionReason`.
|
||||
- [ ] Ops change log updated with incident summary, config overrides, and rollback plan.
|
||||
|
||||
---
|
||||
|
||||
## 8. Reference Material
|
||||
|
||||
- Canonical conflict rules: `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md`.
|
||||
- Merge engine internals: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs`.
|
||||
- Metrics definitions: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs` (identity conflicts) and `AdvisoryPrecedenceMerger`.
|
||||
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/MergeEvents`.
|
||||
|
||||
Keep this runbook synchronized with future sprint notes and update alert thresholds as baseline volumes change.
|
||||
|
||||
---
|
||||
|
||||
## 9. Synthetic Regression Fixtures
|
||||
|
||||
- **Locations** – Canonical conflict snapshots now live at `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/conflict-ghsa.canonical.json`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/conflict-nvd.canonical.json`, and `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/conflict-osv.canonical.json`.
|
||||
- **Validation commands** – To regenerate and verify the fixtures offline, run:
|
||||
|
||||
```bash
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj --filter GhsaConflictFixtureTests
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests/StellaOps.Concelier.Connector.Nvd.Tests.csproj --filter NvdConflictFixtureTests
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj --filter OsvConflictFixtureTests
|
||||
dotnet test src/Concelier/__Tests/StellaOps.Concelier.Merge.Tests/StellaOps.Concelier.Merge.Tests.csproj --filter MergeAsync_AppliesCanonicalRulesAndPersistsDecisions
|
||||
```
|
||||
|
||||
- **Expected signals** – The triple produces one freshness-driven summary override (`primary_source=osv`, `suppressed_source=ghsa`) and one range override for the npm SemVer package while leaving `concelier.merge.conflicts` at zero. Use these values as the baseline when tuning dashboards or load-testing alert pipelines.
|
||||
|
||||
---
|
||||
|
||||
## 10. Change Log
|
||||
|
||||
| Date (UTC) | Change | Notes |
|
||||
|------------|--------|-------|
|
||||
| 2025-10-16 | Ops review signed off after connector expansion (CCCS, CERT-Bund, KISA, ICS CISA, MSRC) landed. Alert thresholds from §3 reaffirmed; dashboards updated to watch attachment signals emitted by ICS CISA connector. | Ops sign-off recorded by Concelier Ops Guild; no additional overrides required. |
|
||||
# Concelier Conflict Resolution Runbook (Sprint 3)
|
||||
|
||||
This runbook equips Concelier operators to detect, triage, and resolve advisory conflicts now that the Sprint 3 merge engine landed (`AdvisoryPrecedenceMerger`, merge-event hashing, and telemetry counters). It builds on the canonical rules defined in `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md` and the metrics/logging instrumentation delivered this sprint.
|
||||
|
||||
---
|
||||
|
||||
## 1. Precedence Model (recap)
|
||||
|
||||
- **Default ranking:** `GHSA -> NVD -> OSV`, with distro/vendor PSIRTs outranking ecosystem feeds (`AdvisoryPrecedenceDefaults`). Use `concelier:merge:precedence:ranks` to override per source when incident response requires it.
|
||||
- **Freshness override:** if a lower-ranked source is >= 48 hours newer for a freshness-sensitive field (title, summary, affected ranges, references, credits), it wins. Every override stamps `provenance[].decisionReason = freshness`.
|
||||
- **Tie-breakers:** when precedence and freshness tie, the engine falls back to (1) primary source order, (2) shortest normalized text, (3) lowest stable hash. Merge-generated provenance records set `decisionReason = tie-breaker`.
|
||||
- **Audit trail:** each merged advisory receives a `merge` provenance entry listing the participating sources plus a `merge_event` record with canonical before/after SHA-256 hashes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Telemetry Shipped This Sprint
|
||||
|
||||
| Instrument | Type | Key Tags | Purpose |
|
||||
|------------|------|----------|---------|
|
||||
| `concelier.merge.operations` | Counter | `inputs` | Total precedence merges executed. |
|
||||
| `concelier.merge.overrides` | Counter | `primary_source`, `suppressed_source`, `primary_rank`, `suppressed_rank` | Field-level overrides chosen by precedence. |
|
||||
| `concelier.merge.range_overrides` | Counter | `advisory_key`, `package_type`, `primary_source`, `suppressed_source`, `primary_range_count`, `suppressed_range_count` | Package range overrides emitted by `AffectedPackagePrecedenceResolver`. |
|
||||
| `concelier.merge.conflicts` | Counter | `type` (`severity`, `precedence_tie`), `reason` (`mismatch`, `primary_missing`, `equal_rank`) | Conflicts requiring operator review. |
|
||||
| `concelier.merge.identity_conflicts` | Counter | `scheme`, `alias_value`, `advisory_count` | Alias collisions surfaced by the identity graph. |
|
||||
|
||||
### Structured logs
|
||||
|
||||
- `AdvisoryOverride` (EventId 1000) - logs merge suppressions with alias/provenance counts.
|
||||
- `PackageRangeOverride` (EventId 1001) - logs package-level precedence decisions.
|
||||
- `PrecedenceConflict` (EventId 1002) - logs mismatched severity or equal-rank scenarios.
|
||||
- `Alias collision ...` (no EventId) - emitted when `concelier.merge.identity_conflicts` increments.
|
||||
|
||||
Expect all logs at `Information`. Ensure OTEL exporters include the scope `StellaOps.Concelier.Merge`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Detection & Alerting
|
||||
|
||||
1. **Dashboard panels**
|
||||
- `concelier.merge.conflicts` - table grouped by `type/reason`. Alert when > 0 in a 15 minute window.
|
||||
- `concelier.merge.range_overrides` - stacked bar by `package_type`. Spikes highlight vendor PSIRT overrides over registry data.
|
||||
- `concelier.merge.overrides` with `primary_source|suppressed_source` - catches unexpected precedence flips (e.g., OSV overtaking GHSA).
|
||||
- `concelier.merge.identity_conflicts` - single-stat; alert when alias collisions occur more than once per day.
|
||||
2. **Log based alerts**
|
||||
- `eventId=1002` with `reason="equal_rank"` - indicates precedence table gaps; page merge owners.
|
||||
- `eventId=1002` with `reason="mismatch"` - severity disagreement; open connector bug if sustained.
|
||||
3. **Job health**
|
||||
- `stellaops-cli db merge` exit code `1` signifies unresolved conflicts. Pipe to automation that captures logs and notifies #concelier-ops.
|
||||
|
||||
### Threshold updates (2025-10-12)
|
||||
|
||||
- `concelier.merge.conflicts` – Page only when ≥ 2 events fire within 30 minutes; the synthetic conflict fixture run produces 0 conflicts, so the first event now routes to Slack for manual review instead of paging.
|
||||
- `concelier.merge.overrides` – Raise a warning when the 30-minute sum exceeds 10 (canonical triple yields exactly 1 summary override with `primary_source=osv`, `suppressed_source=ghsa`).
|
||||
- `concelier.merge.range_overrides` – Maintain the 15-minute alert at ≥ 3 but annotate dashboards that the regression triple emits a single `package_type=semver` override so ops can spot unexpected spikes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Triage Workflow
|
||||
|
||||
1. **Confirm job context**
|
||||
- `stellaops-cli db merge` (CLI) or `POST /jobs/merge:reconcile` (API) to rehydrate the merge job. Use `--verbose` to stream structured logs during triage.
|
||||
2. **Inspect metrics**
|
||||
- Correlate spikes in `concelier.merge.conflicts` with `primary_source`/`suppressed_source` tags from `concelier.merge.overrides`.
|
||||
3. **Pull structured logs**
|
||||
- Example (vector output):
|
||||
```
|
||||
jq 'select(.EventId.Name=="PrecedenceConflict") | {advisory: .State[0].Value, type: .ConflictType, reason: .Reason, primary: .PrimarySources, suppressed: .SuppressedSources}' stellaops-concelier.log
|
||||
```
|
||||
4. **Review merge events**
|
||||
- `mongosh`:
|
||||
```javascript
|
||||
use concelier;
|
||||
db.merge_event.find({ advisoryKey: "CVE-2025-1234" }).sort({ mergedAt: -1 }).limit(5);
|
||||
```
|
||||
- Compare `beforeHash` vs `afterHash` to confirm the merge actually changed canonical output.
|
||||
5. **Interrogate provenance**
|
||||
- `db.advisories.findOne({ advisoryKey: "CVE-2025-1234" }, { title: 1, severity: 1, provenance: 1, "affectedPackages.provenance": 1 })`
|
||||
- Check `provenance[].decisionReason` values (`precedence`, `freshness`, `tie-breaker`) to understand why the winning field was chosen.
|
||||
|
||||
---
|
||||
|
||||
## 5. Conflict Classification Matrix
|
||||
|
||||
| Signal | Likely Cause | Immediate Action |
|
||||
|--------|--------------|------------------|
|
||||
| `reason="mismatch"` with `type="severity"` | Upstream feeds disagree on CVSS vector/severity. | Verify which feed is freshest; if correctness is known, adjust connector mapping or precedence override. |
|
||||
| `reason="primary_missing"` | Higher-ranked source lacks the field entirely. | Backfill connector data or temporarily allow lower-ranked source via precedence override. |
|
||||
| `reason="equal_rank"` | Two feeds share the same precedence rank (custom config or missing entry). | Update `concelier:merge:precedence:ranks` to break the tie; restart merge job. |
|
||||
| Rising `concelier.merge.range_overrides` for a package type | Vendor PSIRT now supplies richer ranges. | Validate connectors emit `decisionReason="precedence"` and update dashboards to treat registry ranges as fallback. |
|
||||
| `concelier.merge.identity_conflicts` > 0 | Alias scheme mapping produced collisions (duplicate CVE <-> advisory pairs). | Inspect `Alias collision` log payload; reconcile the alias graph by adjusting connector alias output. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Resolution Playbook
|
||||
|
||||
1. **Connector data fix**
|
||||
- Re-run the offending connector stages (`stellaops-cli db fetch --source ghsa --stage map` etc.).
|
||||
- Once fixed, rerun merge and verify `decisionReason` reflects `freshness` or `precedence` as expected.
|
||||
2. **Temporary precedence override**
|
||||
- Edit `etc/concelier.yaml`:
|
||||
```yaml
|
||||
concelier:
|
||||
merge:
|
||||
precedence:
|
||||
ranks:
|
||||
osv: 1
|
||||
ghsa: 0
|
||||
```
|
||||
- Restart Concelier workers; confirm tags in `concelier.merge.overrides` show the new ranks.
|
||||
- Document the override with expiry in the change log.
|
||||
3. **Alias remediation**
|
||||
- Update connector mapping rules to weed out duplicate aliases (e.g., skip GHSA aliases that mirror CVE IDs).
|
||||
- Flush cached alias graphs if necessary (`db.alias_graph.drop()` is destructive-coordinate with Storage before issuing).
|
||||
4. **Escalation**
|
||||
- If override metrics spike due to upstream regression, open an incident with Security Guild, referencing merge logs and `merge_event` IDs.
|
||||
|
||||
---
|
||||
|
||||
## 7. Validation Checklist
|
||||
|
||||
- [ ] Merge job rerun returns exit code `0`.
|
||||
- [ ] `concelier.merge.conflicts` baseline returns to zero after corrective action.
|
||||
- [ ] Latest `merge_event` entry shows expected hash delta.
|
||||
- [ ] Affected advisory document shows updated `provenance[].decisionReason`.
|
||||
- [ ] Ops change log updated with incident summary, config overrides, and rollback plan.
|
||||
|
||||
---
|
||||
|
||||
## 8. Reference Material
|
||||
|
||||
- Canonical conflict rules: `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md`.
|
||||
- Merge engine internals: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs`.
|
||||
- Metrics definitions: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs` (identity conflicts) and `AdvisoryPrecedenceMerger`.
|
||||
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/MergeEvents`.
|
||||
|
||||
Keep this runbook synchronized with future sprint notes and update alert thresholds as baseline volumes change.
|
||||
|
||||
---
|
||||
|
||||
## 9. Synthetic Regression Fixtures
|
||||
|
||||
- **Locations** – Canonical conflict snapshots now live at `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/Fixtures/conflict-ghsa.canonical.json`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests/Nvd/Fixtures/conflict-nvd.canonical.json`, and `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/conflict-osv.canonical.json`.
|
||||
- **Validation commands** – To regenerate and verify the fixtures offline, run:
|
||||
|
||||
```bash
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests/StellaOps.Concelier.Connector.Ghsa.Tests.csproj --filter GhsaConflictFixtureTests
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests/StellaOps.Concelier.Connector.Nvd.Tests.csproj --filter NvdConflictFixtureTests
|
||||
dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj --filter OsvConflictFixtureTests
|
||||
dotnet test src/Concelier/__Tests/StellaOps.Concelier.Merge.Tests/StellaOps.Concelier.Merge.Tests.csproj --filter MergeAsync_AppliesCanonicalRulesAndPersistsDecisions
|
||||
```
|
||||
|
||||
- **Expected signals** – The triple produces one freshness-driven summary override (`primary_source=osv`, `suppressed_source=ghsa`) and one range override for the npm SemVer package while leaving `concelier.merge.conflicts` at zero. Use these values as the baseline when tuning dashboards or load-testing alert pipelines.
|
||||
|
||||
---
|
||||
|
||||
## 10. Change Log
|
||||
|
||||
| Date (UTC) | Change | Notes |
|
||||
|------------|--------|-------|
|
||||
| 2025-10-16 | Ops review signed off after connector expansion (CCCS, CERT-Bund, KISA, ICS CISA, MSRC) landed. Alert thresholds from §3 reaffirmed; dashboards updated to watch attachment signals emitted by ICS CISA connector. | Ops sign-off recorded by Concelier Ops Guild; no additional overrides required. |
|
||||
|
||||
@@ -1,77 +1,77 @@
|
||||
# Concelier Apple Security Update Connector Operations
|
||||
|
||||
This runbook covers staging and production rollout for the Apple security updates connector (`source:vndr-apple:*`), including observability checks and fixture maintenance.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Network egress (or mirrored cache) for `https://gdmf.apple.com/v2/pmv` and the Apple Support domain (`https://support.apple.com/`).
|
||||
- Optional: corporate proxy exclusions for the Apple hosts if outbound traffic is normally filtered.
|
||||
- Updated configuration (environment variables or `concelier.yaml`) with an `apple` section. Example baseline:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
apple:
|
||||
softwareLookupUri: "https://gdmf.apple.com/v2/pmv"
|
||||
advisoryBaseUri: "https://support.apple.com/"
|
||||
localeSegment: "en-us"
|
||||
maxAdvisoriesPerFetch: 25
|
||||
initialBackfill: "120.00:00:00"
|
||||
modifiedTolerance: "02:00:00"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> ℹ️ `softwareLookupUri` and `advisoryBaseUri` must stay absolute and aligned with the HTTP allow-list; Concelier automatically adds both hosts to the connector HttpClient.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Deploy the configuration and restart the Concelier workers to ensure the Apple connector options are bound.
|
||||
2. Trigger a full connector cycle:
|
||||
- CLI: `stella db jobs run source:vndr-apple:fetch --and-then source:vndr-apple:parse --and-then source:vndr-apple:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:vndr-apple:fetch", "chain": ["source:vndr-apple:parse", "source:vndr-apple:map"] }`
|
||||
3. Validate metrics exported under meter `StellaOps.Concelier.Connector.Vndr.Apple`:
|
||||
- `apple.fetch.items` (documents fetched)
|
||||
- `apple.fetch.failures`
|
||||
- `apple.fetch.unchanged`
|
||||
- `apple.parse.failures`
|
||||
- `apple.map.affected.count` (histogram of affected package counts)
|
||||
4. Cross-check the shared HTTP counters:
|
||||
- `concelier.source.http.requests_total{concelier_source="vndr-apple"}` should increase for both index and detail phases.
|
||||
- `concelier.source.http.failures_total{concelier_source="vndr-apple"}` should remain flat (0) during a healthy run.
|
||||
5. Inspect the info logs:
|
||||
- `Apple software index fetch … processed=X newDocuments=Y`
|
||||
- `Apple advisory parse complete … aliases=… affected=…`
|
||||
- `Mapped Apple advisory … pendingMappings=0`
|
||||
6. Confirm MongoDB state:
|
||||
- `raw_documents` store contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||||
- `dtos` store has `schemaVersion="apple.security.update.v1"`.
|
||||
- `advisories` collection includes keys `HTxxxxxx` with normalized SemVer rules.
|
||||
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
- **Dashboards** – Add the following expressions to your Concelier Grafana board (OTLP/Prometheus naming assumed):
|
||||
- `rate(apple_fetch_items_total[15m])` vs `rate(concelier_source_http_requests_total{concelier_source="vndr-apple"}[15m])`
|
||||
- `rate(apple_fetch_failures_total[5m])` for error spikes (`severity=warning` at `>0`)
|
||||
- `histogram_quantile(0.95, rate(apple_map_affected_count_bucket[1h]))` to watch affected-package fan-out
|
||||
- `increase(apple_parse_failures_total[6h])` to catch parser drift (alerts at `>0`)
|
||||
- **Alerts** – Page if `rate(apple_fetch_items_total[2h]) == 0` during business hours while other connectors are active. This often indicates lookup feed failures or misconfigured allow-lists.
|
||||
- **Logs** – Surface warnings `Apple document {DocumentId} missing GridFS payload` or `Apple parse failed`—repeated hits imply storage issues or HTML regressions.
|
||||
- **Telemetry pipeline** – `StellaOps.Concelier.WebService` now exports `StellaOps.Concelier.Connector.Vndr.Apple` alongside existing Concelier meters; ensure your OTEL collector or Prometheus scraper includes it.
|
||||
|
||||
## 4. Fixture Maintenance
|
||||
|
||||
Regression fixtures live under `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures`. Refresh them whenever Apple reshapes the HT layout or when new platforms appear.
|
||||
|
||||
1. Run the helper script matching your platform:
|
||||
- Bash: `./scripts/update-apple-fixtures.sh`
|
||||
- PowerShell: `./scripts/update-apple-fixtures.ps1`
|
||||
2. Each script exports `UPDATE_APPLE_FIXTURES=1`, updates the `WSLENV` passthrough, and touches `.update-apple-fixtures` so WSL+VS Code test runs observe the flag. The subsequent test execution fetches the live HT articles listed in `AppleFixtureManager`, sanitises the HTML, and rewrites the `.expected.json` DTO snapshots.
|
||||
3. Review the diff for localisation or nav noise. Once satisfied, re-run the tests without the env var (`dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj`) to verify determinism.
|
||||
4. Commit fixture updates together with any parser/mapping changes that motivated them.
|
||||
|
||||
## 5. Known Issues & Follow-up Tasks
|
||||
|
||||
- Apple occasionally throttles anonymous requests after bursts. The connector backs off automatically, but persistent `apple.fetch.failures` spikes might require mirroring the HT content or scheduling wider fetch windows.
|
||||
- Rapid Security Responses may appear before the general patch notes surface in the lookup JSON. When that happens, the fetch run will log `detailFailures>0`. Collect sample HTML and refresh fixtures to confirm parser coverage.
|
||||
- Multi-locale content is still under regression sweep (`src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md`). Capture non-`en-us` snapshots once the fixture tooling stabilises.
|
||||
# Concelier Apple Security Update Connector Operations
|
||||
|
||||
This runbook covers staging and production rollout for the Apple security updates connector (`source:vndr-apple:*`), including observability checks and fixture maintenance.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Network egress (or mirrored cache) for `https://gdmf.apple.com/v2/pmv` and the Apple Support domain (`https://support.apple.com/`).
|
||||
- Optional: corporate proxy exclusions for the Apple hosts if outbound traffic is normally filtered.
|
||||
- Updated configuration (environment variables or `concelier.yaml`) with an `apple` section. Example baseline:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
apple:
|
||||
softwareLookupUri: "https://gdmf.apple.com/v2/pmv"
|
||||
advisoryBaseUri: "https://support.apple.com/"
|
||||
localeSegment: "en-us"
|
||||
maxAdvisoriesPerFetch: 25
|
||||
initialBackfill: "120.00:00:00"
|
||||
modifiedTolerance: "02:00:00"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> ℹ️ `softwareLookupUri` and `advisoryBaseUri` must stay absolute and aligned with the HTTP allow-list; Concelier automatically adds both hosts to the connector HttpClient.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Deploy the configuration and restart the Concelier workers to ensure the Apple connector options are bound.
|
||||
2. Trigger a full connector cycle:
|
||||
- CLI: `stella db jobs run source:vndr-apple:fetch --and-then source:vndr-apple:parse --and-then source:vndr-apple:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:vndr-apple:fetch", "chain": ["source:vndr-apple:parse", "source:vndr-apple:map"] }`
|
||||
3. Validate metrics exported under meter `StellaOps.Concelier.Connector.Vndr.Apple`:
|
||||
- `apple.fetch.items` (documents fetched)
|
||||
- `apple.fetch.failures`
|
||||
- `apple.fetch.unchanged`
|
||||
- `apple.parse.failures`
|
||||
- `apple.map.affected.count` (histogram of affected package counts)
|
||||
4. Cross-check the shared HTTP counters:
|
||||
- `concelier.source.http.requests_total{concelier_source="vndr-apple"}` should increase for both index and detail phases.
|
||||
- `concelier.source.http.failures_total{concelier_source="vndr-apple"}` should remain flat (0) during a healthy run.
|
||||
5. Inspect the info logs:
|
||||
- `Apple software index fetch … processed=X newDocuments=Y`
|
||||
- `Apple advisory parse complete … aliases=… affected=…`
|
||||
- `Mapped Apple advisory … pendingMappings=0`
|
||||
6. Confirm MongoDB state:
|
||||
- `raw_documents` store contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||||
- `dtos` store has `schemaVersion="apple.security.update.v1"`.
|
||||
- `advisories` collection includes keys `HTxxxxxx` with normalized SemVer rules.
|
||||
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
- **Dashboards** – Add the following expressions to your Concelier Grafana board (OTLP/Prometheus naming assumed):
|
||||
- `rate(apple_fetch_items_total[15m])` vs `rate(concelier_source_http_requests_total{concelier_source="vndr-apple"}[15m])`
|
||||
- `rate(apple_fetch_failures_total[5m])` for error spikes (`severity=warning` at `>0`)
|
||||
- `histogram_quantile(0.95, rate(apple_map_affected_count_bucket[1h]))` to watch affected-package fan-out
|
||||
- `increase(apple_parse_failures_total[6h])` to catch parser drift (alerts at `>0`)
|
||||
- **Alerts** – Page if `rate(apple_fetch_items_total[2h]) == 0` during business hours while other connectors are active. This often indicates lookup feed failures or misconfigured allow-lists.
|
||||
- **Logs** – Surface warnings `Apple document {DocumentId} missing GridFS payload` or `Apple parse failed`—repeated hits imply storage issues or HTML regressions.
|
||||
- **Telemetry pipeline** – `StellaOps.Concelier.WebService` now exports `StellaOps.Concelier.Connector.Vndr.Apple` alongside existing Concelier meters; ensure your OTEL collector or Prometheus scraper includes it.
|
||||
|
||||
## 4. Fixture Maintenance
|
||||
|
||||
Regression fixtures live under `src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/Apple/Fixtures`. Refresh them whenever Apple reshapes the HT layout or when new platforms appear.
|
||||
|
||||
1. Run the helper script matching your platform:
|
||||
- Bash: `./scripts/update-apple-fixtures.sh`
|
||||
- PowerShell: `./scripts/update-apple-fixtures.ps1`
|
||||
2. Each script exports `UPDATE_APPLE_FIXTURES=1`, updates the `WSLENV` passthrough, and touches `.update-apple-fixtures` so WSL+VS Code test runs observe the flag. The subsequent test execution fetches the live HT articles listed in `AppleFixtureManager`, sanitises the HTML, and rewrites the `.expected.json` DTO snapshots.
|
||||
3. Review the diff for localisation or nav noise. Once satisfied, re-run the tests without the env var (`dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj`) to verify determinism.
|
||||
4. Commit fixture updates together with any parser/mapping changes that motivated them.
|
||||
|
||||
## 5. Known Issues & Follow-up Tasks
|
||||
|
||||
- Apple occasionally throttles anonymous requests after bursts. The connector backs off automatically, but persistent `apple.fetch.failures` spikes might require mirroring the HT content or scheduling wider fetch windows.
|
||||
- Rapid Security Responses may appear before the general patch notes surface in the lookup JSON. When that happens, the fetch run will log `detailFailures>0`. Collect sample HTML and refresh fixtures to confirm parser coverage.
|
||||
- Multi-locale content is still under regression sweep (`src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md`). Capture non-`en-us` snapshots once the fixture tooling stabilises.
|
||||
|
||||
@@ -1,72 +1,72 @@
|
||||
# Concelier CCCS Connector Operations
|
||||
|
||||
This runbook covers day‑to‑day operation of the Canadian Centre for Cyber Security (`source:cccs:*`) connector, including configuration, telemetry, and historical backfill guidance for English/French advisories.
|
||||
|
||||
## 1. Configuration Checklist
|
||||
|
||||
- Network egress (or mirrored cache) for `https://www.cyber.gc.ca/` and the JSON API endpoints under `/api/cccs/`.
|
||||
- Set the Concelier options before restarting workers. Example `concelier.yaml` snippet:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
cccs:
|
||||
feeds:
|
||||
- language: "en"
|
||||
uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=en&content_type=cccs_threat"
|
||||
- language: "fr"
|
||||
uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=fr&content_type=cccs_threat"
|
||||
maxEntriesPerFetch: 80 # increase temporarily for backfill runs
|
||||
maxKnownEntries: 512
|
||||
requestTimeout: "00:00:30"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> ℹ️ The `/api/cccs/threats/v1/get` endpoint returns thousands of records per language (≈5 100 rows each as of 2025‑10‑14). The connector honours `maxEntriesPerFetch`, so leave it low for steady‑state and raise it for planned backfills.
|
||||
|
||||
## 2. Telemetry & Logging
|
||||
|
||||
- **Metrics (Meter `StellaOps.Concelier.Connector.Cccs`):**
|
||||
- `cccs.fetch.attempts`, `cccs.fetch.success`, `cccs.fetch.failures`
|
||||
- `cccs.fetch.documents`, `cccs.fetch.unchanged`
|
||||
- `cccs.parse.success`, `cccs.parse.failures`, `cccs.parse.quarantine`
|
||||
- `cccs.map.success`, `cccs.map.failures`
|
||||
- **Shared HTTP metrics** via `SourceDiagnostics`:
|
||||
- `concelier.source.http.requests{concelier.source="cccs"}`
|
||||
- `concelier.source.http.failures{concelier.source="cccs"}`
|
||||
- `concelier.source.http.duration{concelier.source="cccs"}`
|
||||
- **Structured logs**
|
||||
- `CCCS fetch completed feeds=… items=… newDocuments=… pendingDocuments=…`
|
||||
- `CCCS parse completed parsed=… failures=…`
|
||||
- `CCCS map completed mapped=… failures=…`
|
||||
- Warnings fire when GridFS payloads/DTOs go missing or parser sanitisation fails.
|
||||
|
||||
Suggested Grafana alerts:
|
||||
- `increase(cccs.fetch.failures_total[15m]) > 0`
|
||||
- `rate(cccs.map.success_total[1h]) == 0` while other connectors are active
|
||||
- `histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cccs"}[1h])) > 5s`
|
||||
|
||||
## 3. Historical Backfill Plan
|
||||
|
||||
1. **Snapshot the source** – the API accepts `page=<n>` and `lang=<en|fr>` query parameters. `page=0` returns the full dataset (observed earliest `date_created`: 2018‑06‑08 for EN, 2018‑06‑08 for FR). Mirror those responses into Offline Kit storage when operating air‑gapped.
|
||||
2. **Stage ingestion**:
|
||||
- Temporarily raise `maxEntriesPerFetch` (e.g. 500) and restart Concelier workers.
|
||||
- Run chained jobs until `pendingDocuments` drains:
|
||||
`stella db jobs run source:cccs:fetch --and-then source:cccs:parse --and-then source:cccs:map`
|
||||
- Monitor `cccs.fetch.unchanged` growth; once it approaches dataset size the backfill is complete.
|
||||
3. **Optional pagination sweep** – for incremental mirrors, iterate `page=<n>` (0…N) while `response.Count == 50`, persisting JSON to disk. Store alongside metadata (`language`, `page`, SHA256) so repeated runs detect drift.
|
||||
4. **Language split** – keep EN/FR payloads separate to preserve canonical language fields. The connector emits `Language` directly from the feed entry, so mixed ingestion simply produces parallel advisories keyed by the same serial number.
|
||||
5. **Throttle planning** – schedule backfills during maintenance windows; the API tolerates burst downloads but respect the 250 ms request delay or raise it if mirrored traffic is not available.
|
||||
|
||||
## 4. Selector & Sanitiser Notes
|
||||
|
||||
- `CccsHtmlParser` now parses the **unsanitised DOM** (via AngleSharp) and only sanitises when persisting `ContentHtml`.
|
||||
- Product extraction walks headings (`Affected Products`, `Produits touchés`, `Mesures recommandées`) and consumes nested lists within `div/section/article` containers.
|
||||
- `HtmlContentSanitizer` allows `<h1>…<h6>` and `<section>` so stored HTML keeps headings for UI rendering and downstream summarisation.
|
||||
|
||||
## 5. Fixture Maintenance
|
||||
|
||||
- Regression fixtures live in `src/Concelier/__Tests/StellaOps.Concelier.Connector.Cccs.Tests/Fixtures`.
|
||||
- Refresh via `UPDATE_CCCS_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Cccs.Tests/StellaOps.Concelier.Connector.Cccs.Tests.csproj`.
|
||||
- Fixtures capture both EN/FR advisories with nested lists to guard against sanitiser regressions; review diffs for heading/list changes before committing.
|
||||
# Concelier CCCS Connector Operations
|
||||
|
||||
This runbook covers day‑to‑day operation of the Canadian Centre for Cyber Security (`source:cccs:*`) connector, including configuration, telemetry, and historical backfill guidance for English/French advisories.
|
||||
|
||||
## 1. Configuration Checklist
|
||||
|
||||
- Network egress (or mirrored cache) for `https://www.cyber.gc.ca/` and the JSON API endpoints under `/api/cccs/`.
|
||||
- Set the Concelier options before restarting workers. Example `concelier.yaml` snippet:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
cccs:
|
||||
feeds:
|
||||
- language: "en"
|
||||
uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=en&content_type=cccs_threat"
|
||||
- language: "fr"
|
||||
uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=fr&content_type=cccs_threat"
|
||||
maxEntriesPerFetch: 80 # increase temporarily for backfill runs
|
||||
maxKnownEntries: 512
|
||||
requestTimeout: "00:00:30"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> ℹ️ The `/api/cccs/threats/v1/get` endpoint returns thousands of records per language (≈5 100 rows each as of 2025‑10‑14). The connector honours `maxEntriesPerFetch`, so leave it low for steady‑state and raise it for planned backfills.
|
||||
|
||||
## 2. Telemetry & Logging
|
||||
|
||||
- **Metrics (Meter `StellaOps.Concelier.Connector.Cccs`):**
|
||||
- `cccs.fetch.attempts`, `cccs.fetch.success`, `cccs.fetch.failures`
|
||||
- `cccs.fetch.documents`, `cccs.fetch.unchanged`
|
||||
- `cccs.parse.success`, `cccs.parse.failures`, `cccs.parse.quarantine`
|
||||
- `cccs.map.success`, `cccs.map.failures`
|
||||
- **Shared HTTP metrics** via `SourceDiagnostics`:
|
||||
- `concelier.source.http.requests{concelier.source="cccs"}`
|
||||
- `concelier.source.http.failures{concelier.source="cccs"}`
|
||||
- `concelier.source.http.duration{concelier.source="cccs"}`
|
||||
- **Structured logs**
|
||||
- `CCCS fetch completed feeds=… items=… newDocuments=… pendingDocuments=…`
|
||||
- `CCCS parse completed parsed=… failures=…`
|
||||
- `CCCS map completed mapped=… failures=…`
|
||||
- Warnings fire when GridFS payloads/DTOs go missing or parser sanitisation fails.
|
||||
|
||||
Suggested Grafana alerts:
|
||||
- `increase(cccs.fetch.failures_total[15m]) > 0`
|
||||
- `rate(cccs.map.success_total[1h]) == 0` while other connectors are active
|
||||
- `histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cccs"}[1h])) > 5s`
|
||||
|
||||
## 3. Historical Backfill Plan
|
||||
|
||||
1. **Snapshot the source** – the API accepts `page=<n>` and `lang=<en|fr>` query parameters. `page=0` returns the full dataset (observed earliest `date_created`: 2018‑06‑08 for EN, 2018‑06‑08 for FR). Mirror those responses into Offline Kit storage when operating air‑gapped.
|
||||
2. **Stage ingestion**:
|
||||
- Temporarily raise `maxEntriesPerFetch` (e.g. 500) and restart Concelier workers.
|
||||
- Run chained jobs until `pendingDocuments` drains:
|
||||
`stella db jobs run source:cccs:fetch --and-then source:cccs:parse --and-then source:cccs:map`
|
||||
- Monitor `cccs.fetch.unchanged` growth; once it approaches dataset size the backfill is complete.
|
||||
3. **Optional pagination sweep** – for incremental mirrors, iterate `page=<n>` (0…N) while `response.Count == 50`, persisting JSON to disk. Store alongside metadata (`language`, `page`, SHA256) so repeated runs detect drift.
|
||||
4. **Language split** – keep EN/FR payloads separate to preserve canonical language fields. The connector emits `Language` directly from the feed entry, so mixed ingestion simply produces parallel advisories keyed by the same serial number.
|
||||
5. **Throttle planning** – schedule backfills during maintenance windows; the API tolerates burst downloads but respect the 250 ms request delay or raise it if mirrored traffic is not available.
|
||||
|
||||
## 4. Selector & Sanitiser Notes
|
||||
|
||||
- `CccsHtmlParser` now parses the **unsanitised DOM** (via AngleSharp) and only sanitises when persisting `ContentHtml`.
|
||||
- Product extraction walks headings (`Affected Products`, `Produits touchés`, `Mesures recommandées`) and consumes nested lists within `div/section/article` containers.
|
||||
- `HtmlContentSanitizer` allows `<h1>…<h6>` and `<section>` so stored HTML keeps headings for UI rendering and downstream summarisation.
|
||||
|
||||
## 5. Fixture Maintenance
|
||||
|
||||
- Regression fixtures live in `src/Concelier/__Tests/StellaOps.Concelier.Connector.Cccs.Tests/Fixtures`.
|
||||
- Refresh via `UPDATE_CCCS_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Cccs.Tests/StellaOps.Concelier.Connector.Cccs.Tests.csproj`.
|
||||
- Fixtures capture both EN/FR advisories with nested lists to guard against sanitiser regressions; review diffs for heading/list changes before committing.
|
||||
|
||||
@@ -1,143 +1,143 @@
|
||||
# Concelier CVE & KEV Connector Operations
|
||||
|
||||
This playbook equips operators with the steps required to roll out and monitor the CVE Services and CISA KEV connectors across environments.
|
||||
|
||||
## 1. CVE Services Connector (`source:cve:*`)
|
||||
|
||||
### 1.1 Prerequisites
|
||||
|
||||
- CVE Services API credentials (organisation ID, user ID, API key) with access to the JSON 5 API.
|
||||
- Network egress to `https://cveawg.mitre.org` (or a mirrored endpoint) from the Concelier workers.
|
||||
- Updated `concelier.yaml` (or the matching environment variables) with the following section:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
cve:
|
||||
baseEndpoint: "https://cveawg.mitre.org/api/"
|
||||
apiOrg: "ORG123"
|
||||
apiUser: "user@example.org"
|
||||
apiKeyFile: "/var/run/secrets/concelier/cve-api-key"
|
||||
seedDirectory: "./seed-data/cve"
|
||||
pageSize: 200
|
||||
maxPagesPerFetch: 5
|
||||
initialBackfill: "30.00:00:00"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:10:00"
|
||||
```
|
||||
|
||||
> ℹ️ Store the API key outside source control. When using `apiKeyFile`, mount the secret file into the container/host; alternatively supply `apiKey` via `CONCELIER_SOURCES__CVE__APIKEY`.
|
||||
|
||||
> 🪙 When credentials are not yet available, configure `seedDirectory` to point at mirrored CVE JSON (for example, the repo’s `seed-data/cve/` bundle). The connector will ingest those records and log a warning instead of failing the job; live fetching resumes automatically once `apiOrg` / `apiUser` / `apiKey` are supplied.
|
||||
|
||||
### 1.2 Smoke Test (staging)
|
||||
|
||||
1. Deploy the updated configuration and restart the Concelier service so the connector picks up the credentials.
|
||||
2. Trigger one end-to-end cycle:
|
||||
- Concelier CLI: `stella db jobs run source:cve:fetch --and-then source:cve:parse --and-then source:cve:map`
|
||||
- REST fallback: `POST /jobs/run { "kind": "source:cve:fetch", "chain": ["source:cve:parse", "source:cve:map"] }`
|
||||
3. Observe the following metrics (exported via OTEL meter `StellaOps.Concelier.Connector.Cve`):
|
||||
- `cve.fetch.attempts`, `cve.fetch.success`, `cve.fetch.documents`, `cve.fetch.failures`, `cve.fetch.unchanged`
|
||||
- `cve.parse.success`, `cve.parse.failures`, `cve.parse.quarantine`
|
||||
- `cve.map.success`
|
||||
4. Verify Prometheus shows matching `concelier.source.http.requests_total{concelier_source="cve"}` deltas (list vs detail phases) while `concelier.source.http.failures_total{concelier_source="cve"}` stays flat.
|
||||
5. Confirm the info-level summary log `CVEs fetch window … pages=X detailDocuments=Y detailFailures=Z` appears once per fetch run and shows `detailFailures=0`.
|
||||
6. Verify the MongoDB advisory store contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` collection) advanced.
|
||||
|
||||
### 1.3 Production Monitoring
|
||||
|
||||
- **Dashboards** – Plot `rate(cve_fetch_success_total[5m])`, `rate(cve_fetch_failures_total[5m])`, and `rate(cve_fetch_documents_total[5m])` alongside `concelier_source_http_requests_total{concelier_source="cve"}` to confirm HTTP and connector counters stay aligned. Keep `concelier.range.primitives{scheme=~"semver|vendor"}` on the same board for range coverage. Example alerts:
|
||||
- `rate(cve_fetch_failures_total[5m]) > 0` for 10 minutes (`severity=warning`)
|
||||
- `rate(cve_map_success_total[15m]) == 0` while `rate(cve_fetch_success_total[15m]) > 0` (`severity=critical`)
|
||||
- `sum_over_time(cve_parse_quarantine_total[1h]) > 0` to catch schema anomalies
|
||||
- **Logs** – Monitor warnings such as `Failed fetching CVE record {CveId}` and `Malformed CVE JSON`, and surface the summary info log `CVEs fetch window … detailFailures=0 detailUnchanged=0` on dashboards. A non-zero `detailFailures` usually indicates rate-limit or auth issues on detail requests.
|
||||
- **Grafana pack** – Import `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` and filter by panel legend (`CVE`, `KEV`) to reuse the canned layout.
|
||||
- **Backfill window** – Operators can tighten or widen `initialBackfill` / `maxPagesPerFetch` after validating throughput. Update config and restart Concelier to apply changes.
|
||||
|
||||
### 1.4 Staging smoke log (2025-10-15)
|
||||
|
||||
While Ops finalises long-lived CVE Services credentials, we validated the connector end-to-end against the recorded CVE-2024-0001 payloads used in regression tests:
|
||||
|
||||
- Command: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve.Tests/StellaOps.Concelier.Connector.Cve.Tests.csproj -l "console;verbosity=detailed"`
|
||||
- Summary log emitted by the connector:
|
||||
```
|
||||
CVEs fetch window 2024-09-01T00:00:00Z->2024-10-01T00:00:00Z pages=1 listSuccess=1 detailDocuments=1 detailFailures=0 detailUnchanged=0 pendingDocuments=0->1 pendingMappings=0->1 hasMorePages=False nextWindowStart=2024-09-15T12:00:00Z nextWindowEnd=(none) nextPage=1
|
||||
```
|
||||
- Telemetry captured by `Meter` `StellaOps.Concelier.Connector.Cve`:
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| `cve.fetch.attempts` | 1 |
|
||||
| `cve.fetch.success` | 1 |
|
||||
| `cve.fetch.documents` | 1 |
|
||||
| `cve.parse.success` | 1 |
|
||||
| `cve.map.success` | 1 |
|
||||
|
||||
The Grafana pack `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` has been imported into staging so the panels referenced above render against these counters once the live API keys are in place.
|
||||
|
||||
## 2. CISA KEV Connector (`source:kev:*`)
|
||||
|
||||
### 2.1 Prerequisites
|
||||
|
||||
- Network egress (or mirrored content) for `https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json`.
|
||||
- No credentials are required, but the HTTP allow-list must include `www.cisa.gov`.
|
||||
- Confirm the following snippet in `concelier.yaml` (defaults shown; tune as needed):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
kev:
|
||||
feedUri: "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"
|
||||
requestTimeout: "00:01:00"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
### 2.2 Schema validation & anomaly handling
|
||||
|
||||
The connector validates each catalog against `Schemas/kev-catalog.schema.json`. Failures increment `kev.parse.failures_total{reason="schema"}` and the document is quarantined (status `Failed`). Additional failure reasons include `download`, `invalidJson`, `deserialize`, `missingPayload`, and `emptyCatalog`. Entry-level anomalies are surfaced through `kev.parse.anomalies_total` with reasons:
|
||||
|
||||
| Reason | Meaning |
|
||||
| --- | --- |
|
||||
| `missingCveId` | Catalog entry omitted `cveID`; the entry is skipped. |
|
||||
| `countMismatch` | Catalog `count` field disagreed with the actual entry total. |
|
||||
| `nullEntry` | Upstream emitted a `null` entry object (rare upstream defect). |
|
||||
|
||||
Treat repeated schema failures or growing anomaly counts as an upstream regression and coordinate with CISA or mirror maintainers.
|
||||
|
||||
### 2.3 Smoke Test (staging)
|
||||
|
||||
1. Deploy the configuration and restart Concelier.
|
||||
2. Trigger a pipeline run:
|
||||
- CLI: `stella db jobs run source:kev:fetch --and-then source:kev:parse --and-then source:kev:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:kev:fetch", "chain": ["source:kev:parse", "source:kev:map"] }`
|
||||
3. Verify the metrics exposed by meter `StellaOps.Concelier.Connector.Kev`:
|
||||
- `kev.fetch.attempts`, `kev.fetch.success`, `kev.fetch.unchanged`, `kev.fetch.failures`
|
||||
- `kev.parse.entries` (tag `catalogVersion`), `kev.parse.failures`, `kev.parse.anomalies` (tag `reason`)
|
||||
- `kev.map.advisories` (tag `catalogVersion`)
|
||||
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
||||
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
||||
6. Confirm MongoDB documents exist for the catalog JSON (`raw_documents` & `dtos`) and that advisories with prefix `kev/` are written.
|
||||
|
||||
### 2.4 Production Monitoring
|
||||
|
||||
- Alert when `rate(kev_fetch_success_total[8h]) == 0` during working hours (daily cadence breach) and when `increase(kev_fetch_failures_total[1h]) > 0`.
|
||||
- Page the on-call if `increase(kev_parse_failures_total{reason="schema"}[6h]) > 0`—this usually signals an upstream payload change. Treat repeated `reason="download"` spikes as networking issues to the mirror.
|
||||
- Track anomaly spikes through `sum_over_time(kev_parse_anomalies_total{reason="missingCveId"}[24h])`. Rising `countMismatch` trends point to catalog publishing bugs.
|
||||
- Surface the fetch/mapping info logs (`Fetched KEV catalog document …` and `Mapped X/Y KEV advisories … skipped=S`) on dashboards; absence of those logs while metrics show success typically means schema validation short-circuited the run.
|
||||
|
||||
### 2.5 Known good dashboard tiles
|
||||
|
||||
Add the following panels to the Concelier observability board:
|
||||
|
||||
| Metric | Recommended visualisation |
|
||||
|--------|---------------------------|
|
||||
| `rate(kev_fetch_success_total[30m])` | Single-stat (last 24 h) with warning threshold `>0` |
|
||||
| `rate(kev_parse_entries_total[1h])` by `catalogVersion` | Stacked area – highlights daily release size |
|
||||
| `sum_over_time(kev_parse_anomalies_total[1d])` by `reason` | Table – anomaly breakdown (matches dashboard panel) |
|
||||
| `rate(cve_map_success_total[15m])` vs `rate(kev_map_advisories_total[24h])` | Comparative timeseries for advisories emitted |
|
||||
|
||||
## 3. Runbook updates
|
||||
|
||||
- Record staging/production smoke test results (date, catalog version, advisory counts) in your team’s change log.
|
||||
- Add the CVE/KEV job kinds to the standard maintenance checklist so operators can manually trigger them after planned downtime.
|
||||
- Keep this document in sync with future connector changes (for example, new anomaly reasons or additional metrics).
|
||||
- Version-control dashboard tweaks alongside `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` so operations can re-import the observability pack during restores.
|
||||
# Concelier CVE & KEV Connector Operations
|
||||
|
||||
This playbook equips operators with the steps required to roll out and monitor the CVE Services and CISA KEV connectors across environments.
|
||||
|
||||
## 1. CVE Services Connector (`source:cve:*`)
|
||||
|
||||
### 1.1 Prerequisites
|
||||
|
||||
- CVE Services API credentials (organisation ID, user ID, API key) with access to the JSON 5 API.
|
||||
- Network egress to `https://cveawg.mitre.org` (or a mirrored endpoint) from the Concelier workers.
|
||||
- Updated `concelier.yaml` (or the matching environment variables) with the following section:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
cve:
|
||||
baseEndpoint: "https://cveawg.mitre.org/api/"
|
||||
apiOrg: "ORG123"
|
||||
apiUser: "user@example.org"
|
||||
apiKeyFile: "/var/run/secrets/concelier/cve-api-key"
|
||||
seedDirectory: "./seed-data/cve"
|
||||
pageSize: 200
|
||||
maxPagesPerFetch: 5
|
||||
initialBackfill: "30.00:00:00"
|
||||
requestDelay: "00:00:00.250"
|
||||
failureBackoff: "00:10:00"
|
||||
```
|
||||
|
||||
> ℹ️ Store the API key outside source control. When using `apiKeyFile`, mount the secret file into the container/host; alternatively supply `apiKey` via `CONCELIER_SOURCES__CVE__APIKEY`.
|
||||
|
||||
> 🪙 When credentials are not yet available, configure `seedDirectory` to point at mirrored CVE JSON (for example, the repo’s `seed-data/cve/` bundle). The connector will ingest those records and log a warning instead of failing the job; live fetching resumes automatically once `apiOrg` / `apiUser` / `apiKey` are supplied.
|
||||
|
||||
### 1.2 Smoke Test (staging)
|
||||
|
||||
1. Deploy the updated configuration and restart the Concelier service so the connector picks up the credentials.
|
||||
2. Trigger one end-to-end cycle:
|
||||
- Concelier CLI: `stella db jobs run source:cve:fetch --and-then source:cve:parse --and-then source:cve:map`
|
||||
- REST fallback: `POST /jobs/run { "kind": "source:cve:fetch", "chain": ["source:cve:parse", "source:cve:map"] }`
|
||||
3. Observe the following metrics (exported via OTEL meter `StellaOps.Concelier.Connector.Cve`):
|
||||
- `cve.fetch.attempts`, `cve.fetch.success`, `cve.fetch.documents`, `cve.fetch.failures`, `cve.fetch.unchanged`
|
||||
- `cve.parse.success`, `cve.parse.failures`, `cve.parse.quarantine`
|
||||
- `cve.map.success`
|
||||
4. Verify Prometheus shows matching `concelier.source.http.requests_total{concelier_source="cve"}` deltas (list vs detail phases) while `concelier.source.http.failures_total{concelier_source="cve"}` stays flat.
|
||||
5. Confirm the info-level summary log `CVEs fetch window … pages=X detailDocuments=Y detailFailures=Z` appears once per fetch run and shows `detailFailures=0`.
|
||||
6. Verify the MongoDB advisory store contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` collection) advanced.
|
||||
|
||||
### 1.3 Production Monitoring
|
||||
|
||||
- **Dashboards** – Plot `rate(cve_fetch_success_total[5m])`, `rate(cve_fetch_failures_total[5m])`, and `rate(cve_fetch_documents_total[5m])` alongside `concelier_source_http_requests_total{concelier_source="cve"}` to confirm HTTP and connector counters stay aligned. Keep `concelier.range.primitives{scheme=~"semver|vendor"}` on the same board for range coverage. Example alerts:
|
||||
- `rate(cve_fetch_failures_total[5m]) > 0` for 10 minutes (`severity=warning`)
|
||||
- `rate(cve_map_success_total[15m]) == 0` while `rate(cve_fetch_success_total[15m]) > 0` (`severity=critical`)
|
||||
- `sum_over_time(cve_parse_quarantine_total[1h]) > 0` to catch schema anomalies
|
||||
- **Logs** – Monitor warnings such as `Failed fetching CVE record {CveId}` and `Malformed CVE JSON`, and surface the summary info log `CVEs fetch window … detailFailures=0 detailUnchanged=0` on dashboards. A non-zero `detailFailures` usually indicates rate-limit or auth issues on detail requests.
|
||||
- **Grafana pack** – Import `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` and filter by panel legend (`CVE`, `KEV`) to reuse the canned layout.
|
||||
- **Backfill window** – Operators can tighten or widen `initialBackfill` / `maxPagesPerFetch` after validating throughput. Update config and restart Concelier to apply changes.
|
||||
|
||||
### 1.4 Staging smoke log (2025-10-15)
|
||||
|
||||
While Ops finalises long-lived CVE Services credentials, we validated the connector end-to-end against the recorded CVE-2024-0001 payloads used in regression tests:
|
||||
|
||||
- Command: `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve.Tests/StellaOps.Concelier.Connector.Cve.Tests.csproj -l "console;verbosity=detailed"`
|
||||
- Summary log emitted by the connector:
|
||||
```
|
||||
CVEs fetch window 2024-09-01T00:00:00Z->2024-10-01T00:00:00Z pages=1 listSuccess=1 detailDocuments=1 detailFailures=0 detailUnchanged=0 pendingDocuments=0->1 pendingMappings=0->1 hasMorePages=False nextWindowStart=2024-09-15T12:00:00Z nextWindowEnd=(none) nextPage=1
|
||||
```
|
||||
- Telemetry captured by `Meter` `StellaOps.Concelier.Connector.Cve`:
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| `cve.fetch.attempts` | 1 |
|
||||
| `cve.fetch.success` | 1 |
|
||||
| `cve.fetch.documents` | 1 |
|
||||
| `cve.parse.success` | 1 |
|
||||
| `cve.map.success` | 1 |
|
||||
|
||||
The Grafana pack `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` has been imported into staging so the panels referenced above render against these counters once the live API keys are in place.
|
||||
|
||||
## 2. CISA KEV Connector (`source:kev:*`)
|
||||
|
||||
### 2.1 Prerequisites
|
||||
|
||||
- Network egress (or mirrored content) for `https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json`.
|
||||
- No credentials are required, but the HTTP allow-list must include `www.cisa.gov`.
|
||||
- Confirm the following snippet in `concelier.yaml` (defaults shown; tune as needed):
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
kev:
|
||||
feedUri: "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"
|
||||
requestTimeout: "00:01:00"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
### 2.2 Schema validation & anomaly handling
|
||||
|
||||
The connector validates each catalog against `Schemas/kev-catalog.schema.json`. Failures increment `kev.parse.failures_total{reason="schema"}` and the document is quarantined (status `Failed`). Additional failure reasons include `download`, `invalidJson`, `deserialize`, `missingPayload`, and `emptyCatalog`. Entry-level anomalies are surfaced through `kev.parse.anomalies_total` with reasons:
|
||||
|
||||
| Reason | Meaning |
|
||||
| --- | --- |
|
||||
| `missingCveId` | Catalog entry omitted `cveID`; the entry is skipped. |
|
||||
| `countMismatch` | Catalog `count` field disagreed with the actual entry total. |
|
||||
| `nullEntry` | Upstream emitted a `null` entry object (rare upstream defect). |
|
||||
|
||||
Treat repeated schema failures or growing anomaly counts as an upstream regression and coordinate with CISA or mirror maintainers.
|
||||
|
||||
### 2.3 Smoke Test (staging)
|
||||
|
||||
1. Deploy the configuration and restart Concelier.
|
||||
2. Trigger a pipeline run:
|
||||
- CLI: `stella db jobs run source:kev:fetch --and-then source:kev:parse --and-then source:kev:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:kev:fetch", "chain": ["source:kev:parse", "source:kev:map"] }`
|
||||
3. Verify the metrics exposed by meter `StellaOps.Concelier.Connector.Kev`:
|
||||
- `kev.fetch.attempts`, `kev.fetch.success`, `kev.fetch.unchanged`, `kev.fetch.failures`
|
||||
- `kev.parse.entries` (tag `catalogVersion`), `kev.parse.failures`, `kev.parse.anomalies` (tag `reason`)
|
||||
- `kev.map.advisories` (tag `catalogVersion`)
|
||||
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
||||
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
||||
6. Confirm MongoDB documents exist for the catalog JSON (`raw_documents` & `dtos`) and that advisories with prefix `kev/` are written.
|
||||
|
||||
### 2.4 Production Monitoring
|
||||
|
||||
- Alert when `rate(kev_fetch_success_total[8h]) == 0` during working hours (daily cadence breach) and when `increase(kev_fetch_failures_total[1h]) > 0`.
|
||||
- Page the on-call if `increase(kev_parse_failures_total{reason="schema"}[6h]) > 0`—this usually signals an upstream payload change. Treat repeated `reason="download"` spikes as networking issues to the mirror.
|
||||
- Track anomaly spikes through `sum_over_time(kev_parse_anomalies_total{reason="missingCveId"}[24h])`. Rising `countMismatch` trends point to catalog publishing bugs.
|
||||
- Surface the fetch/mapping info logs (`Fetched KEV catalog document …` and `Mapped X/Y KEV advisories … skipped=S`) on dashboards; absence of those logs while metrics show success typically means schema validation short-circuited the run.
|
||||
|
||||
### 2.5 Known good dashboard tiles
|
||||
|
||||
Add the following panels to the Concelier observability board:
|
||||
|
||||
| Metric | Recommended visualisation |
|
||||
|--------|---------------------------|
|
||||
| `rate(kev_fetch_success_total[30m])` | Single-stat (last 24 h) with warning threshold `>0` |
|
||||
| `rate(kev_parse_entries_total[1h])` by `catalogVersion` | Stacked area – highlights daily release size |
|
||||
| `sum_over_time(kev_parse_anomalies_total[1d])` by `reason` | Table – anomaly breakdown (matches dashboard panel) |
|
||||
| `rate(cve_map_success_total[15m])` vs `rate(kev_map_advisories_total[24h])` | Comparative timeseries for advisories emitted |
|
||||
|
||||
## 3. Runbook updates
|
||||
|
||||
- Record staging/production smoke test results (date, catalog version, advisory counts) in your team’s change log.
|
||||
- Add the CVE/KEV job kinds to the standard maintenance checklist so operators can manually trigger them after planned downtime.
|
||||
- Keep this document in sync with future connector changes (for example, new anomaly reasons or additional metrics).
|
||||
- Version-control dashboard tweaks alongside `docs/modules/concelier/operations/connectors/cve-kev-grafana-dashboard.json` so operations can re-import the observability pack during restores.
|
||||
|
||||
@@ -1,74 +1,74 @@
|
||||
# Concelier KISA Connector Operations
|
||||
|
||||
Operational guidance for the Korea Internet & Security Agency (KISA / KNVD) connector (`source:kisa:*`). Pair this with the engineering brief in `docs/dev/kisa_connector_notes.md`.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Outbound HTTPS (or mirrored cache) for `https://knvd.krcert.or.kr/`.
|
||||
- Connector options defined under `concelier:sources:kisa`:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
kisa:
|
||||
feedUri: "https://knvd.krcert.or.kr/rss/securityInfo.do"
|
||||
detailApiUri: "https://knvd.krcert.or.kr/rssDetailData.do"
|
||||
detailPageUri: "https://knvd.krcert.or.kr/detailDos.do"
|
||||
maxAdvisoriesPerFetch: 10
|
||||
requestDelay: "00:00:01"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> Ensure the URIs stay absolute—Concelier adds the `feedUri`/`detailApiUri` hosts to the HttpClient allow-list automatically.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Restart the Concelier workers so the KISA options bind.
|
||||
2. Run a full connector cycle:
|
||||
- CLI: `stella db jobs run source:kisa:fetch --and-then source:kisa:parse --and-then source:kisa:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:kisa:fetch", "chain": ["source:kisa:parse", "source:kisa:map"] }`
|
||||
3. Confirm telemetry (Meter `StellaOps.Concelier.Connector.Kisa`):
|
||||
- `kisa.feed.success`, `kisa.feed.items`
|
||||
- `kisa.detail.success` / `.failures`
|
||||
- `kisa.parse.success` / `.failures`
|
||||
- `kisa.map.success` / `.failures`
|
||||
- `kisa.cursor.updates`
|
||||
4. Inspect logs for structured entries:
|
||||
- `KISA feed returned {ItemCount}`
|
||||
- `KISA fetched detail for {Idx} … category={Category}`
|
||||
- `KISA mapped advisory {AdvisoryId} (severity={Severity})`
|
||||
- Absence of warnings such as `document missing GridFS payload`.
|
||||
5. Validate MongoDB state:
|
||||
- `raw_documents.metadata` has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||
- DTO store contains `schemaVersion="kisa.detail.v1"`.
|
||||
- Advisories include aliases (`IDX`, CVE) and `language="ko"`.
|
||||
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
- **Dashboards** – Add the following Prometheus/OTEL expressions:
|
||||
- `rate(kisa_feed_items_total[15m])` versus `rate(concelier_source_http_requests_total{concelier_source="kisa"}[15m])`
|
||||
- `increase(kisa_detail_failures_total{reason!="empty-document"}[1h])` alert at `>0`
|
||||
- `increase(kisa_parse_failures_total[1h])` for storage/JSON issues
|
||||
- `increase(kisa_map_failures_total[1h])` to flag schema drift
|
||||
- `increase(kisa_cursor_updates_total[6h]) == 0` during active windows → warn
|
||||
- **Alerts** – Page when `rate(kisa_feed_success_total[2h]) == 0` while other connectors are active; back off for maintenance windows announced on `https://knvd.krcert.or.kr/`.
|
||||
- **Logs** – Watch for repeated warnings (`document missing`, `DTO missing`) or errors with reason tags `HttpRequestException`, `download`, `parse`, `map`.
|
||||
|
||||
## 4. Localisation Handling
|
||||
|
||||
- Hangul categories (for example `취약점정보`) flow into telemetry tags (`category=…`) and logs. Dashboards must render UTF‑8 and avoid transliteration.
|
||||
- HTML content is sanitised before storage; translation teams can consume the `ContentHtml` field safely.
|
||||
- Advisory severity remains as provided by KISA (`High`, `Medium`, etc.). Map-level failures include the severity tag for filtering.
|
||||
|
||||
## 5. Fixture & Regression Maintenance
|
||||
|
||||
- Regression fixtures: `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-feed.xml` and `kisa-detail.json`.
|
||||
- Refresh via `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`.
|
||||
- The telemetry regression (`KisaConnectorTests.Telemetry_RecordsMetrics`) will fail if counters/log wiring drifts—treat failures as gating.
|
||||
|
||||
## 6. Known Issues
|
||||
|
||||
- RSS feeds only expose the latest 10 advisories; long outages require replay via archived feeds or manual IDX seeds.
|
||||
- Detail endpoint occasionally throttles; the connector honours `requestDelay` and reports failures with reason `HttpRequestException`. Consider increasing delay for weekend backfills.
|
||||
- If `kisa.category` tags suddenly appear as `unknown`, verify KISA has not renamed RSS elements; update the parser fixtures before production rollout.
|
||||
# Concelier KISA Connector Operations
|
||||
|
||||
Operational guidance for the Korea Internet & Security Agency (KISA / KNVD) connector (`source:kisa:*`). Pair this with the engineering brief in `docs/dev/kisa_connector_notes.md`.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Outbound HTTPS (or mirrored cache) for `https://knvd.krcert.or.kr/`.
|
||||
- Connector options defined under `concelier:sources:kisa`:
|
||||
|
||||
```yaml
|
||||
concelier:
|
||||
sources:
|
||||
kisa:
|
||||
feedUri: "https://knvd.krcert.or.kr/rss/securityInfo.do"
|
||||
detailApiUri: "https://knvd.krcert.or.kr/rssDetailData.do"
|
||||
detailPageUri: "https://knvd.krcert.or.kr/detailDos.do"
|
||||
maxAdvisoriesPerFetch: 10
|
||||
requestDelay: "00:00:01"
|
||||
failureBackoff: "00:05:00"
|
||||
```
|
||||
|
||||
> Ensure the URIs stay absolute—Concelier adds the `feedUri`/`detailApiUri` hosts to the HttpClient allow-list automatically.
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Restart the Concelier workers so the KISA options bind.
|
||||
2. Run a full connector cycle:
|
||||
- CLI: `stella db jobs run source:kisa:fetch --and-then source:kisa:parse --and-then source:kisa:map`
|
||||
- REST: `POST /jobs/run { "kind": "source:kisa:fetch", "chain": ["source:kisa:parse", "source:kisa:map"] }`
|
||||
3. Confirm telemetry (Meter `StellaOps.Concelier.Connector.Kisa`):
|
||||
- `kisa.feed.success`, `kisa.feed.items`
|
||||
- `kisa.detail.success` / `.failures`
|
||||
- `kisa.parse.success` / `.failures`
|
||||
- `kisa.map.success` / `.failures`
|
||||
- `kisa.cursor.updates`
|
||||
4. Inspect logs for structured entries:
|
||||
- `KISA feed returned {ItemCount}`
|
||||
- `KISA fetched detail for {Idx} … category={Category}`
|
||||
- `KISA mapped advisory {AdvisoryId} (severity={Severity})`
|
||||
- Absence of warnings such as `document missing GridFS payload`.
|
||||
5. Validate MongoDB state:
|
||||
- `raw_documents.metadata` has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||
- DTO store contains `schemaVersion="kisa.detail.v1"`.
|
||||
- Advisories include aliases (`IDX`, CVE) and `language="ko"`.
|
||||
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
- **Dashboards** – Add the following Prometheus/OTEL expressions:
|
||||
- `rate(kisa_feed_items_total[15m])` versus `rate(concelier_source_http_requests_total{concelier_source="kisa"}[15m])`
|
||||
- `increase(kisa_detail_failures_total{reason!="empty-document"}[1h])` alert at `>0`
|
||||
- `increase(kisa_parse_failures_total[1h])` for storage/JSON issues
|
||||
- `increase(kisa_map_failures_total[1h])` to flag schema drift
|
||||
- `increase(kisa_cursor_updates_total[6h]) == 0` during active windows → warn
|
||||
- **Alerts** – Page when `rate(kisa_feed_success_total[2h]) == 0` while other connectors are active; back off for maintenance windows announced on `https://knvd.krcert.or.kr/`.
|
||||
- **Logs** – Watch for repeated warnings (`document missing`, `DTO missing`) or errors with reason tags `HttpRequestException`, `download`, `parse`, `map`.
|
||||
|
||||
## 4. Localisation Handling
|
||||
|
||||
- Hangul categories (for example `취약점정보`) flow into telemetry tags (`category=…`) and logs. Dashboards must render UTF‑8 and avoid transliteration.
|
||||
- HTML content is sanitised before storage; translation teams can consume the `ContentHtml` field safely.
|
||||
- Advisory severity remains as provided by KISA (`High`, `Medium`, etc.). Map-level failures include the severity tag for filtering.
|
||||
|
||||
## 5. Fixture & Regression Maintenance
|
||||
|
||||
- Regression fixtures: `src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/Fixtures/kisa-feed.xml` and `kisa-detail.json`.
|
||||
- Refresh via `UPDATE_KISA_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Kisa.Tests/StellaOps.Concelier.Connector.Kisa.Tests.csproj`.
|
||||
- The telemetry regression (`KisaConnectorTests.Telemetry_RecordsMetrics`) will fail if counters/log wiring drifts—treat failures as gating.
|
||||
|
||||
## 6. Known Issues
|
||||
|
||||
- RSS feeds only expose the latest 10 advisories; long outages require replay via archived feeds or manual IDX seeds.
|
||||
- Detail endpoint occasionally throttles; the connector honours `requestDelay` and reports failures with reason `HttpRequestException`. Consider increasing delay for weekend backfills.
|
||||
- If `kisa.category` tags suddenly appear as `unknown`, verify KISA has not renamed RSS elements; update the parser fixtures before production rollout.
|
||||
|
||||
@@ -1,48 +1,48 @@
|
||||
# NKCKI Connector Operations Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The NKCKI connector ingests JSON bulletin archives from cert.gov.ru, expanding each `*.json.zip` attachment into per-vulnerability DTOs before canonical mapping. The fetch pipeline now supports cache-backed recovery, deterministic pagination, and telemetry suitable for production monitoring.
|
||||
|
||||
## Configuration
|
||||
|
||||
Key options exposed through `concelier:sources:ru-nkcki:http`:
|
||||
|
||||
- `maxBulletinsPerFetch` – limits new bulletin downloads in a single run (default `5`).
|
||||
- `maxListingPagesPerFetch` – maximum listing pages visited during pagination (default `3`).
|
||||
- `listingCacheDuration` – minimum interval between listing fetches before falling back to cached artefacts (default `00:10:00`).
|
||||
- `cacheDirectory` – optional path for persisted bulletin archives used during offline or failure scenarios.
|
||||
- `requestDelay` – delay inserted between bulletin downloads to respect upstream politeness.
|
||||
|
||||
When operating in offline-first mode, set `cacheDirectory` to a writable path (e.g. `/var/lib/concelier/cache/ru-nkcki`) and pre-populate bulletin archives via the offline kit.
|
||||
|
||||
## Telemetry
|
||||
|
||||
`RuNkckiDiagnostics` emits the following metrics under meter `StellaOps.Concelier.Connector.Ru.Nkcki`:
|
||||
|
||||
- `nkcki.listing.fetch.attempts` / `nkcki.listing.fetch.success` / `nkcki.listing.fetch.failures`
|
||||
- `nkcki.listing.pages.visited` (histogram, `pages`)
|
||||
- `nkcki.listing.attachments.discovered` / `nkcki.listing.attachments.new`
|
||||
- `nkcki.bulletin.fetch.success` / `nkcki.bulletin.fetch.cached` / `nkcki.bulletin.fetch.failures`
|
||||
- `nkcki.entries.processed` (histogram, `entries`)
|
||||
|
||||
Integrate these counters into standard Concelier observability dashboards to track crawl coverage and cache hit rates.
|
||||
|
||||
## Archive Backfill Strategy
|
||||
|
||||
Bitrix pagination surfaces archives via `?PAGEN_1=n`. The connector now walks up to `maxListingPagesPerFetch` pages, deduplicating bulletin IDs and maintaining a rolling `knownBulletins` window. Backfill strategy:
|
||||
|
||||
1. Enumerate pages from newest to oldest, respecting `maxListingPagesPerFetch` and `listingCacheDuration` to avoid refetch storms.
|
||||
2. Persist every `*.json.zip` attachment to the configured cache directory. This enables replay when listing access is temporarily blocked.
|
||||
3. During archive replay, `ProcessCachedBulletinsAsync` enqueues missing documents while respecting `maxVulnerabilitiesPerFetch`.
|
||||
4. For historical HTML-only advisories, collect page URLs and metadata while offline (future work: HTML and PDF extraction pipeline documented in `docs/concelier-connector-research-20251011.md`).
|
||||
|
||||
For large migrations, seed caches with archived zip bundles, then run fetch/parse/map cycles in chronological order to maintain deterministic outputs.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
- Listing failures mark the source state with exponential backoff while attempting cache replay.
|
||||
- Bulletin fetches fall back to cached copies before surfacing an error.
|
||||
- Mongo integration tests rely on bundled OpenSSL 1.1 libraries (`src/Tools/openssl/linux-x64`) to keep `Mongo2Go` operational on modern distros.
|
||||
|
||||
Refer to `ru-nkcki` entries in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md` for outstanding items.
|
||||
# NKCKI Connector Operations Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The NKCKI connector ingests JSON bulletin archives from cert.gov.ru, expanding each `*.json.zip` attachment into per-vulnerability DTOs before canonical mapping. The fetch pipeline now supports cache-backed recovery, deterministic pagination, and telemetry suitable for production monitoring.
|
||||
|
||||
## Configuration
|
||||
|
||||
Key options exposed through `concelier:sources:ru-nkcki:http`:
|
||||
|
||||
- `maxBulletinsPerFetch` – limits new bulletin downloads in a single run (default `5`).
|
||||
- `maxListingPagesPerFetch` – maximum listing pages visited during pagination (default `3`).
|
||||
- `listingCacheDuration` – minimum interval between listing fetches before falling back to cached artefacts (default `00:10:00`).
|
||||
- `cacheDirectory` – optional path for persisted bulletin archives used during offline or failure scenarios.
|
||||
- `requestDelay` – delay inserted between bulletin downloads to respect upstream politeness.
|
||||
|
||||
When operating in offline-first mode, set `cacheDirectory` to a writable path (e.g. `/var/lib/concelier/cache/ru-nkcki`) and pre-populate bulletin archives via the offline kit.
|
||||
|
||||
## Telemetry
|
||||
|
||||
`RuNkckiDiagnostics` emits the following metrics under meter `StellaOps.Concelier.Connector.Ru.Nkcki`:
|
||||
|
||||
- `nkcki.listing.fetch.attempts` / `nkcki.listing.fetch.success` / `nkcki.listing.fetch.failures`
|
||||
- `nkcki.listing.pages.visited` (histogram, `pages`)
|
||||
- `nkcki.listing.attachments.discovered` / `nkcki.listing.attachments.new`
|
||||
- `nkcki.bulletin.fetch.success` / `nkcki.bulletin.fetch.cached` / `nkcki.bulletin.fetch.failures`
|
||||
- `nkcki.entries.processed` (histogram, `entries`)
|
||||
|
||||
Integrate these counters into standard Concelier observability dashboards to track crawl coverage and cache hit rates.
|
||||
|
||||
## Archive Backfill Strategy
|
||||
|
||||
Bitrix pagination surfaces archives via `?PAGEN_1=n`. The connector now walks up to `maxListingPagesPerFetch` pages, deduplicating bulletin IDs and maintaining a rolling `knownBulletins` window. Backfill strategy:
|
||||
|
||||
1. Enumerate pages from newest to oldest, respecting `maxListingPagesPerFetch` and `listingCacheDuration` to avoid refetch storms.
|
||||
2. Persist every `*.json.zip` attachment to the configured cache directory. This enables replay when listing access is temporarily blocked.
|
||||
3. During archive replay, `ProcessCachedBulletinsAsync` enqueues missing documents while respecting `maxVulnerabilitiesPerFetch`.
|
||||
4. For historical HTML-only advisories, collect page URLs and metadata while offline (future work: HTML and PDF extraction pipeline documented in `docs/concelier-connector-research-20251011.md`).
|
||||
|
||||
For large migrations, seed caches with archived zip bundles, then run fetch/parse/map cycles in chronological order to maintain deterministic outputs.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
- Listing failures mark the source state with exponential backoff while attempting cache replay.
|
||||
- Bulletin fetches fall back to cached copies before surfacing an error.
|
||||
- Mongo integration tests rely on bundled OpenSSL 1.1 libraries (`src/Tools/openssl/linux-x64`) to keep `Mongo2Go` operational on modern distros.
|
||||
|
||||
Refer to `ru-nkcki` entries in `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md` for outstanding items.
|
||||
|
||||
@@ -1,24 +1,24 @@
|
||||
# Concelier OSV Connector – Operations Notes
|
||||
|
||||
_Last updated: 2025-10-16_
|
||||
|
||||
The OSV connector ingests advisories from OSV.dev across OSS ecosystems. This note highlights the additional merge/export expectations introduced with the canonical metric fallback work in Sprint 4.
|
||||
|
||||
## 1. Canonical metric fallbacks
|
||||
- When OSV omits CVSS vectors (common for CVSS v4-only payloads) the mapper now emits a deterministic canonical metric id in the form `osv:severity/<level>` and normalises the advisory severity to the same `<level>`.
|
||||
- Metric: `osv.map.canonical_metric_fallbacks` (counter) with tags `severity`, `canonical_metric_id`, `ecosystem`, `reason=no_cvss`. Watch this alongside merge parity dashboards to catch spikes where OSV publishes severity-only advisories.
|
||||
- Merge precedence still prefers GHSA over OSV; the shared severity-based canonical id keeps Merge/export parity deterministic even when only OSV supplies severity data.
|
||||
|
||||
## 2. CWE provenance
|
||||
- `database_specific.cwe_ids` now populates provenance decision reasons for every mapped weakness. Expect `decisionReason="database_specific.cwe_ids"` on OSV weakness provenance and confirm exporters preserve the value.
|
||||
- If OSV ever attaches `database_specific.cwe_notes`, the connector will surface the joined note string in `decisionReason` instead of the default marker.
|
||||
|
||||
## 3. Dashboards & alerts
|
||||
- Extend existing merge dashboards with the new counter:
|
||||
- Overlay `sum(osv.map.canonical_metric_fallbacks{ecosystem=~".+"})` with Merge severity overrides to confirm fallback advisories are reconciling cleanly.
|
||||
- Alert when the 1-hour sum exceeds 50 for any ecosystem; baseline volume is currently <5 per day (mostly GHSA mirrors emitting CVSS v4 only).
|
||||
- Exporters already surface `canonicalMetricId`; no schema change is required, but ORAS/Trivy bundles should be spot-checked after deploying the connector update.
|
||||
|
||||
## 4. Runbook updates
|
||||
- Fixture parity suites (`osv-ghsa.*`) now assert the fallback id and provenance notes. Regenerate via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
- When investigating merge severity conflicts, include the fallback counter and confirm OSV advisories carry the expected `osv:severity/<level>` id before raising connector bugs.
|
||||
# Concelier OSV Connector – Operations Notes
|
||||
|
||||
_Last updated: 2025-10-16_
|
||||
|
||||
The OSV connector ingests advisories from OSV.dev across OSS ecosystems. This note highlights the additional merge/export expectations introduced with the canonical metric fallback work in Sprint 4.
|
||||
|
||||
## 1. Canonical metric fallbacks
|
||||
- When OSV omits CVSS vectors (common for CVSS v4-only payloads) the mapper now emits a deterministic canonical metric id in the form `osv:severity/<level>` and normalises the advisory severity to the same `<level>`.
|
||||
- Metric: `osv.map.canonical_metric_fallbacks` (counter) with tags `severity`, `canonical_metric_id`, `ecosystem`, `reason=no_cvss`. Watch this alongside merge parity dashboards to catch spikes where OSV publishes severity-only advisories.
|
||||
- Merge precedence still prefers GHSA over OSV; the shared severity-based canonical id keeps Merge/export parity deterministic even when only OSV supplies severity data.
|
||||
|
||||
## 2. CWE provenance
|
||||
- `database_specific.cwe_ids` now populates provenance decision reasons for every mapped weakness. Expect `decisionReason="database_specific.cwe_ids"` on OSV weakness provenance and confirm exporters preserve the value.
|
||||
- If OSV ever attaches `database_specific.cwe_notes`, the connector will surface the joined note string in `decisionReason` instead of the default marker.
|
||||
|
||||
## 3. Dashboards & alerts
|
||||
- Extend existing merge dashboards with the new counter:
|
||||
- Overlay `sum(osv.map.canonical_metric_fallbacks{ecosystem=~".+"})` with Merge severity overrides to confirm fallback advisories are reconciling cleanly.
|
||||
- Alert when the 1-hour sum exceeds 50 for any ecosystem; baseline volume is currently <5 per day (mostly GHSA mirrors emitting CVSS v4 only).
|
||||
- Exporters already surface `canonicalMetricId`; no schema change is required, but ORAS/Trivy bundles should be spot-checked after deploying the connector update.
|
||||
|
||||
## 4. Runbook updates
|
||||
- Fixture parity suites (`osv-ghsa.*`) now assert the fallback id and provenance notes. Regenerate via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj`.
|
||||
- When investigating merge severity conflicts, include the fallback counter and confirm OSV advisories carry the expected `osv:severity/<level>` id before raising connector bugs.
|
||||
|
||||
@@ -1,238 +1,238 @@
|
||||
# Concelier & Excititor Mirror Operations
|
||||
|
||||
This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
|
||||
It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
|
||||
authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- **Authority access** – client credentials (`client_id` + secret) authorised for
|
||||
`concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
|
||||
- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
|
||||
Store them under `deploy/compose/mirror-gateway/tls/` or in Kubernetes secrets.
|
||||
- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
|
||||
`htpasswd -B`. Operators distribute credentials to downstream consumers.
|
||||
- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
|
||||
that hold `concelier` JSON bundles and `excititor` VEX exports.
|
||||
- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
|
||||
For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
|
||||
`excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
|
||||
|
||||
### 1.1 Service configuration quick reference
|
||||
|
||||
Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
|
||||
Key knobs:
|
||||
|
||||
- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
|
||||
- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
|
||||
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
|
||||
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
|
||||
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
|
||||
|
||||
> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
|
||||
|
||||
Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
|
||||
|
||||
### 1.2 Mirror connector configuration
|
||||
|
||||
Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):
|
||||
|
||||
- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
|
||||
- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
|
||||
- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
|
||||
- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
|
||||
- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
|
||||
- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
|
||||
- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.
|
||||
|
||||
The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.
|
||||
|
||||
## 2. Secret & certificate layout
|
||||
|
||||
### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`)
|
||||
|
||||
- `deploy/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
|
||||
- `deploy/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
|
||||
- `concelier-authority-client` – Authority client secret.
|
||||
- `excititor-authority-client` (optional) – reserve for future authn.
|
||||
- `deploy/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
|
||||
- `mirror-primary.crt`, `mirror-primary.key`
|
||||
- `mirror-community.crt`, `mirror-community.key`
|
||||
- `deploy/compose/mirror-gateway/secrets/` – htpasswd files:
|
||||
- `mirror-primary.htpasswd`
|
||||
- `mirror-community.htpasswd`
|
||||
|
||||
### Helm (`deploy/helm/stellaops/values-mirror.yaml`)
|
||||
|
||||
Create secrets in the target namespace:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic concelier-mirror-auth \
|
||||
--from-file=concelier-authority-client=concelier-authority-client
|
||||
|
||||
kubectl create secret generic excititor-mirror-auth \
|
||||
--from-file=excititor-authority-client=excititor-authority-client
|
||||
|
||||
kubectl create secret tls mirror-gateway-tls \
|
||||
--cert=mirror-primary.crt --key=mirror-primary.key
|
||||
|
||||
kubectl create secret generic mirror-gateway-htpasswd \
|
||||
--from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
|
||||
```
|
||||
|
||||
> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.
|
||||
|
||||
## 3. Deployment
|
||||
|
||||
### 3.1 Docker Compose (edge mirrors, lab validation)
|
||||
|
||||
1. `cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env`
|
||||
2. Populate secrets/tls directories as described above.
|
||||
3. Sync mirror bundles (see §4) into `deploy/compose/mirror-data/…` and ensure they are mounted
|
||||
on the host path backing the `concelier-exports` and `excititor-exports` volumes.
|
||||
4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
|
||||
5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.
|
||||
|
||||
### 3.2 Helm (production mirrors)
|
||||
|
||||
1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
|
||||
2. Create secrets/tls config maps (§2).
|
||||
3. `helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml`.
|
||||
4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
|
||||
your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).
|
||||
|
||||
## 4. Artifact sync workflow
|
||||
|
||||
Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
|
||||
export jobs. Recommended sync pattern:
|
||||
|
||||
### 4.1 Compose host (systemd timer)
|
||||
|
||||
`/usr/local/bin/mirror-sync.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export AWS_ACCESS_KEY_ID=…
|
||||
export AWS_SECRET_ACCESS_KEY=…
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest \
|
||||
/opt/stellaops/mirror-data/concelier --delete --size-only
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest \
|
||||
/opt/stellaops/mirror-data/excititor --delete --size-only
|
||||
```
|
||||
|
||||
Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
|
||||
into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
|
||||
`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.
|
||||
|
||||
### 4.2 Kubernetes (CronJob)
|
||||
|
||||
Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: mirror-sync
|
||||
spec:
|
||||
schedule: "*/5 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: sync
|
||||
image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
|
||||
volumeMounts:
|
||||
- name: concelier-exports
|
||||
mountPath: /exports/concelier
|
||||
- name: excititor-exports
|
||||
mountPath: /exports/excititor
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: mirror-sync-aws
|
||||
restartPolicy: OnFailure
|
||||
volumes:
|
||||
- name: concelier-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: concelier-mirror-exports
|
||||
- name: excititor-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: excititor-mirror-exports
|
||||
```
|
||||
|
||||
## 5. CDN integration
|
||||
|
||||
1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
|
||||
2. Honour the response headers emitted by the gateway and Concelier/Excititor:
|
||||
`Cache-Control: public, max-age=300, immutable` for mirror payloads.
|
||||
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
|
||||
- Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
|
||||
- Bundle/manifest payloads → 300 s.
|
||||
4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
|
||||
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
|
||||
to SIEM for anomaly detection.
|
||||
|
||||
## 6. Smoke tests
|
||||
|
||||
After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
|
||||
|
||||
```bash
|
||||
# Index with Basic Auth
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
|
||||
|
||||
# Mirror manifest signature and cache headers
|
||||
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
|
||||
| tee /tmp/manifest-headers.txt
|
||||
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt # expect public, max-age=300, immutable
|
||||
|
||||
# Excititor consensus bundle metadata
|
||||
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
|
||||
| jq '.exports[].exportKey'
|
||||
|
||||
# Signed bundle + detached JWS (spot check digests)
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
|
||||
-o bundle.json.jws
|
||||
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
|
||||
|
||||
# Service-level auth check (inside cluster – no gateway credentials)
|
||||
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
|
||||
| head -n 5 # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
|
||||
|
||||
# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
|
||||
for i in 1 2; do
|
||||
curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
|
||||
-u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
|
||||
sleep 1
|
||||
done
|
||||
```
|
||||
|
||||
Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
|
||||
should show `X-Cache-Status: HIT/MISS`.
|
||||
|
||||
## 7. Maintenance & rotation
|
||||
|
||||
- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
|
||||
`Mirror export root is not configured`.
|
||||
- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
|
||||
Update the mounted secrets and restart deployments (`docker compose restart concelier` or
|
||||
`kubectl rollout restart deploy/stellaops-concelier`).
|
||||
- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
|
||||
- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
|
||||
Align CDN rate limits and inform downstreams.
|
||||
|
||||
## 8. References
|
||||
|
||||
- Deployment profiles: `deploy/compose/docker-compose.mirror.yaml`,
|
||||
`deploy/helm/stellaops/values-mirror.yaml`
|
||||
- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
|
||||
`docs/modules/excititor/mirrors.md`
|
||||
- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7
|
||||
# Concelier & Excititor Mirror Operations
|
||||
|
||||
This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`.
|
||||
It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant
|
||||
authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- **Authority access** – client credentials (`client_id` + secret) authorised for
|
||||
`concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git.
|
||||
- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`).
|
||||
Store them under `deploy/compose/mirror-gateway/tls/` or in Kubernetes secrets.
|
||||
- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with
|
||||
`htpasswd -B`. Operators distribute credentials to downstream consumers.
|
||||
- **Export artifact source** – read access to the canonical S3 buckets (or rsync share)
|
||||
that hold `concelier` JSON bundles and `excititor` VEX exports.
|
||||
- **Persistent volumes** – storage for Concelier job metadata and mirror export trees.
|
||||
For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`,
|
||||
`excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout.
|
||||
|
||||
### 1.1 Service configuration quick reference
|
||||
|
||||
Concelier.WebService exposes the mirror HTTP endpoints once `CONCELIER__MIRROR__ENABLED=true`.
|
||||
Key knobs:
|
||||
|
||||
- `CONCELIER__MIRROR__EXPORTROOT` – root folder containing export snapshots (`<exportId>/mirror/*`).
|
||||
- `CONCELIER__MIRROR__ACTIVEEXPORTID` – optional explicit export id; otherwise the service auto-falls back to the `latest/` symlink or newest directory.
|
||||
- `CONCELIER__MIRROR__REQUIREAUTHENTICATION` – default auth requirement; override per domain with `CONCELIER__MIRROR__DOMAINS__{n}__REQUIREAUTHENTICATION`.
|
||||
- `CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR` – budget for `/concelier/exports/index.json`. Domains inherit this value unless they define `__MAXDOWNLOADREQUESTSPERHOUR`.
|
||||
- `CONCELIER__MIRROR__DOMAINS__{n}__ID` – domain identifier matching the exporter manifest; additional keys configure display name and rate budgets.
|
||||
|
||||
> The service honours Stella Ops Authority when `CONCELIER__AUTHORITY__ENABLED=true` and `ALLOWANONYMOUSFALLBACK=false`. Use the bypass CIDR list (`CONCELIER__AUTHORITY__BYPASSNETWORKS__*`) for in-cluster ingress gateways that terminate Basic Auth. Unauthorized requests emit `WWW-Authenticate: Bearer` so downstream automation can detect token failures.
|
||||
|
||||
Mirror responses carry deterministic cache headers: `/index.json` returns `Cache-Control: public, max-age=60`, while per-domain manifests/bundles include `Cache-Control: public, max-age=300, immutable`. Rate limiting surfaces `Retry-After` when quotas are exceeded.
|
||||
|
||||
### 1.2 Mirror connector configuration
|
||||
|
||||
Downstream Concelier instances ingest published bundles using the `StellaOpsMirrorConnector`. Operators running the connector in air‑gapped or limited connectivity environments can tune the following options (environment prefix `CONCELIER__SOURCES__STELLAOPSMIRROR__`):
|
||||
|
||||
- `BASEADDRESS` – absolute mirror root (e.g., `https://mirror-primary.stella-ops.org`).
|
||||
- `INDEXPATH` – relative path to the mirror index (`/concelier/exports/index.json` by default).
|
||||
- `DOMAINID` – mirror domain identifier from the index (`primary`, `community`, etc.).
|
||||
- `HTTPTIMEOUT` – request timeout; raise when mirrors sit behind slow WAN links.
|
||||
- `SIGNATURE__ENABLED` – require detached JWS verification for `bundle.json`.
|
||||
- `SIGNATURE__KEYID` / `SIGNATURE__PROVIDER` – expected signing key metadata.
|
||||
- `SIGNATURE__PUBLICKEYPATH` – PEM fallback used when the mirror key registry is offline.
|
||||
|
||||
The connector keeps a per-export fingerprint (bundle digest + generated-at timestamp) and tracks outstanding document IDs. If a scan is interrupted, the next run resumes parse/map work using the stored fingerprint and pending document lists—no network requests are reissued unless the upstream digest changes.
|
||||
|
||||
## 2. Secret & certificate layout
|
||||
|
||||
### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`)
|
||||
|
||||
- `deploy/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs.
|
||||
- `deploy/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place:
|
||||
- `concelier-authority-client` – Authority client secret.
|
||||
- `excititor-authority-client` (optional) – reserve for future authn.
|
||||
- `deploy/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs:
|
||||
- `mirror-primary.crt`, `mirror-primary.key`
|
||||
- `mirror-community.crt`, `mirror-community.key`
|
||||
- `deploy/compose/mirror-gateway/secrets/` – htpasswd files:
|
||||
- `mirror-primary.htpasswd`
|
||||
- `mirror-community.htpasswd`
|
||||
|
||||
### Helm (`deploy/helm/stellaops/values-mirror.yaml`)
|
||||
|
||||
Create secrets in the target namespace:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic concelier-mirror-auth \
|
||||
--from-file=concelier-authority-client=concelier-authority-client
|
||||
|
||||
kubectl create secret generic excititor-mirror-auth \
|
||||
--from-file=excititor-authority-client=excititor-authority-client
|
||||
|
||||
kubectl create secret tls mirror-gateway-tls \
|
||||
--cert=mirror-primary.crt --key=mirror-primary.key
|
||||
|
||||
kubectl create secret generic mirror-gateway-htpasswd \
|
||||
--from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd
|
||||
```
|
||||
|
||||
> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients.
|
||||
|
||||
## 3. Deployment
|
||||
|
||||
### 3.1 Docker Compose (edge mirrors, lab validation)
|
||||
|
||||
1. `cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env`
|
||||
2. Populate secrets/tls directories as described above.
|
||||
3. Sync mirror bundles (see §4) into `deploy/compose/mirror-data/…` and ensure they are mounted
|
||||
on the host path backing the `concelier-exports` and `excititor-exports` volumes.
|
||||
4. Run the profile validator: `deploy/tools/validate-profiles.sh`.
|
||||
5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`.
|
||||
|
||||
### 3.2 Helm (production mirrors)
|
||||
|
||||
1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain).
|
||||
2. Create secrets/tls config maps (§2).
|
||||
3. `helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml`.
|
||||
4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by
|
||||
your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout).
|
||||
|
||||
## 4. Artifact sync workflow
|
||||
|
||||
Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor
|
||||
export jobs. Recommended sync pattern:
|
||||
|
||||
### 4.1 Compose host (systemd timer)
|
||||
|
||||
`/usr/local/bin/mirror-sync.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export AWS_ACCESS_KEY_ID=…
|
||||
export AWS_SECRET_ACCESS_KEY=…
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest \
|
||||
/opt/stellaops/mirror-data/concelier --delete --size-only
|
||||
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest \
|
||||
/opt/stellaops/mirror-data/excititor --delete --size-only
|
||||
```
|
||||
|
||||
Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*`
|
||||
into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and
|
||||
`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`.
|
||||
|
||||
### 4.2 Kubernetes (CronJob)
|
||||
|
||||
Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs:
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: mirror-sync
|
||||
spec:
|
||||
schedule: "*/5 * * * *"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: sync
|
||||
image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only &&
|
||||
aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only
|
||||
volumeMounts:
|
||||
- name: concelier-exports
|
||||
mountPath: /exports/concelier
|
||||
- name: excititor-exports
|
||||
mountPath: /exports/excititor
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: mirror-sync-aws
|
||||
restartPolicy: OnFailure
|
||||
volumes:
|
||||
- name: concelier-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: concelier-mirror-exports
|
||||
- name: excititor-exports
|
||||
persistentVolumeClaim:
|
||||
claimName: excititor-mirror-exports
|
||||
```
|
||||
|
||||
## 5. CDN integration
|
||||
|
||||
1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer).
|
||||
2. Honour the response headers emitted by the gateway and Concelier/Excititor:
|
||||
`Cache-Control: public, max-age=300, immutable` for mirror payloads.
|
||||
3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs:
|
||||
- Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s.
|
||||
- Bundle/manifest payloads → 300 s.
|
||||
4. Forward the `Authorization` header—Basic Auth terminates at the gateway.
|
||||
5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging
|
||||
to SIEM for anomaly detection.
|
||||
|
||||
## 6. Smoke tests
|
||||
|
||||
After each deployment or sync cycle (temporarily set low budgets if you need to observe 429 responses):
|
||||
|
||||
```bash
|
||||
# Index with Basic Auth
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys'
|
||||
|
||||
# Mirror manifest signature and cache headers
|
||||
curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json \
|
||||
| tee /tmp/manifest-headers.txt
|
||||
grep -E '^Cache-Control: ' /tmp/manifest-headers.txt # expect public, max-age=300, immutable
|
||||
|
||||
# Excititor consensus bundle metadata
|
||||
curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \
|
||||
| jq '.exports[].exportKey'
|
||||
|
||||
# Signed bundle + detached JWS (spot check digests)
|
||||
curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \
|
||||
-o bundle.json.jws
|
||||
cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json
|
||||
|
||||
# Service-level auth check (inside cluster – no gateway credentials)
|
||||
kubectl exec deploy/stellaops-concelier -- curl -si http://localhost:8443/concelier/exports/mirror/primary/manifest.json \
|
||||
| head -n 5 # expect HTTP/1.1 401 with WWW-Authenticate: Bearer
|
||||
|
||||
# Rate limit smoke (repeat quickly; second call should return 429 + Retry-After)
|
||||
for i in 1 2; do
|
||||
curl -s -o /dev/null -D - https://mirror-primary.stella-ops.org/concelier/exports/index.json \
|
||||
-u $PRIMARY_CREDS | grep -E '^(HTTP/|Retry-After:)'
|
||||
sleep 1
|
||||
done
|
||||
```
|
||||
|
||||
Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway`
|
||||
should show `X-Cache-Status: HIT/MISS`.
|
||||
|
||||
## 7. Maintenance & rotation
|
||||
|
||||
- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs
|
||||
`Mirror export root is not configured`.
|
||||
- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly.
|
||||
Update the mounted secrets and restart deployments (`docker compose restart concelier` or
|
||||
`kubectl rollout restart deploy/stellaops-concelier`).
|
||||
- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`).
|
||||
- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file.
|
||||
Align CDN rate limits and inform downstreams.
|
||||
|
||||
## 8. References
|
||||
|
||||
- Deployment profiles: `deploy/compose/docker-compose.mirror.yaml`,
|
||||
`deploy/helm/stellaops/values-mirror.yaml`
|
||||
- Mirror architecture dossiers: `docs/modules/concelier/architecture.md`,
|
||||
`docs/modules/excititor/mirrors.md`
|
||||
- Export bundling: `docs/modules/devops/architecture.md` §3, `docs/modules/excititor/architecture.md` §7
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# DevOps agent guide
|
||||
|
||||
## Mission
|
||||
The DevOps module captures release, deployment, and migration playbooks that keep StellaOps deterministic across environments.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# DevOps agent guide
|
||||
|
||||
## Mission
|
||||
The DevOps module captures release, deployment, and migration playbooks that keep StellaOps deterministic across environments.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,41 +1,41 @@
|
||||
# StellaOps DevOps
|
||||
|
||||
The DevOps module captures release, deployment, and migration playbooks that keep StellaOps deterministic across environments.
|
||||
|
||||
## Responsibilities
|
||||
- Maintain CI pipelines, signing workflows, and release packaging steps.
|
||||
- Operate shared runbooks for launch readiness, upgrades, and NuGet previews.
|
||||
- Provide offline kit assembly instructions and tooling integration.
|
||||
- Wrap observability/telemetry bootstrap flows for platform teams.
|
||||
|
||||
## Key components
|
||||
- Runbooks under ./runbooks/ (launch, deployment, nuget).
|
||||
- Migration guidance under ./migrations/.
|
||||
- Architecture overview bridging CI/CD & infrastructure concerns.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Ops pipelines (Gitea, GitHub Actions) and artifact registries.
|
||||
- Authority/Signer for supply chain signing.
|
||||
- Telemetry stack bootstrap scripts.
|
||||
|
||||
## Operational notes
|
||||
- Offline bundle packaging guidance in docs/modules/export-center/operations/runbook.md.
|
||||
- Dashboards for launch cutover rehearsals.
|
||||
- Coordination with Security for enforced guardrails.
|
||||
|
||||
## Related resources
|
||||
- ./runbooks/launch-readiness.md
|
||||
- ./runbooks/launch-cutover.md
|
||||
- ./runbooks/deployment-upgrade.md
|
||||
- ./runbooks/nuget-preview-bootstrap.md
|
||||
- ./migrations/semver-style.md
|
||||
|
||||
## Backlog references
|
||||
- DEVOPS-LAUNCH-18-001 / 18-900 runbooks in ../../TASKS.md.
|
||||
- Telemetry bootstrap automation tracked in `ops/devops/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** bake AOC verifier steps, CI guards, and schema validation into pipelines.
|
||||
- **Epic 9 – Orchestrator Dashboard:** support operational dashboards, job recovery runbooks, and rate-limit governance.
|
||||
- **Epic 10 – Export Center:** manage signing workflows, Offline Kit packaging, and release promotion for exports.
|
||||
- **Epic 15 – Observability & Forensics:** coordinate telemetry deployment, evidence retention, and forensic automation.
|
||||
# StellaOps DevOps
|
||||
|
||||
The DevOps module captures release, deployment, and migration playbooks that keep StellaOps deterministic across environments.
|
||||
|
||||
## Responsibilities
|
||||
- Maintain CI pipelines, signing workflows, and release packaging steps.
|
||||
- Operate shared runbooks for launch readiness, upgrades, and NuGet previews.
|
||||
- Provide offline kit assembly instructions and tooling integration.
|
||||
- Wrap observability/telemetry bootstrap flows for platform teams.
|
||||
|
||||
## Key components
|
||||
- Runbooks under ./runbooks/ (launch, deployment, nuget).
|
||||
- Migration guidance under ./migrations/.
|
||||
- Architecture overview bridging CI/CD & infrastructure concerns.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Ops pipelines (Gitea, GitHub Actions) and artifact registries.
|
||||
- Authority/Signer for supply chain signing.
|
||||
- Telemetry stack bootstrap scripts.
|
||||
|
||||
## Operational notes
|
||||
- Offline bundle packaging guidance in docs/modules/export-center/operations/runbook.md.
|
||||
- Dashboards for launch cutover rehearsals.
|
||||
- Coordination with Security for enforced guardrails.
|
||||
|
||||
## Related resources
|
||||
- ./runbooks/launch-readiness.md
|
||||
- ./runbooks/launch-cutover.md
|
||||
- ./runbooks/deployment-upgrade.md
|
||||
- ./runbooks/nuget-preview-bootstrap.md
|
||||
- ./migrations/semver-style.md
|
||||
|
||||
## Backlog references
|
||||
- DEVOPS-LAUNCH-18-001 / 18-900 runbooks in ../../TASKS.md.
|
||||
- Telemetry bootstrap automation tracked in `ops/devops/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** bake AOC verifier steps, CI guards, and schema validation into pipelines.
|
||||
- **Epic 9 – Orchestrator Dashboard:** support operational dashboards, job recovery runbooks, and rate-limit governance.
|
||||
- **Epic 10 – Export Center:** manage signing workflows, Offline Kit packaging, and release promotion for exports.
|
||||
- **Epic 15 – Observability & Forensics:** coordinate telemetry deployment, evidence retention, and forensic automation.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — DevOps
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| DEVOPS-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| DEVOPS-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| DEVOPS-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — DevOps
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| DEVOPS-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| DEVOPS-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| DEVOPS-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
@@ -1,488 +1,488 @@
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
## 0) Product vision (operations lens)
|
||||
|
||||
Stella Ops must be **trustable at a glance** and **boringly operable**:
|
||||
|
||||
* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime.
|
||||
* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels.
|
||||
* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**.
|
||||
* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import.
|
||||
* Artifacts expire predictably; operators know what’s kept, for how long, and why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Release trains & versioning
|
||||
|
||||
### 1.1 Channels
|
||||
|
||||
* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4).
|
||||
* **Stable** (default): monthly rollup (bug fixes + compatible features).
|
||||
* **Edge**: weekly; for early adopters, no guarantees.
|
||||
|
||||
### 1.2 Version strings
|
||||
|
||||
Semantic core + calendar tag:
|
||||
|
||||
```
|
||||
<MAJOR>.<MINOR>.<PATCH> (<YYYY>.<MM>) e.g., 2.4.1 (2027.06)
|
||||
```
|
||||
|
||||
* **MAJOR**: breaking API/DB changes (rare).
|
||||
* **MINOR**: new features, compatible schema migrations (expand/contract pattern).
|
||||
* **PATCH**: bug fixes, perf and security updates.
|
||||
* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**.
|
||||
|
||||
### 1.3 Component alignment
|
||||
|
||||
A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew:
|
||||
|
||||
* **Web UI ↔ backend**: `±1 minor`.
|
||||
* **Scanner ↔ Policy/Excititor/Concelier**: `±1 minor`.
|
||||
* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules).
|
||||
|
||||
At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**.
|
||||
|
||||
---
|
||||
|
||||
## 2) Supply‑chain pipeline (how a release is built)
|
||||
|
||||
### 2.1 Deterministic builds
|
||||
|
||||
* **Builders**: isolated **BuildKit** workers with pinned base images (digest only).
|
||||
* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag.
|
||||
* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars.
|
||||
* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap).
|
||||
|
||||
### 2.2 First‑party SBOMs & provenance
|
||||
|
||||
* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**.
|
||||
* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift).
|
||||
* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs.
|
||||
|
||||
### 2.3 Signing & transparency
|
||||
|
||||
* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required.
|
||||
* SBOM and provenance attestations are **DSSE** and also transparency‑logged.
|
||||
* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side).
|
||||
|
||||
### 2.4 Gates & tests
|
||||
|
||||
* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests).
|
||||
* **Unit/integration**: per‑component, plus **end‑to‑end** flows (scan→vex→policy→sign→attest).
|
||||
* **Perf SLOs**: hot paths (SBOM compose, diff, export) measured against budgets.
|
||||
* **Security**: dependency audit vs Concelier export; container hardening tests; minimal caps.
|
||||
* **Analyzer smoke**: restart-time language plug-ins (currently Python) verified via `dotnet run --project src/Tools/LanguageAnalyzerSmoke` to ensure manifest integrity plus cold vs warm determinism (< 30 s / < 5 s budgets); the harness logs deviations from repository goldens for follow-up.
|
||||
* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag.
|
||||
|
||||
### 2.5 Debug-store artefacts
|
||||
|
||||
* Every release exports stripped debug information for ELF binaries discovered in service images. Debug files follow the GNU build-id layout (`debug/.build-id/<aa>/<rest>.debug`) and are generated via `objcopy --only-keep-debug`.
|
||||
* `debug/debug-manifest.json` captures build-id → component/image/source mappings with SHA-256 checksums so operators can mirror the directory into debuginfod or offline symbol stores. The manifest (and its `.sha256` companion) ships with every release bundle and Offline Kit.
|
||||
|
||||
---
|
||||
|
||||
## 3) Distribution & activation
|
||||
|
||||
### 3.1 Registries
|
||||
|
||||
* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API).
|
||||
* **Mirrors**: GHCR (read‑only), regional mirrors for latency.
|
||||
* Operational runbook: see `docs/modules/concelier/operations/mirror.md` for deployment profiles, CDN guidance, and sync automation.
|
||||
* **Pull by digest only** in Kubernetes/Compose manifests.
|
||||
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Concelier, Excititor, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/Registry/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
### 3.2 OAuth2 token service (for private repos)
|
||||
|
||||
* Docker Registry’s token flow backed by **Authority**:
|
||||
|
||||
1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`).
|
||||
2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`.
|
||||
3. Registry allows pull for the requested repo.
|
||||
* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**.
|
||||
|
||||
The token service enforces plan gating via `registry-token.yaml` (see `docs/modules/registry/operations/token-service.md`) and exposes Prometheus metrics (`registry_token_issued_total`, `registry_token_rejected_total`). Revoked licence identifiers halt issuance even when scope requirements are met.
|
||||
|
||||
### 3.3 Offline kits (air‑gapped)
|
||||
|
||||
* Tarball per release channel:
|
||||
|
||||
```
|
||||
stellaops-kit-<ver>-<channel>.tar.zst
|
||||
/images/ OCI layout with all first-party images (multi-arch)
|
||||
/sboms/ CycloneDX JSON+PB for each image
|
||||
/attest/ DSSE bundles + Rekor proofs
|
||||
/charts/ Helm charts + values templates
|
||||
/compose/ docker-compose.yml + .env template
|
||||
/plugins/ Concelier/Excititor connectors (restart-time)
|
||||
/policy/ example policies
|
||||
/manifest/ release.yaml (see §6.1)
|
||||
```
|
||||
* Import via CLI `offline kit import`; checks digests and signatures before load.
|
||||
|
||||
---
|
||||
|
||||
## 4) Licensing (PoE) & monetization
|
||||
|
||||
**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE.
|
||||
|
||||
### 4.1 PoE issuance
|
||||
|
||||
* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`:
|
||||
|
||||
* **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**.
|
||||
* Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs.
|
||||
|
||||
### 4.2 Online enforcement
|
||||
|
||||
* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc).
|
||||
* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason.
|
||||
* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned.
|
||||
* UI badges: “**Verified by Stella Ops**” with link to the public log.
|
||||
|
||||
### 4.3 Air‑gapped / offline
|
||||
|
||||
* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days).
|
||||
* Signer accepts the lease and emits **provisional** attestations (clearly labeled).
|
||||
* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**.
|
||||
* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot).
|
||||
|
||||
### 4.4 Stolen/abused PoE
|
||||
|
||||
* Customers report theft; **Licensing** flags `license_id` as **revoked**.
|
||||
* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE).
|
||||
|
||||
---
|
||||
|
||||
## 5) Deployment path (customer side)
|
||||
|
||||
### 5.1 First install
|
||||
|
||||
* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s):
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://charts.stella-ops.org
|
||||
helm install stella stellaops/platform \
|
||||
--version 2.4.0 \
|
||||
--set global.channel=stable \
|
||||
--set authority.issuer=https://authority.stella.local \
|
||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||
--set scanner.mongo.uri=mongodb://mongo/scanner \
|
||||
--set concelier.mongo.uri=mongodb://mongo/concelier \
|
||||
--set excititor.mongo.uri=mongodb://mongo/excititor
|
||||
```
|
||||
|
||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||
* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?).
|
||||
|
||||
### 5.2 Updates
|
||||
|
||||
* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic.
|
||||
|
||||
* **Rolling**: upgrade stateful components in safe order:
|
||||
|
||||
1. Authority (stateless, dual‑key rotation ready)
|
||||
2. Signer/Attestor (same minor)
|
||||
3. Scanner WebService & Workers
|
||||
4. Concelier, then Excititor (schema migrations are expand/contract)
|
||||
5. UI last
|
||||
|
||||
* **DB migrations** are **expand/contract**:
|
||||
|
||||
* Phase A (release N): **add** new fields/indexes, write old+new.
|
||||
* Phase B (N+1): **read** new fields; **drop** old.
|
||||
* Rollback is a matter of redeploying previous images and keeping both schemas valid.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||
* Mongo migrations are additive; **no destructive changes** within a single minor.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release payloads & manifests
|
||||
|
||||
### 6.1 Release manifest (`release.yaml`)
|
||||
|
||||
```yaml
|
||||
release:
|
||||
version: "2.4.1"
|
||||
channel: "stable"
|
||||
date: "2027-06-20T12:00:00Z"
|
||||
calendar: "2027.06"
|
||||
components:
|
||||
- name: scanner-webservice
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb
|
||||
sbom: oci://.../referrers/cdx-json@sha256:11..22
|
||||
provenance: oci://.../attest/provenance@sha256:33..44
|
||||
signature: { rekorUUID: "…" }
|
||||
- name: signer
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd
|
||||
signature: { rekorUUID: "…" }
|
||||
charts:
|
||||
- name: platform
|
||||
version: "2.4.1"
|
||||
digest: "sha256:ee..ff"
|
||||
compose:
|
||||
file: "docker-compose.yml"
|
||||
digest: "sha256:77..88"
|
||||
checksums:
|
||||
sha256: "… digest of this release.yaml …"
|
||||
```
|
||||
|
||||
The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries.
|
||||
|
||||
> Deployment guardrails – The repository keeps channel-aligned Compose bundles
|
||||
> in `deploy/compose/` and Helm overlays in `deploy/helm/stellaops/`. Both sets
|
||||
> pull their digests from `deploy/releases/` and are validated by
|
||||
> `deploy/tools/validate-profiles.sh` to guarantee lint/dry-run cleanliness.
|
||||
|
||||
### 6.2 Image labels (release metadata)
|
||||
|
||||
Each image sets OCI labels:
|
||||
|
||||
```
|
||||
org.opencontainers.image.version = "2.4.1"
|
||||
org.opencontainers.image.revision = "<git sha>"
|
||||
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
|
||||
org.stellaops.release.calendar = "2027.06"
|
||||
org.stellaops.release.channel = "stable"
|
||||
org.stellaops.build.slsaProvenance = "oci://…"
|
||||
```
|
||||
|
||||
Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (RustFS/Mongo)
|
||||
|
||||
### 7.1 Buckets & prefixes (RustFS)
|
||||
|
||||
```
|
||||
rustfs://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json
|
||||
concelier/
|
||||
json/<exportId>/...
|
||||
trivy/<exportId>/...
|
||||
excititor/
|
||||
exports/<exportId>/...
|
||||
attestor/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
```
|
||||
|
||||
### 7.2 ILM classes
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**:
|
||||
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||
|
||||
### 7.4 Mongo retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||
|
||||
### 7.5 Mongo server baseline
|
||||
|
||||
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
||||
* **Deploy images:** Compose/Helm defaults stay on `mongo:7.x`. For air-gapped installs, refresh Offline Kit bundles so the packaged `mongod` matches ≥4.2.
|
||||
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & SLOs (operations)
|
||||
|
||||
* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Excititor/Concelier 99.0%.
|
||||
* **Error budgets**: tracked per month; dashboards show burn rates.
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
Prometheus + OTLP; Grafana dashboards ship in the charts.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security & compliance operations
|
||||
|
||||
* **Key rotation**:
|
||||
|
||||
* Authority JWKS: 60‑day cadence, dual‑key overlap.
|
||||
* Release signing identities: rotate per minor or quarterly.
|
||||
* Sigstore roots mirrored and pinned; alarms on drift.
|
||||
|
||||
* **FIPS mode** (Gov build):
|
||||
|
||||
* Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only.
|
||||
* Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA.
|
||||
|
||||
* **Vulnerability response**:
|
||||
|
||||
* Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
|
||||
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; repacked the local `Mongo2Go` feed so test fixtures inherit the patched dependencies; future bumps follow the same central override pattern.
|
||||
|
||||
* **Backups/DR**:
|
||||
|
||||
* Mongo nightly snapshots; MinIO versioning + replication (if configured).
|
||||
* Restore runbooks tested quarterly with synthetic data.
|
||||
|
||||
---
|
||||
|
||||
## 10) Customer update flow (how versions are fetched & activated)
|
||||
|
||||
### 10.1 Online clusters
|
||||
|
||||
* **UI** surfaces update banner with **release manifest** diff and risk notes.
|
||||
* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision.
|
||||
* Post‑switch, **schema Phase B** migrations (if any) run automatically.
|
||||
|
||||
### 10.2 Air‑gapped clusters
|
||||
|
||||
* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`.
|
||||
* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest.
|
||||
* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged.
|
||||
|
||||
### 10.3 CLI self‑update (optional)
|
||||
|
||||
* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable).
|
||||
|
||||
---
|
||||
|
||||
## 11) Compatibility & deprecation policy
|
||||
|
||||
* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor.
|
||||
* **Storage**: expand/contract; “drop old fields” only after one minor grace.
|
||||
* **Config**: feature flags (default off) for risky features (e.g., eBPF).
|
||||
|
||||
---
|
||||
|
||||
## 12) Runbooks (selected)
|
||||
|
||||
### 12.1 Lost PoE
|
||||
|
||||
1. Suspend **automatic attestation** jobs.
|
||||
2. Use CLI `stellaops signer status` to confirm `entitlement_denied`.
|
||||
3. Obtain new PoE from portal; verify on Signer `/poe/verify`.
|
||||
4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch).
|
||||
|
||||
### 12.2 Rekor outage (self‑hosted)
|
||||
|
||||
* Attestor returns `202 (pending)` with queued proof fetch.
|
||||
* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**.
|
||||
* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored.
|
||||
|
||||
### 12.3 Emergency downgrade
|
||||
|
||||
* Identify prior release manifest (UI → Admin → Releases).
|
||||
* `helm rollback stella <revision>` (or compose apply previous file).
|
||||
* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together.
|
||||
|
||||
---
|
||||
|
||||
## 13) Example: cluster bootstrap (Compose)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
authority:
|
||||
image: registry.stella-ops.org/stellaops/authority@sha256:...
|
||||
env_file: ./env/authority.env
|
||||
ports: ["8440:8440"]
|
||||
signer:
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:...
|
||||
depends_on: [authority]
|
||||
environment:
|
||||
- SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
|
||||
attestor:
|
||||
image: registry.stella-ops.org/stellaops/attestor@sha256:...
|
||||
depends_on: [signer]
|
||||
scanner-web:
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
|
||||
environment:
|
||||
- SCANNER__S3__ENDPOINT=http://minio:9000
|
||||
scanner-worker:
|
||||
image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
|
||||
deploy: { replicas: 4 }
|
||||
concelier:
|
||||
image: registry.stella-ops.org/stellaops/concelier@sha256:...
|
||||
excititor:
|
||||
image: registry.stella-ops.org/stellaops/excititor@sha256:...
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:...
|
||||
mongo:
|
||||
image: mongo:7
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2025-07-10T00-00-00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Governance & keys (who owns the trust root)
|
||||
|
||||
* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future.
|
||||
* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported.
|
||||
* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap (Ops)
|
||||
|
||||
* **Windows containers GA** (Scanner + Zastava).
|
||||
* **Key Transparency** for Signer certs.
|
||||
* **Delta‑kit** (offline) for incremental updates.
|
||||
* **Operator CRDs** (K8s) to manage policy and ILM declaratively.
|
||||
* **SBOM **protobuf** as default transport at rest (smaller, faster).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — Minimal SLO monitors
|
||||
|
||||
* `authority.tokens_issued_total` slope ≈ normal.
|
||||
* `signer.requests_total{result="success"}/minute` > 0 (when scans occur).
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
* Verify **release manifest** signature.
|
||||
* Ensure **Signer/Authority/Attestor** are same minor.
|
||||
* Verify **DB backups** < 24h old.
|
||||
* Confirm **ILM** won’t purge compliance artifacts during upgrade window.
|
||||
* Roll **one component** at a time; watch SLOs; abort on regression.
|
||||
|
||||
---
|
||||
|
||||
**End — component_architecture_devops.md**
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
## 0) Product vision (operations lens)
|
||||
|
||||
Stella Ops must be **trustable at a glance** and **boringly operable**:
|
||||
|
||||
* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime.
|
||||
* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels.
|
||||
* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**.
|
||||
* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import.
|
||||
* Artifacts expire predictably; operators know what’s kept, for how long, and why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Release trains & versioning
|
||||
|
||||
### 1.1 Channels
|
||||
|
||||
* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4).
|
||||
* **Stable** (default): monthly rollup (bug fixes + compatible features).
|
||||
* **Edge**: weekly; for early adopters, no guarantees.
|
||||
|
||||
### 1.2 Version strings
|
||||
|
||||
Semantic core + calendar tag:
|
||||
|
||||
```
|
||||
<MAJOR>.<MINOR>.<PATCH> (<YYYY>.<MM>) e.g., 2.4.1 (2027.06)
|
||||
```
|
||||
|
||||
* **MAJOR**: breaking API/DB changes (rare).
|
||||
* **MINOR**: new features, compatible schema migrations (expand/contract pattern).
|
||||
* **PATCH**: bug fixes, perf and security updates.
|
||||
* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**.
|
||||
|
||||
### 1.3 Component alignment
|
||||
|
||||
A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew:
|
||||
|
||||
* **Web UI ↔ backend**: `±1 minor`.
|
||||
* **Scanner ↔ Policy/Excititor/Concelier**: `±1 minor`.
|
||||
* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules).
|
||||
|
||||
At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**.
|
||||
|
||||
---
|
||||
|
||||
## 2) Supply‑chain pipeline (how a release is built)
|
||||
|
||||
### 2.1 Deterministic builds
|
||||
|
||||
* **Builders**: isolated **BuildKit** workers with pinned base images (digest only).
|
||||
* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag.
|
||||
* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars.
|
||||
* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap).
|
||||
|
||||
### 2.2 First‑party SBOMs & provenance
|
||||
|
||||
* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**.
|
||||
* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift).
|
||||
* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs.
|
||||
|
||||
### 2.3 Signing & transparency
|
||||
|
||||
* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required.
|
||||
* SBOM and provenance attestations are **DSSE** and also transparency‑logged.
|
||||
* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side).
|
||||
|
||||
### 2.4 Gates & tests
|
||||
|
||||
* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests).
|
||||
* **Unit/integration**: per‑component, plus **end‑to‑end** flows (scan→vex→policy→sign→attest).
|
||||
* **Perf SLOs**: hot paths (SBOM compose, diff, export) measured against budgets.
|
||||
* **Security**: dependency audit vs Concelier export; container hardening tests; minimal caps.
|
||||
* **Analyzer smoke**: restart-time language plug-ins (currently Python) verified via `dotnet run --project src/Tools/LanguageAnalyzerSmoke` to ensure manifest integrity plus cold vs warm determinism (< 30 s / < 5 s budgets); the harness logs deviations from repository goldens for follow-up.
|
||||
* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag.
|
||||
|
||||
### 2.5 Debug-store artefacts
|
||||
|
||||
* Every release exports stripped debug information for ELF binaries discovered in service images. Debug files follow the GNU build-id layout (`debug/.build-id/<aa>/<rest>.debug`) and are generated via `objcopy --only-keep-debug`.
|
||||
* `debug/debug-manifest.json` captures build-id → component/image/source mappings with SHA-256 checksums so operators can mirror the directory into debuginfod or offline symbol stores. The manifest (and its `.sha256` companion) ships with every release bundle and Offline Kit.
|
||||
|
||||
---
|
||||
|
||||
## 3) Distribution & activation
|
||||
|
||||
### 3.1 Registries
|
||||
|
||||
* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API).
|
||||
* **Mirrors**: GHCR (read‑only), regional mirrors for latency.
|
||||
* Operational runbook: see `docs/modules/concelier/operations/mirror.md` for deployment profiles, CDN guidance, and sync automation.
|
||||
* **Pull by digest only** in Kubernetes/Compose manifests.
|
||||
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Concelier, Excititor, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/Registry/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
### 3.2 OAuth2 token service (for private repos)
|
||||
|
||||
* Docker Registry’s token flow backed by **Authority**:
|
||||
|
||||
1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`).
|
||||
2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`.
|
||||
3. Registry allows pull for the requested repo.
|
||||
* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**.
|
||||
|
||||
The token service enforces plan gating via `registry-token.yaml` (see `docs/modules/registry/operations/token-service.md`) and exposes Prometheus metrics (`registry_token_issued_total`, `registry_token_rejected_total`). Revoked licence identifiers halt issuance even when scope requirements are met.
|
||||
|
||||
### 3.3 Offline kits (air‑gapped)
|
||||
|
||||
* Tarball per release channel:
|
||||
|
||||
```
|
||||
stellaops-kit-<ver>-<channel>.tar.zst
|
||||
/images/ OCI layout with all first-party images (multi-arch)
|
||||
/sboms/ CycloneDX JSON+PB for each image
|
||||
/attest/ DSSE bundles + Rekor proofs
|
||||
/charts/ Helm charts + values templates
|
||||
/compose/ docker-compose.yml + .env template
|
||||
/plugins/ Concelier/Excititor connectors (restart-time)
|
||||
/policy/ example policies
|
||||
/manifest/ release.yaml (see §6.1)
|
||||
```
|
||||
* Import via CLI `offline kit import`; checks digests and signatures before load.
|
||||
|
||||
---
|
||||
|
||||
## 4) Licensing (PoE) & monetization
|
||||
|
||||
**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE.
|
||||
|
||||
### 4.1 PoE issuance
|
||||
|
||||
* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`:
|
||||
|
||||
* **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**.
|
||||
* Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs.
|
||||
|
||||
### 4.2 Online enforcement
|
||||
|
||||
* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc).
|
||||
* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason.
|
||||
* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned.
|
||||
* UI badges: “**Verified by Stella Ops**” with link to the public log.
|
||||
|
||||
### 4.3 Air‑gapped / offline
|
||||
|
||||
* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days).
|
||||
* Signer accepts the lease and emits **provisional** attestations (clearly labeled).
|
||||
* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**.
|
||||
* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot).
|
||||
|
||||
### 4.4 Stolen/abused PoE
|
||||
|
||||
* Customers report theft; **Licensing** flags `license_id` as **revoked**.
|
||||
* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE).
|
||||
|
||||
---
|
||||
|
||||
## 5) Deployment path (customer side)
|
||||
|
||||
### 5.1 First install
|
||||
|
||||
* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s):
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://charts.stella-ops.org
|
||||
helm install stella stellaops/platform \
|
||||
--version 2.4.0 \
|
||||
--set global.channel=stable \
|
||||
--set authority.issuer=https://authority.stella.local \
|
||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||
--set scanner.mongo.uri=mongodb://mongo/scanner \
|
||||
--set concelier.mongo.uri=mongodb://mongo/concelier \
|
||||
--set excititor.mongo.uri=mongodb://mongo/excititor
|
||||
```
|
||||
|
||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||
* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?).
|
||||
|
||||
### 5.2 Updates
|
||||
|
||||
* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic.
|
||||
|
||||
* **Rolling**: upgrade stateful components in safe order:
|
||||
|
||||
1. Authority (stateless, dual‑key rotation ready)
|
||||
2. Signer/Attestor (same minor)
|
||||
3. Scanner WebService & Workers
|
||||
4. Concelier, then Excititor (schema migrations are expand/contract)
|
||||
5. UI last
|
||||
|
||||
* **DB migrations** are **expand/contract**:
|
||||
|
||||
* Phase A (release N): **add** new fields/indexes, write old+new.
|
||||
* Phase B (N+1): **read** new fields; **drop** old.
|
||||
* Rollback is a matter of redeploying previous images and keeping both schemas valid.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||
* Mongo migrations are additive; **no destructive changes** within a single minor.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release payloads & manifests
|
||||
|
||||
### 6.1 Release manifest (`release.yaml`)
|
||||
|
||||
```yaml
|
||||
release:
|
||||
version: "2.4.1"
|
||||
channel: "stable"
|
||||
date: "2027-06-20T12:00:00Z"
|
||||
calendar: "2027.06"
|
||||
components:
|
||||
- name: scanner-webservice
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb
|
||||
sbom: oci://.../referrers/cdx-json@sha256:11..22
|
||||
provenance: oci://.../attest/provenance@sha256:33..44
|
||||
signature: { rekorUUID: "…" }
|
||||
- name: signer
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd
|
||||
signature: { rekorUUID: "…" }
|
||||
charts:
|
||||
- name: platform
|
||||
version: "2.4.1"
|
||||
digest: "sha256:ee..ff"
|
||||
compose:
|
||||
file: "docker-compose.yml"
|
||||
digest: "sha256:77..88"
|
||||
checksums:
|
||||
sha256: "… digest of this release.yaml …"
|
||||
```
|
||||
|
||||
The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries.
|
||||
|
||||
> Deployment guardrails – The repository keeps channel-aligned Compose bundles
|
||||
> in `deploy/compose/` and Helm overlays in `deploy/helm/stellaops/`. Both sets
|
||||
> pull their digests from `deploy/releases/` and are validated by
|
||||
> `deploy/tools/validate-profiles.sh` to guarantee lint/dry-run cleanliness.
|
||||
|
||||
### 6.2 Image labels (release metadata)
|
||||
|
||||
Each image sets OCI labels:
|
||||
|
||||
```
|
||||
org.opencontainers.image.version = "2.4.1"
|
||||
org.opencontainers.image.revision = "<git sha>"
|
||||
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
|
||||
org.stellaops.release.calendar = "2027.06"
|
||||
org.stellaops.release.channel = "stable"
|
||||
org.stellaops.build.slsaProvenance = "oci://…"
|
||||
```
|
||||
|
||||
Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (RustFS/Mongo)
|
||||
|
||||
### 7.1 Buckets & prefixes (RustFS)
|
||||
|
||||
```
|
||||
rustfs://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json
|
||||
concelier/
|
||||
json/<exportId>/...
|
||||
trivy/<exportId>/...
|
||||
excititor/
|
||||
exports/<exportId>/...
|
||||
attestor/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
```
|
||||
|
||||
### 7.2 ILM classes
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**:
|
||||
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||
|
||||
### 7.4 Mongo retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||
|
||||
### 7.5 Mongo server baseline
|
||||
|
||||
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
||||
* **Deploy images:** Compose/Helm defaults stay on `mongo:7.x`. For air-gapped installs, refresh Offline Kit bundles so the packaged `mongod` matches ≥4.2.
|
||||
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & SLOs (operations)
|
||||
|
||||
* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Excititor/Concelier 99.0%.
|
||||
* **Error budgets**: tracked per month; dashboards show burn rates.
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
Prometheus + OTLP; Grafana dashboards ship in the charts.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security & compliance operations
|
||||
|
||||
* **Key rotation**:
|
||||
|
||||
* Authority JWKS: 60‑day cadence, dual‑key overlap.
|
||||
* Release signing identities: rotate per minor or quarterly.
|
||||
* Sigstore roots mirrored and pinned; alarms on drift.
|
||||
|
||||
* **FIPS mode** (Gov build):
|
||||
|
||||
* Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only.
|
||||
* Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA.
|
||||
|
||||
* **Vulnerability response**:
|
||||
|
||||
* Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
|
||||
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; repacked the local `Mongo2Go` feed so test fixtures inherit the patched dependencies; future bumps follow the same central override pattern.
|
||||
|
||||
* **Backups/DR**:
|
||||
|
||||
* Mongo nightly snapshots; MinIO versioning + replication (if configured).
|
||||
* Restore runbooks tested quarterly with synthetic data.
|
||||
|
||||
---
|
||||
|
||||
## 10) Customer update flow (how versions are fetched & activated)
|
||||
|
||||
### 10.1 Online clusters
|
||||
|
||||
* **UI** surfaces update banner with **release manifest** diff and risk notes.
|
||||
* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision.
|
||||
* Post‑switch, **schema Phase B** migrations (if any) run automatically.
|
||||
|
||||
### 10.2 Air‑gapped clusters
|
||||
|
||||
* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`.
|
||||
* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest.
|
||||
* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged.
|
||||
|
||||
### 10.3 CLI self‑update (optional)
|
||||
|
||||
* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable).
|
||||
|
||||
---
|
||||
|
||||
## 11) Compatibility & deprecation policy
|
||||
|
||||
* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor.
|
||||
* **Storage**: expand/contract; “drop old fields” only after one minor grace.
|
||||
* **Config**: feature flags (default off) for risky features (e.g., eBPF).
|
||||
|
||||
---
|
||||
|
||||
## 12) Runbooks (selected)
|
||||
|
||||
### 12.1 Lost PoE
|
||||
|
||||
1. Suspend **automatic attestation** jobs.
|
||||
2. Use CLI `stellaops signer status` to confirm `entitlement_denied`.
|
||||
3. Obtain new PoE from portal; verify on Signer `/poe/verify`.
|
||||
4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch).
|
||||
|
||||
### 12.2 Rekor outage (self‑hosted)
|
||||
|
||||
* Attestor returns `202 (pending)` with queued proof fetch.
|
||||
* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**.
|
||||
* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored.
|
||||
|
||||
### 12.3 Emergency downgrade
|
||||
|
||||
* Identify prior release manifest (UI → Admin → Releases).
|
||||
* `helm rollback stella <revision>` (or compose apply previous file).
|
||||
* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together.
|
||||
|
||||
---
|
||||
|
||||
## 13) Example: cluster bootstrap (Compose)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
authority:
|
||||
image: registry.stella-ops.org/stellaops/authority@sha256:...
|
||||
env_file: ./env/authority.env
|
||||
ports: ["8440:8440"]
|
||||
signer:
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:...
|
||||
depends_on: [authority]
|
||||
environment:
|
||||
- SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
|
||||
attestor:
|
||||
image: registry.stella-ops.org/stellaops/attestor@sha256:...
|
||||
depends_on: [signer]
|
||||
scanner-web:
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
|
||||
environment:
|
||||
- SCANNER__S3__ENDPOINT=http://minio:9000
|
||||
scanner-worker:
|
||||
image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
|
||||
deploy: { replicas: 4 }
|
||||
concelier:
|
||||
image: registry.stella-ops.org/stellaops/concelier@sha256:...
|
||||
excititor:
|
||||
image: registry.stella-ops.org/stellaops/excititor@sha256:...
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:...
|
||||
mongo:
|
||||
image: mongo:7
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2025-07-10T00-00-00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Governance & keys (who owns the trust root)
|
||||
|
||||
* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future.
|
||||
* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported.
|
||||
* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap (Ops)
|
||||
|
||||
* **Windows containers GA** (Scanner + Zastava).
|
||||
* **Key Transparency** for Signer certs.
|
||||
* **Delta‑kit** (offline) for incremental updates.
|
||||
* **Operator CRDs** (K8s) to manage policy and ILM declaratively.
|
||||
* **SBOM **protobuf** as default transport at rest (smaller, faster).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — Minimal SLO monitors
|
||||
|
||||
* `authority.tokens_issued_total` slope ≈ normal.
|
||||
* `signer.requests_total{result="success"}/minute` > 0 (when scans occur).
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
* Verify **release manifest** signature.
|
||||
* Ensure **Signer/Authority/Attestor** are same minor.
|
||||
* Verify **DB backups** < 24h old.
|
||||
* Confirm **ILM** won’t purge compliance artifacts during upgrade window.
|
||||
* Roll **one component** at a time; watch SLOs; abort on regression.
|
||||
|
||||
---
|
||||
|
||||
**End — component_architecture_devops.md**
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Implementation plan — DevOps
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** ensure CI/CD guardrails, schema validation, and verifier pipelines are enforced.
|
||||
- **Epic 9 – Orchestrator Dashboard:** deliver dashboards, recovery runbooks, and rate-limit governance.
|
||||
- **Epic 10 – Export Center:** manage signing/promotions and Offline Kit bundle publishing.
|
||||
- **Epic 15 – Observability & Forensics:** coordinate telemetry deployments, evidence retention, and forensic automation.
|
||||
- Track module runbooks (DEVOPS-LAUNCH-18-001/900) and telemetry automation via ../../TASKS.md and ops/devops/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — DevOps
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** ensure CI/CD guardrails, schema validation, and verifier pipelines are enforced.
|
||||
- **Epic 9 – Orchestrator Dashboard:** deliver dashboards, recovery runbooks, and rate-limit governance.
|
||||
- **Epic 10 – Export Center:** manage signing/promotions and Offline Kit bundle publishing.
|
||||
- **Epic 15 – Observability & Forensics:** coordinate telemetry deployments, evidence retention, and forensic automation.
|
||||
- Track module runbooks (DEVOPS-LAUNCH-18-001/900) and telemetry automation via ../../TASKS.md and ops/devops/TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,151 +1,151 @@
|
||||
# Stella Ops Deployment Upgrade & Rollback Runbook
|
||||
|
||||
_Last updated: 2025-10-26 (Sprint 14 – DEVOPS-OPS-14-003)._
|
||||
|
||||
This runbook describes how to promote a new release across the supported deployment profiles (Helm and Docker Compose), how to roll back safely, and how to keep channels (`edge`, `stable`, `airgap`) aligned. All steps assume you are working from a clean checkout of the release branch/tag.
|
||||
|
||||
---
|
||||
|
||||
## 1. Channel overview
|
||||
|
||||
| Channel | Release manifest | Helm values | Compose profile |
|
||||
|---------|------------------|-------------|-----------------|
|
||||
| `edge` | `deploy/releases/2025.10-edge.yaml` | `deploy/helm/stellaops/values-dev.yaml` | `deploy/compose/docker-compose.dev.yaml` |
|
||||
| `stable` | `deploy/releases/2025.09-stable.yaml` | `deploy/helm/stellaops/values-stage.yaml`, `deploy/helm/stellaops/values-prod.yaml` | `deploy/compose/docker-compose.stage.yaml`, `deploy/compose/docker-compose.prod.yaml` |
|
||||
| `airgap` | `deploy/releases/2025.09-airgap.yaml` | `deploy/helm/stellaops/values-airgap.yaml` | `deploy/compose/docker-compose.airgap.yaml` |
|
||||
|
||||
Infrastructure components (MongoDB, MinIO, RustFS) are pinned in the release manifests and inherited by the deployment profiles. Supporting dependencies such as `nats` remain on upstream LTS tags; review `deploy/compose/*.yaml` for the authoritative set.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pre-flight checklist
|
||||
|
||||
1. **Refresh release manifest**
|
||||
Pull the latest manifest for the channel you are promoting (`deploy/releases/<version>-<channel>.yaml`).
|
||||
|
||||
2. **Align deployment bundles with the manifest**
|
||||
Run the alignment checker for every profile that should pick up the release. Pass `--ignore-repo nats` to skip auxiliary services.
|
||||
```bash
|
||||
./deploy/tools/check-channel-alignment.py \
|
||||
--release deploy/releases/2025.10-edge.yaml \
|
||||
--target deploy/helm/stellaops/values-dev.yaml \
|
||||
--target deploy/compose/docker-compose.dev.yaml \
|
||||
--ignore-repo nats
|
||||
```
|
||||
Repeat for other channels (`stable`, `airgap`), substituting the manifest and target files.
|
||||
|
||||
3. **Lint and template profiles**
|
||||
```bash
|
||||
./deploy/tools/validate-profiles.sh
|
||||
```
|
||||
|
||||
4. **Smoke the Offline Kit debug store (edge/stable only)**
|
||||
When the release pipeline has generated `out/release/debug/.build-id/**`, mirror the assets into the Offline Kit staging tree:
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
Archive the resulting `out/offline-kit/metadata/debug-store.json` alongside the kit bundle.
|
||||
|
||||
5. **Review compatibility matrix**
|
||||
Confirm MongoDB, MinIO, and RustFS versions in the release manifest match platform SLOs. The default targets are `mongo@sha256:c258…`, `minio@sha256:14ce…`, `rustfs:2025.10.0-edge`.
|
||||
|
||||
6. **Create a rollback bookmark**
|
||||
Record the current Helm revision (`helm history stellaops -n stellaops`) and compose tag (`git describe --tags`) before applying changes.
|
||||
|
||||
---
|
||||
|
||||
## 3. Helm upgrade procedure (staging → production)
|
||||
|
||||
1. Switch to the deployment branch and ensure secrets/config maps are current.
|
||||
2. Apply the upgrade in the staging cluster:
|
||||
```bash
|
||||
helm upgrade stellaops deploy/helm/stellaops \
|
||||
-f deploy/helm/stellaops/values-stage.yaml \
|
||||
--namespace stellaops \
|
||||
--atomic \
|
||||
--timeout 15m
|
||||
```
|
||||
3. Run smoke tests (`scripts/smoke-tests.sh` or environment-specific checks).
|
||||
4. Promote to production using the prod values file and the same command.
|
||||
5. Record the new revision number and Git SHA in the change log.
|
||||
|
||||
### Rollback (Helm)
|
||||
|
||||
1. Identify the previous revision: `helm history stellaops -n stellaops`.
|
||||
2. Execute:
|
||||
```bash
|
||||
helm rollback stellaops <revision> \
|
||||
--namespace stellaops \
|
||||
--wait \
|
||||
--timeout 10m
|
||||
```
|
||||
3. Verify `kubectl get pods` returns healthy workloads; rerun smoke tests.
|
||||
4. Update the incident/operations log with root cause and rollback details.
|
||||
|
||||
---
|
||||
|
||||
## 4. Docker Compose upgrade procedure
|
||||
|
||||
1. Update environment files (`deploy/compose/env/*.env.example`) with any new settings and sync secrets to hosts.
|
||||
2. Pull the tagged repository state corresponding to the release (e.g. `git checkout 2025.09.2` for stable).
|
||||
3. Apply the upgrade:
|
||||
```bash
|
||||
docker compose \
|
||||
--env-file deploy/compose/env/prod.env \
|
||||
-f deploy/compose/docker-compose.prod.yaml \
|
||||
pull
|
||||
|
||||
docker compose \
|
||||
--env-file deploy/compose/env/prod.env \
|
||||
-f deploy/compose/docker-compose.prod.yaml \
|
||||
up -d
|
||||
```
|
||||
4. Tail logs for critical services (`docker compose logs -f authority concelier`).
|
||||
5. Update monitoring dashboards/alerts to confirm normal operation.
|
||||
|
||||
### Rollback (Compose)
|
||||
|
||||
1. Check out the previous release tag (e.g. `git checkout 2025.09.1`).
|
||||
2. Re-run `docker compose pull` and `docker compose up -d` with that profile. Docker will restore the prior digests.
|
||||
3. If reverting to a known-good snapshot is required, restore volume backups (see `docs/modules/authority/operations/backup-restore.md` and associated service guides).
|
||||
4. Log the rollback in the operations journal.
|
||||
|
||||
---
|
||||
|
||||
## 5. Channel promotion workflow
|
||||
|
||||
1. Author or update the channel manifest under `deploy/releases/`.
|
||||
2. Mirror the new digests into Helm/Compose values and run the alignment script for each profile.
|
||||
3. Commit the changes with a message that references the release version and channel (e.g. `deploy: promote 2025.10.0-edge`).
|
||||
4. Publish release notes and update `deploy/releases/README.md` (if applicable).
|
||||
5. Tag the repository when promoting stable or airgap builds.
|
||||
|
||||
---
|
||||
|
||||
## 6. Upgrade rehearsal & rollback drill log
|
||||
|
||||
Maintain rehearsal notes in `docs/modules/devops/runbooks/launch-cutover.md` or the relevant sprint planning document. After each drill capture:
|
||||
|
||||
- Release version tested
|
||||
- Date/time
|
||||
- Participants
|
||||
- Issues encountered & fixes
|
||||
- Rollback duration (if executed)
|
||||
|
||||
Attach the log to the sprint retro or operational wiki.
|
||||
|
||||
| Date (UTC) | Channel | Outcome | Notes |
|
||||
|------------|---------|---------|-------|
|
||||
| 2025-10-26 | Documentation dry-run | Planned | Runbook refreshed; next live drill scheduled for 2025-11 edge → stable promotion.
|
||||
|
||||
---
|
||||
|
||||
## 7. References
|
||||
|
||||
- `deploy/README.md` – structure and validation workflow for deployment bundles.
|
||||
- `docs/13_RELEASE_ENGINEERING_PLAYBOOK.md` – release automation and signing pipeline.
|
||||
- `docs/modules/devops/architecture.md` – high-level DevOps architecture, SLOs, and compliance requirements.
|
||||
- `ops/offline-kit/mirror_debug_store.py` – debug-store mirroring helper.
|
||||
- `deploy/tools/check-channel-alignment.py` – release vs deployment digest alignment checker.
|
||||
# Stella Ops Deployment Upgrade & Rollback Runbook
|
||||
|
||||
_Last updated: 2025-10-26 (Sprint 14 – DEVOPS-OPS-14-003)._
|
||||
|
||||
This runbook describes how to promote a new release across the supported deployment profiles (Helm and Docker Compose), how to roll back safely, and how to keep channels (`edge`, `stable`, `airgap`) aligned. All steps assume you are working from a clean checkout of the release branch/tag.
|
||||
|
||||
---
|
||||
|
||||
## 1. Channel overview
|
||||
|
||||
| Channel | Release manifest | Helm values | Compose profile |
|
||||
|---------|------------------|-------------|-----------------|
|
||||
| `edge` | `deploy/releases/2025.10-edge.yaml` | `deploy/helm/stellaops/values-dev.yaml` | `deploy/compose/docker-compose.dev.yaml` |
|
||||
| `stable` | `deploy/releases/2025.09-stable.yaml` | `deploy/helm/stellaops/values-stage.yaml`, `deploy/helm/stellaops/values-prod.yaml` | `deploy/compose/docker-compose.stage.yaml`, `deploy/compose/docker-compose.prod.yaml` |
|
||||
| `airgap` | `deploy/releases/2025.09-airgap.yaml` | `deploy/helm/stellaops/values-airgap.yaml` | `deploy/compose/docker-compose.airgap.yaml` |
|
||||
|
||||
Infrastructure components (MongoDB, MinIO, RustFS) are pinned in the release manifests and inherited by the deployment profiles. Supporting dependencies such as `nats` remain on upstream LTS tags; review `deploy/compose/*.yaml` for the authoritative set.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pre-flight checklist
|
||||
|
||||
1. **Refresh release manifest**
|
||||
Pull the latest manifest for the channel you are promoting (`deploy/releases/<version>-<channel>.yaml`).
|
||||
|
||||
2. **Align deployment bundles with the manifest**
|
||||
Run the alignment checker for every profile that should pick up the release. Pass `--ignore-repo nats` to skip auxiliary services.
|
||||
```bash
|
||||
./deploy/tools/check-channel-alignment.py \
|
||||
--release deploy/releases/2025.10-edge.yaml \
|
||||
--target deploy/helm/stellaops/values-dev.yaml \
|
||||
--target deploy/compose/docker-compose.dev.yaml \
|
||||
--ignore-repo nats
|
||||
```
|
||||
Repeat for other channels (`stable`, `airgap`), substituting the manifest and target files.
|
||||
|
||||
3. **Lint and template profiles**
|
||||
```bash
|
||||
./deploy/tools/validate-profiles.sh
|
||||
```
|
||||
|
||||
4. **Smoke the Offline Kit debug store (edge/stable only)**
|
||||
When the release pipeline has generated `out/release/debug/.build-id/**`, mirror the assets into the Offline Kit staging tree:
|
||||
```bash
|
||||
./ops/offline-kit/mirror_debug_store.py \
|
||||
--release-dir out/release \
|
||||
--offline-kit-dir out/offline-kit
|
||||
```
|
||||
Archive the resulting `out/offline-kit/metadata/debug-store.json` alongside the kit bundle.
|
||||
|
||||
5. **Review compatibility matrix**
|
||||
Confirm MongoDB, MinIO, and RustFS versions in the release manifest match platform SLOs. The default targets are `mongo@sha256:c258…`, `minio@sha256:14ce…`, `rustfs:2025.10.0-edge`.
|
||||
|
||||
6. **Create a rollback bookmark**
|
||||
Record the current Helm revision (`helm history stellaops -n stellaops`) and compose tag (`git describe --tags`) before applying changes.
|
||||
|
||||
---
|
||||
|
||||
## 3. Helm upgrade procedure (staging → production)
|
||||
|
||||
1. Switch to the deployment branch and ensure secrets/config maps are current.
|
||||
2. Apply the upgrade in the staging cluster:
|
||||
```bash
|
||||
helm upgrade stellaops deploy/helm/stellaops \
|
||||
-f deploy/helm/stellaops/values-stage.yaml \
|
||||
--namespace stellaops \
|
||||
--atomic \
|
||||
--timeout 15m
|
||||
```
|
||||
3. Run smoke tests (`scripts/smoke-tests.sh` or environment-specific checks).
|
||||
4. Promote to production using the prod values file and the same command.
|
||||
5. Record the new revision number and Git SHA in the change log.
|
||||
|
||||
### Rollback (Helm)
|
||||
|
||||
1. Identify the previous revision: `helm history stellaops -n stellaops`.
|
||||
2. Execute:
|
||||
```bash
|
||||
helm rollback stellaops <revision> \
|
||||
--namespace stellaops \
|
||||
--wait \
|
||||
--timeout 10m
|
||||
```
|
||||
3. Verify `kubectl get pods` returns healthy workloads; rerun smoke tests.
|
||||
4. Update the incident/operations log with root cause and rollback details.
|
||||
|
||||
---
|
||||
|
||||
## 4. Docker Compose upgrade procedure
|
||||
|
||||
1. Update environment files (`deploy/compose/env/*.env.example`) with any new settings and sync secrets to hosts.
|
||||
2. Pull the tagged repository state corresponding to the release (e.g. `git checkout 2025.09.2` for stable).
|
||||
3. Apply the upgrade:
|
||||
```bash
|
||||
docker compose \
|
||||
--env-file deploy/compose/env/prod.env \
|
||||
-f deploy/compose/docker-compose.prod.yaml \
|
||||
pull
|
||||
|
||||
docker compose \
|
||||
--env-file deploy/compose/env/prod.env \
|
||||
-f deploy/compose/docker-compose.prod.yaml \
|
||||
up -d
|
||||
```
|
||||
4. Tail logs for critical services (`docker compose logs -f authority concelier`).
|
||||
5. Update monitoring dashboards/alerts to confirm normal operation.
|
||||
|
||||
### Rollback (Compose)
|
||||
|
||||
1. Check out the previous release tag (e.g. `git checkout 2025.09.1`).
|
||||
2. Re-run `docker compose pull` and `docker compose up -d` with that profile. Docker will restore the prior digests.
|
||||
3. If reverting to a known-good snapshot is required, restore volume backups (see `docs/modules/authority/operations/backup-restore.md` and associated service guides).
|
||||
4. Log the rollback in the operations journal.
|
||||
|
||||
---
|
||||
|
||||
## 5. Channel promotion workflow
|
||||
|
||||
1. Author or update the channel manifest under `deploy/releases/`.
|
||||
2. Mirror the new digests into Helm/Compose values and run the alignment script for each profile.
|
||||
3. Commit the changes with a message that references the release version and channel (e.g. `deploy: promote 2025.10.0-edge`).
|
||||
4. Publish release notes and update `deploy/releases/README.md` (if applicable).
|
||||
5. Tag the repository when promoting stable or airgap builds.
|
||||
|
||||
---
|
||||
|
||||
## 6. Upgrade rehearsal & rollback drill log
|
||||
|
||||
Maintain rehearsal notes in `docs/modules/devops/runbooks/launch-cutover.md` or the relevant sprint planning document. After each drill capture:
|
||||
|
||||
- Release version tested
|
||||
- Date/time
|
||||
- Participants
|
||||
- Issues encountered & fixes
|
||||
- Rollback duration (if executed)
|
||||
|
||||
Attach the log to the sprint retro or operational wiki.
|
||||
|
||||
| Date (UTC) | Channel | Outcome | Notes |
|
||||
|------------|---------|---------|-------|
|
||||
| 2025-10-26 | Documentation dry-run | Planned | Runbook refreshed; next live drill scheduled for 2025-11 edge → stable promotion.
|
||||
|
||||
---
|
||||
|
||||
## 7. References
|
||||
|
||||
- `deploy/README.md` – structure and validation workflow for deployment bundles.
|
||||
- `docs/13_RELEASE_ENGINEERING_PLAYBOOK.md` – release automation and signing pipeline.
|
||||
- `docs/modules/devops/architecture.md` – high-level DevOps architecture, SLOs, and compliance requirements.
|
||||
- `ops/offline-kit/mirror_debug_store.py` – debug-store mirroring helper.
|
||||
- `deploy/tools/check-channel-alignment.py` – release vs deployment digest alignment checker.
|
||||
|
||||
@@ -1,128 +1,128 @@
|
||||
# Launch Cutover Runbook - Stella Ops
|
||||
|
||||
_Document owner: DevOps Guild (2025-10-26)_
|
||||
_Scope:_ Full-platform launch from staging to production for release `2025.09.2`.
|
||||
|
||||
## 1. Roles and Communication
|
||||
|
||||
| Role | Primary | Backup | Contact |
|
||||
| --- | --- | --- | --- |
|
||||
| Cutover lead | DevOps Guild (on-call engineer) | Platform Ops lead | `#launch-bridge` (Mattermost) |
|
||||
| Authority stack | Authority Core guild rep | Security guild rep | `#authority` |
|
||||
| Scanner / Queue | Scanner WebService guild rep | Runtime guild rep | `#scanner` |
|
||||
| Storage | Mongo/MinIO operators | Backup DB admin | Pager escalation |
|
||||
| Observability | Telemetry guild rep | SRE on-call | `#telemetry` |
|
||||
| Approvals | Product owner + CTO | DevOps lead | Approval recorded in change ticket |
|
||||
|
||||
Set up a bridge call 30 minutes before start and keep `#launch-bridge` updated every 10 minutes.
|
||||
|
||||
## 2. Timeline Overview (UTC)
|
||||
|
||||
| Time | Activity | Owner |
|
||||
| --- | --- | --- |
|
||||
| T-24h | Change ticket approved, prod secrets verified, offline kit build status checked (`DEVOPS-OFFLINE-18-005`). | DevOps lead |
|
||||
| T-12h | Run `deploy/tools/validate-profiles.sh`; capture logs in ticket. | DevOps engineer |
|
||||
| T-6h | Freeze non-launch deployments; notify guild leads. | Product owner |
|
||||
| T-2h | Execute rehearsal in staging (Section 3) using `values-stage.yaml` to verify scripts. | DevOps + module reps |
|
||||
| T-30m | Final go/no-go with guild leads; confirm monitoring dashboards green. | Cutover lead |
|
||||
| T0 | Execute production cutover steps (Section 4). | Cutover team |
|
||||
| T+45m | Smoke tests complete (Section 5); announce success or trigger rollback. | Cutover lead |
|
||||
| T+4h | Post-cutover metrics review, notify stakeholders, close ticket. | DevOps + product owner |
|
||||
|
||||
## 3. Rehearsal (Staging) Checklist
|
||||
|
||||
1. `docker network create stellaops_frontdoor || true` (if not present on staging jump host).
|
||||
2. Run `deploy/tools/validate-profiles.sh` and archive output.
|
||||
3. Apply staging secrets (`kubectl apply -f secrets/stage/*.yaml` or `helm secrets upgrade`) ensuring `stellaops-stage` credentials align with `values-stage.yaml`.
|
||||
4. Perform `helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-stage.yaml` in staging cluster.
|
||||
5. Verify health endpoints: `curl https://authority.stage.../healthz`, `curl https://scanner.stage.../healthz`.
|
||||
6. Execute smoke CLI: `stellaops-cli scan submit --profile staging --sbom samples/sbom/demo.json` and confirm report status in UI.
|
||||
7. Document total wall time and any deviations in the rehearsal log.
|
||||
|
||||
Rehearsal must complete without manual interventions before proceeding to production.
|
||||
|
||||
## 4. Production Cutover Steps
|
||||
|
||||
### 4.1 Pre-flight
|
||||
- Confirm production secrets in the appropriate secret store (`stellaops-prod-core`, `stellaops-prod-mongo`, `stellaops-prod-minio`, `stellaops-prod-notify`) contain the keys referenced in `values-prod.yaml`.
|
||||
- Ensure the external reverse proxy network exists: `docker network create stellaops_frontdoor || true` on each compose host.
|
||||
- Back up current configuration and data:
|
||||
- Mongo snapshot: `mongodump --uri "$MONGO_BACKUP_URI" --out /backups/launch-$(date -Iseconds)`.
|
||||
- MinIO policy export: `mc mirror --overwrite minio/stellaops minio-backup/stellaops-$(date +%Y%m%d%H%M)`.
|
||||
|
||||
### 4.2 Apply Updates (Compose)
|
||||
1. On each compose node, pull updated images for release `2025.09.2`:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml pull
|
||||
```
|
||||
2. Deploy changes:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml up -d
|
||||
```
|
||||
3. Confirm containers healthy via `docker compose ps` and `docker logs <service> --tail 50`.
|
||||
|
||||
### 4.3 Apply Updates (Helm/Kubernetes)
|
||||
If using Kubernetes, perform:
|
||||
```bash
|
||||
helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-prod.yaml --atomic --timeout 15m
|
||||
```
|
||||
Monitor rollout with `kubectl get pods -n stellaops --watch` and `kubectl rollout status deployment/<service>`.
|
||||
|
||||
### 4.4 Configuration Validation
|
||||
- Verify Authority issuer metadata: `curl https://authority.prod.../.well-known/openid-configuration`.
|
||||
- Validate Signer DSSE endpoint: `stellaops-cli signer verify --base-url https://signer.prod... --bundle samples/dsse/demo.json`.
|
||||
- Check Scanner queue connectivity: `docker exec stellaops-scanner-web dotnet StellaOps.Scanner.WebService.dll health queue` (returns success).
|
||||
- Ensure Notify (legacy) still accessible while Notifier migration pending.
|
||||
|
||||
## 5. Smoke Tests
|
||||
|
||||
| Test | Command / Action | Expected Result |
|
||||
| --- | --- | --- |
|
||||
| API health | `curl https://scanner.prod.../healthz` | HTTP 200 with `status":"Healthy"` |
|
||||
| Scan submit | `stellaops-cli scan submit --profile prod --sbom samples/sbom/demo.json` | Scan completes < 5 minutes; report accessible with signed DSSE |
|
||||
| Runtime event ingest | Post sample event from Zastava observer fixture | `/runtime/events` responds 202 Accepted; record visible in Mongo `runtime_events` |
|
||||
| Signing | `stellaops-cli signer sign --bundle demo.json` | Returns DSSE with matching SHA256 and signer metadata |
|
||||
| Attestor verify | `stellaops-cli attestor verify --uuid <uuid>` | Verification result `ok=true` |
|
||||
| Web UI | Manual login, verify dashboards render and latency within budget | UI loads under 2 seconds; policy views consistent |
|
||||
|
||||
Log results in the change ticket with timestamps and screenshots where applicable.
|
||||
|
||||
## 6. Rollback Procedure
|
||||
|
||||
1. Assess failure scope; if systemic, initiate rollback immediately while preserving logs/artifacts.
|
||||
2. For Compose:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml down
|
||||
docker compose --env-file stage.env -f deploy/compose/docker-compose.stage.yaml up -d
|
||||
```
|
||||
3. For Helm:
|
||||
```bash
|
||||
helm rollback stellaops <previous-release-number> --namespace stellaops
|
||||
```
|
||||
4. Restore Mongo snapshot if data inconsistency detected: `mongorestore --uri "$MONGO_BACKUP_URI" --drop /backups/launch-<timestamp>`.
|
||||
5. Restore MinIO mirror if required: `mc mirror minio-backup/stellaops-<timestamp> minio/stellaops`.
|
||||
6. Notify stakeholders of rollback and capture root cause notes in incident ticket.
|
||||
|
||||
## 7. Post-cutover Actions
|
||||
|
||||
- Keep heightened monitoring for 4 hours post cutover; track latency, error rates, and queue depth.
|
||||
- Confirm audit trails: Authority tokens issued, Scanner events recorded, Attestor submissions stored.
|
||||
- Update `docs/modules/devops/runbooks/launch-readiness.md` if any new gaps or follow-ups discovered.
|
||||
- Schedule retrospective within 48 hours; include DevOps, module guilds, and product owner.
|
||||
|
||||
## 8. Approval Matrix
|
||||
|
||||
| Step | Required Approvers | Record Location |
|
||||
| --- | --- | --- |
|
||||
| Production deployment plan | CTO + DevOps lead | Change ticket comment |
|
||||
| Cutover start (T0) | DevOps lead + module reps | `#launch-bridge` summary |
|
||||
| Post-smoke success | DevOps lead + product owner | Change ticket closure |
|
||||
| Rollback (if invoked) | DevOps lead + CTO | Incident ticket |
|
||||
|
||||
Retain all approvals and logs for audit. Update this runbook after each execution to record actual timings and lessons learned.
|
||||
|
||||
## 9. Rehearsal Log
|
||||
|
||||
| Date (UTC) | What We Exercised | Outcome | Follow-up |
|
||||
| --- | --- | --- | --- |
|
||||
| 2025-10-26 | Dry-run of compose/Helm validation via `deploy/tools/validate-profiles.sh` (dev/stage/prod/airgap/mirror). Network creation simulated (`docker network create stellaops_frontdoor` planned) and stage CLI submission reviewed. | Validation script succeeded; all profiles templated cleanly. Stage deployment apply deferred because no staging cluster is accessible from the current environment. | Schedule full stage rehearsal once staging cluster credentials are available; reuse this log section to capture timings. |
|
||||
# Launch Cutover Runbook - Stella Ops
|
||||
|
||||
_Document owner: DevOps Guild (2025-10-26)_
|
||||
_Scope:_ Full-platform launch from staging to production for release `2025.09.2`.
|
||||
|
||||
## 1. Roles and Communication
|
||||
|
||||
| Role | Primary | Backup | Contact |
|
||||
| --- | --- | --- | --- |
|
||||
| Cutover lead | DevOps Guild (on-call engineer) | Platform Ops lead | `#launch-bridge` (Mattermost) |
|
||||
| Authority stack | Authority Core guild rep | Security guild rep | `#authority` |
|
||||
| Scanner / Queue | Scanner WebService guild rep | Runtime guild rep | `#scanner` |
|
||||
| Storage | Mongo/MinIO operators | Backup DB admin | Pager escalation |
|
||||
| Observability | Telemetry guild rep | SRE on-call | `#telemetry` |
|
||||
| Approvals | Product owner + CTO | DevOps lead | Approval recorded in change ticket |
|
||||
|
||||
Set up a bridge call 30 minutes before start and keep `#launch-bridge` updated every 10 minutes.
|
||||
|
||||
## 2. Timeline Overview (UTC)
|
||||
|
||||
| Time | Activity | Owner |
|
||||
| --- | --- | --- |
|
||||
| T-24h | Change ticket approved, prod secrets verified, offline kit build status checked (`DEVOPS-OFFLINE-18-005`). | DevOps lead |
|
||||
| T-12h | Run `deploy/tools/validate-profiles.sh`; capture logs in ticket. | DevOps engineer |
|
||||
| T-6h | Freeze non-launch deployments; notify guild leads. | Product owner |
|
||||
| T-2h | Execute rehearsal in staging (Section 3) using `values-stage.yaml` to verify scripts. | DevOps + module reps |
|
||||
| T-30m | Final go/no-go with guild leads; confirm monitoring dashboards green. | Cutover lead |
|
||||
| T0 | Execute production cutover steps (Section 4). | Cutover team |
|
||||
| T+45m | Smoke tests complete (Section 5); announce success or trigger rollback. | Cutover lead |
|
||||
| T+4h | Post-cutover metrics review, notify stakeholders, close ticket. | DevOps + product owner |
|
||||
|
||||
## 3. Rehearsal (Staging) Checklist
|
||||
|
||||
1. `docker network create stellaops_frontdoor || true` (if not present on staging jump host).
|
||||
2. Run `deploy/tools/validate-profiles.sh` and archive output.
|
||||
3. Apply staging secrets (`kubectl apply -f secrets/stage/*.yaml` or `helm secrets upgrade`) ensuring `stellaops-stage` credentials align with `values-stage.yaml`.
|
||||
4. Perform `helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-stage.yaml` in staging cluster.
|
||||
5. Verify health endpoints: `curl https://authority.stage.../healthz`, `curl https://scanner.stage.../healthz`.
|
||||
6. Execute smoke CLI: `stellaops-cli scan submit --profile staging --sbom samples/sbom/demo.json` and confirm report status in UI.
|
||||
7. Document total wall time and any deviations in the rehearsal log.
|
||||
|
||||
Rehearsal must complete without manual interventions before proceeding to production.
|
||||
|
||||
## 4. Production Cutover Steps
|
||||
|
||||
### 4.1 Pre-flight
|
||||
- Confirm production secrets in the appropriate secret store (`stellaops-prod-core`, `stellaops-prod-mongo`, `stellaops-prod-minio`, `stellaops-prod-notify`) contain the keys referenced in `values-prod.yaml`.
|
||||
- Ensure the external reverse proxy network exists: `docker network create stellaops_frontdoor || true` on each compose host.
|
||||
- Back up current configuration and data:
|
||||
- Mongo snapshot: `mongodump --uri "$MONGO_BACKUP_URI" --out /backups/launch-$(date -Iseconds)`.
|
||||
- MinIO policy export: `mc mirror --overwrite minio/stellaops minio-backup/stellaops-$(date +%Y%m%d%H%M)`.
|
||||
|
||||
### 4.2 Apply Updates (Compose)
|
||||
1. On each compose node, pull updated images for release `2025.09.2`:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml pull
|
||||
```
|
||||
2. Deploy changes:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml up -d
|
||||
```
|
||||
3. Confirm containers healthy via `docker compose ps` and `docker logs <service> --tail 50`.
|
||||
|
||||
### 4.3 Apply Updates (Helm/Kubernetes)
|
||||
If using Kubernetes, perform:
|
||||
```bash
|
||||
helm upgrade stellaops deploy/helm/stellaops -f deploy/helm/stellaops/values-prod.yaml --atomic --timeout 15m
|
||||
```
|
||||
Monitor rollout with `kubectl get pods -n stellaops --watch` and `kubectl rollout status deployment/<service>`.
|
||||
|
||||
### 4.4 Configuration Validation
|
||||
- Verify Authority issuer metadata: `curl https://authority.prod.../.well-known/openid-configuration`.
|
||||
- Validate Signer DSSE endpoint: `stellaops-cli signer verify --base-url https://signer.prod... --bundle samples/dsse/demo.json`.
|
||||
- Check Scanner queue connectivity: `docker exec stellaops-scanner-web dotnet StellaOps.Scanner.WebService.dll health queue` (returns success).
|
||||
- Ensure Notify (legacy) still accessible while Notifier migration pending.
|
||||
|
||||
## 5. Smoke Tests
|
||||
|
||||
| Test | Command / Action | Expected Result |
|
||||
| --- | --- | --- |
|
||||
| API health | `curl https://scanner.prod.../healthz` | HTTP 200 with `status":"Healthy"` |
|
||||
| Scan submit | `stellaops-cli scan submit --profile prod --sbom samples/sbom/demo.json` | Scan completes < 5 minutes; report accessible with signed DSSE |
|
||||
| Runtime event ingest | Post sample event from Zastava observer fixture | `/runtime/events` responds 202 Accepted; record visible in Mongo `runtime_events` |
|
||||
| Signing | `stellaops-cli signer sign --bundle demo.json` | Returns DSSE with matching SHA256 and signer metadata |
|
||||
| Attestor verify | `stellaops-cli attestor verify --uuid <uuid>` | Verification result `ok=true` |
|
||||
| Web UI | Manual login, verify dashboards render and latency within budget | UI loads under 2 seconds; policy views consistent |
|
||||
|
||||
Log results in the change ticket with timestamps and screenshots where applicable.
|
||||
|
||||
## 6. Rollback Procedure
|
||||
|
||||
1. Assess failure scope; if systemic, initiate rollback immediately while preserving logs/artifacts.
|
||||
2. For Compose:
|
||||
```bash
|
||||
docker compose --env-file prod.env -f deploy/compose/docker-compose.prod.yaml down
|
||||
docker compose --env-file stage.env -f deploy/compose/docker-compose.stage.yaml up -d
|
||||
```
|
||||
3. For Helm:
|
||||
```bash
|
||||
helm rollback stellaops <previous-release-number> --namespace stellaops
|
||||
```
|
||||
4. Restore Mongo snapshot if data inconsistency detected: `mongorestore --uri "$MONGO_BACKUP_URI" --drop /backups/launch-<timestamp>`.
|
||||
5. Restore MinIO mirror if required: `mc mirror minio-backup/stellaops-<timestamp> minio/stellaops`.
|
||||
6. Notify stakeholders of rollback and capture root cause notes in incident ticket.
|
||||
|
||||
## 7. Post-cutover Actions
|
||||
|
||||
- Keep heightened monitoring for 4 hours post cutover; track latency, error rates, and queue depth.
|
||||
- Confirm audit trails: Authority tokens issued, Scanner events recorded, Attestor submissions stored.
|
||||
- Update `docs/modules/devops/runbooks/launch-readiness.md` if any new gaps or follow-ups discovered.
|
||||
- Schedule retrospective within 48 hours; include DevOps, module guilds, and product owner.
|
||||
|
||||
## 8. Approval Matrix
|
||||
|
||||
| Step | Required Approvers | Record Location |
|
||||
| --- | --- | --- |
|
||||
| Production deployment plan | CTO + DevOps lead | Change ticket comment |
|
||||
| Cutover start (T0) | DevOps lead + module reps | `#launch-bridge` summary |
|
||||
| Post-smoke success | DevOps lead + product owner | Change ticket closure |
|
||||
| Rollback (if invoked) | DevOps lead + CTO | Incident ticket |
|
||||
|
||||
Retain all approvals and logs for audit. Update this runbook after each execution to record actual timings and lessons learned.
|
||||
|
||||
## 9. Rehearsal Log
|
||||
|
||||
| Date (UTC) | What We Exercised | Outcome | Follow-up |
|
||||
| --- | --- | --- | --- |
|
||||
| 2025-10-26 | Dry-run of compose/Helm validation via `deploy/tools/validate-profiles.sh` (dev/stage/prod/airgap/mirror). Network creation simulated (`docker network create stellaops_frontdoor` planned) and stage CLI submission reviewed. | Validation script succeeded; all profiles templated cleanly. Stage deployment apply deferred because no staging cluster is accessible from the current environment. | Schedule full stage rehearsal once staging cluster credentials are available; reuse this log section to capture timings. |
|
||||
|
||||
@@ -1,49 +1,49 @@
|
||||
# Launch Readiness Record - Stella Ops
|
||||
|
||||
_Updated: 2025-10-26 (UTC)_
|
||||
|
||||
This document captures production launch sign-offs, deployment readiness checkpoints, and any open risks that must be tracked before GA cutover.
|
||||
|
||||
## 1. Sign-off Summary
|
||||
|
||||
| Module / Service | Guild / Point of Contact | Evidence (Task or Runbook) | Status | Timestamp (UTC) | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Authority (Issuer) | Authority Core Guild | `AUTH-AOC-19-001` - scope issuance & configuration complete (DONE 2025-10-26) | READY | 2025-10-26T14:05Z | Tenant scope propagation follow-up (`AUTH-AOC-19-002`) tracked in gaps section. |
|
||||
| Signer | Signer Guild | `SIGNER-API-11-101` / `SIGNER-REF-11-102` / `SIGNER-QUOTA-11-103` (DONE 2025-10-21) | READY | 2025-10-26T14:07Z | DSSE signing, referrer verification, and quota enforcement validated in CI. |
|
||||
| Attestor | Attestor Guild | `ATTESTOR-API-11-201` / `ATTESTOR-VERIFY-11-202` / `ATTESTOR-OBS-11-203` (DONE 2025-10-19) | READY | 2025-10-26T14:10Z | Rekor submission/verification pipeline green; telemetry pack published. |
|
||||
| Scanner Web + Worker | Scanner WebService Guild | `SCANNER-WEB-09-10x`, `SCANNER-RUNTIME-12-30x` (DONE 2025-10-18 -> 2025-10-24) | READY* | 2025-10-26T14:20Z | Orchestrator envelope work (`SCANNER-EVENTS-16-301/302`) still open; see gaps. |
|
||||
| Concelier Core & Connectors | Concelier Core / Ops Guild | Ops runbook sign-off in `docs/modules/concelier/operations/conflict-resolution.md` (2025-10-16) | READY | 2025-10-26T14:25Z | Conflict resolution & connector coverage accepted; Mongo schema hardening pending (see gaps). |
|
||||
| Excititor API | Excititor Core Guild | Wave 0 connector ingest sign-offs (EXECPLAN.Section Wave 0) | READY | 2025-10-26T14:28Z | VEX linkset publishing complete for launch datasets. |
|
||||
| Notify Web (legacy) | Notify Guild | Existing stack carried forward; Notifier program tracked separately (Sprint 38-40) | PENDING | 2025-10-26T14:32Z | Legacy notify web remains operational; migration to Notifier blocked on `SCANNER-EVENTS-16-301`. |
|
||||
| Web UI | UI Guild | Stable build `registry.stella-ops.org/.../web-ui@sha256:10d9248...` deployed in stage and smoke-tested | READY | 2025-10-26T14:35Z | Policy editor GA items (Sprint 20) outside launch scope. |
|
||||
| DevOps / Release | DevOps Guild | `deploy/tools/validate-profiles.sh` run (2025-10-26) covering dev/stage/prod/airgap/mirror | READY | 2025-10-26T15:02Z | Compose/Helm lint + docker compose config validated; see Section 2 for details. |
|
||||
| Offline Kit | Offline Kit Guild | `DEVOPS-OFFLINE-18-004` (Go analyzer) and `DEVOPS-OFFLINE-18-005` (Python analyzer) complete; debug-store mirror pending (`DEVOPS-OFFLINE-17-004`). | PENDING | 2025-10-26T15:05Z | Awaiting release debug artefacts to finalise `DEVOPS-OFFLINE-17-004`; tracked in Section 3. |
|
||||
|
||||
_\* READY with caveat - remaining work noted in Section 3._
|
||||
|
||||
## 2. Deployment Readiness Checklist
|
||||
|
||||
- **Production profiles committed:** `deploy/compose/docker-compose.prod.yaml` and `deploy/helm/stellaops/values-prod.yaml` added with front-door network hand-off and secret references for Mongo/MinIO/core services.
|
||||
- **Secrets placeholders documented:** `deploy/compose/env/prod.env.example` enumerates required credentials (`MONGO_INITDB_ROOT_PASSWORD`, `MINIO_ROOT_PASSWORD`, Redis/NATS endpoints, `FRONTDOOR_NETWORK`). Helm values reference Kubernetes secrets (`stellaops-prod-core`, `stellaops-prod-mongo`, `stellaops-prod-minio`, `stellaops-prod-notify`).
|
||||
- **Static validation executed:** `deploy/tools/validate-profiles.sh` run on 2025-10-26 (docker compose config + helm lint/template) with all profiles passing.
|
||||
- **Ingress model defined:** Production compose profile introduces external `frontdoor` network; README updated with creation instructions and scope of externally reachable services.
|
||||
- **Observability hooks:** Authority/Signer/Attestor telemetry packs verified; scanner runtime build-id metrics landed (`SCANNER-RUNTIME-17-401`). Grafana dashboards referenced in component runbooks.
|
||||
- **Rollback assets:** Stage Compose profile remains aligned (`docker-compose.stage.yaml`), enabling rehearsals before prod cutover; release manifests (`deploy/releases/2025.09-stable.yaml`) map digests for reproducible rollback.
|
||||
- **Rehearsal status:** 2025-10-26 validation dry-run executed (`deploy/tools/validate-profiles.sh` across dev/stage/prod/airgap/mirror). Full stage Helm rollout pending access to the managed staging cluster; target to complete once credentials are provisioned.
|
||||
|
||||
## 3. Outstanding Gaps & Follow-ups
|
||||
|
||||
| Item | Owner | Tracking Ref | Target / Next Step | Impact |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Tenant scope propagation and audit coverage | Authority Core Guild | `AUTH-AOC-19-002` (DOING 2025-10-26) | Land enforcement + audit fixtures by Sprint 19 freeze | Medium - required for multi-tenant GA but does not block initial cutover if tenants scoped manually. |
|
||||
| Orchestrator event envelopes + Notifier handshake | Scanner WebService Guild | `SCANNER-EVENTS-16-301` (BLOCKED), `SCANNER-EVENTS-16-302` (DOING) | Coordinate with Gateway/Notifier owners on preview package replacement or binding redirects; rerun `dotnet test` once patch lands and refresh schema docs. Share envelope samples in `docs/events/` after tests pass. | High — gating Notifier migration; legacy notify path remains functional meanwhile. |
|
||||
| Offline Kit Python analyzer bundle | Offline Kit Guild + Scanner Guild | `DEVOPS-OFFLINE-18-005` (DONE 2025-10-26) | Monitor for follow-up manifest updates and rerun smoke script when analyzers change. | Medium - ensures language analyzer coverage stays current for offline installs. |
|
||||
| Offline Kit debug store mirror | Offline Kit Guild + DevOps Guild | `DEVOPS-OFFLINE-17-004` (BLOCKED 2025-10-26) | Release pipeline must publish `out/release/debug` artefacts; once available, run `mirror_debug_store.py` and commit `metadata/debug-store.json`. | Low - symbol lookup remains accessible from staging assets but required before next Offline Kit tag. |
|
||||
| Mongo schema validators for advisory ingestion | Concelier Storage Guild | `CONCELIER-STORE-AOC-19-001` (TODO) | Finalize JSON schema + migration toggles; coordinate with Ops for rollout window | Low - current validation handled in app layer; schema guard adds defense-in-depth. |
|
||||
| Authority plugin telemetry alignment | Security Guild | `SEC2.PLG`, `SEC3.PLG`, `SEC5.PLG` (BLOCKED pending AUTH DPoP/MTLS tasks) | Resume once upstream auth surfacing stabilises | Low - plugin remains optional; launch uses default Authority configuration. |
|
||||
|
||||
## 4. Approvals & Distribution
|
||||
|
||||
- Record shared in `#launch-readiness` (Mattermost) 2025-10-26 15:15 UTC with DevOps + Guild leads for acknowledgement.
|
||||
- Updates to this document require dual sign-off from DevOps Guild (owner) and impacted module guild lead; retain change log via Git history.
|
||||
- Cutover rehearsal and rollback drills are tracked separately in `docs/modules/devops/runbooks/launch-cutover.md` (see associated Task `DEVOPS-LAUNCH-18-001`). *** End Patch
|
||||
# Launch Readiness Record - Stella Ops
|
||||
|
||||
_Updated: 2025-10-26 (UTC)_
|
||||
|
||||
This document captures production launch sign-offs, deployment readiness checkpoints, and any open risks that must be tracked before GA cutover.
|
||||
|
||||
## 1. Sign-off Summary
|
||||
|
||||
| Module / Service | Guild / Point of Contact | Evidence (Task or Runbook) | Status | Timestamp (UTC) | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Authority (Issuer) | Authority Core Guild | `AUTH-AOC-19-001` - scope issuance & configuration complete (DONE 2025-10-26) | READY | 2025-10-26T14:05Z | Tenant scope propagation follow-up (`AUTH-AOC-19-002`) tracked in gaps section. |
|
||||
| Signer | Signer Guild | `SIGNER-API-11-101` / `SIGNER-REF-11-102` / `SIGNER-QUOTA-11-103` (DONE 2025-10-21) | READY | 2025-10-26T14:07Z | DSSE signing, referrer verification, and quota enforcement validated in CI. |
|
||||
| Attestor | Attestor Guild | `ATTESTOR-API-11-201` / `ATTESTOR-VERIFY-11-202` / `ATTESTOR-OBS-11-203` (DONE 2025-10-19) | READY | 2025-10-26T14:10Z | Rekor submission/verification pipeline green; telemetry pack published. |
|
||||
| Scanner Web + Worker | Scanner WebService Guild | `SCANNER-WEB-09-10x`, `SCANNER-RUNTIME-12-30x` (DONE 2025-10-18 -> 2025-10-24) | READY* | 2025-10-26T14:20Z | Orchestrator envelope work (`SCANNER-EVENTS-16-301/302`) still open; see gaps. |
|
||||
| Concelier Core & Connectors | Concelier Core / Ops Guild | Ops runbook sign-off in `docs/modules/concelier/operations/conflict-resolution.md` (2025-10-16) | READY | 2025-10-26T14:25Z | Conflict resolution & connector coverage accepted; Mongo schema hardening pending (see gaps). |
|
||||
| Excititor API | Excititor Core Guild | Wave 0 connector ingest sign-offs (EXECPLAN.Section Wave 0) | READY | 2025-10-26T14:28Z | VEX linkset publishing complete for launch datasets. |
|
||||
| Notify Web (legacy) | Notify Guild | Existing stack carried forward; Notifier program tracked separately (Sprint 38-40) | PENDING | 2025-10-26T14:32Z | Legacy notify web remains operational; migration to Notifier blocked on `SCANNER-EVENTS-16-301`. |
|
||||
| Web UI | UI Guild | Stable build `registry.stella-ops.org/.../web-ui@sha256:10d9248...` deployed in stage and smoke-tested | READY | 2025-10-26T14:35Z | Policy editor GA items (Sprint 20) outside launch scope. |
|
||||
| DevOps / Release | DevOps Guild | `deploy/tools/validate-profiles.sh` run (2025-10-26) covering dev/stage/prod/airgap/mirror | READY | 2025-10-26T15:02Z | Compose/Helm lint + docker compose config validated; see Section 2 for details. |
|
||||
| Offline Kit | Offline Kit Guild | `DEVOPS-OFFLINE-18-004` (Go analyzer) and `DEVOPS-OFFLINE-18-005` (Python analyzer) complete; debug-store mirror pending (`DEVOPS-OFFLINE-17-004`). | PENDING | 2025-10-26T15:05Z | Awaiting release debug artefacts to finalise `DEVOPS-OFFLINE-17-004`; tracked in Section 3. |
|
||||
|
||||
_\* READY with caveat - remaining work noted in Section 3._
|
||||
|
||||
## 2. Deployment Readiness Checklist
|
||||
|
||||
- **Production profiles committed:** `deploy/compose/docker-compose.prod.yaml` and `deploy/helm/stellaops/values-prod.yaml` added with front-door network hand-off and secret references for Mongo/MinIO/core services.
|
||||
- **Secrets placeholders documented:** `deploy/compose/env/prod.env.example` enumerates required credentials (`MONGO_INITDB_ROOT_PASSWORD`, `MINIO_ROOT_PASSWORD`, Redis/NATS endpoints, `FRONTDOOR_NETWORK`). Helm values reference Kubernetes secrets (`stellaops-prod-core`, `stellaops-prod-mongo`, `stellaops-prod-minio`, `stellaops-prod-notify`).
|
||||
- **Static validation executed:** `deploy/tools/validate-profiles.sh` run on 2025-10-26 (docker compose config + helm lint/template) with all profiles passing.
|
||||
- **Ingress model defined:** Production compose profile introduces external `frontdoor` network; README updated with creation instructions and scope of externally reachable services.
|
||||
- **Observability hooks:** Authority/Signer/Attestor telemetry packs verified; scanner runtime build-id metrics landed (`SCANNER-RUNTIME-17-401`). Grafana dashboards referenced in component runbooks.
|
||||
- **Rollback assets:** Stage Compose profile remains aligned (`docker-compose.stage.yaml`), enabling rehearsals before prod cutover; release manifests (`deploy/releases/2025.09-stable.yaml`) map digests for reproducible rollback.
|
||||
- **Rehearsal status:** 2025-10-26 validation dry-run executed (`deploy/tools/validate-profiles.sh` across dev/stage/prod/airgap/mirror). Full stage Helm rollout pending access to the managed staging cluster; target to complete once credentials are provisioned.
|
||||
|
||||
## 3. Outstanding Gaps & Follow-ups
|
||||
|
||||
| Item | Owner | Tracking Ref | Target / Next Step | Impact |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Tenant scope propagation and audit coverage | Authority Core Guild | `AUTH-AOC-19-002` (DOING 2025-10-26) | Land enforcement + audit fixtures by Sprint 19 freeze | Medium - required for multi-tenant GA but does not block initial cutover if tenants scoped manually. |
|
||||
| Orchestrator event envelopes + Notifier handshake | Scanner WebService Guild | `SCANNER-EVENTS-16-301` (BLOCKED), `SCANNER-EVENTS-16-302` (DOING) | Coordinate with Gateway/Notifier owners on preview package replacement or binding redirects; rerun `dotnet test` once patch lands and refresh schema docs. Share envelope samples in `docs/events/` after tests pass. | High — gating Notifier migration; legacy notify path remains functional meanwhile. |
|
||||
| Offline Kit Python analyzer bundle | Offline Kit Guild + Scanner Guild | `DEVOPS-OFFLINE-18-005` (DONE 2025-10-26) | Monitor for follow-up manifest updates and rerun smoke script when analyzers change. | Medium - ensures language analyzer coverage stays current for offline installs. |
|
||||
| Offline Kit debug store mirror | Offline Kit Guild + DevOps Guild | `DEVOPS-OFFLINE-17-004` (BLOCKED 2025-10-26) | Release pipeline must publish `out/release/debug` artefacts; once available, run `mirror_debug_store.py` and commit `metadata/debug-store.json`. | Low - symbol lookup remains accessible from staging assets but required before next Offline Kit tag. |
|
||||
| Mongo schema validators for advisory ingestion | Concelier Storage Guild | `CONCELIER-STORE-AOC-19-001` (TODO) | Finalize JSON schema + migration toggles; coordinate with Ops for rollout window | Low - current validation handled in app layer; schema guard adds defense-in-depth. |
|
||||
| Authority plugin telemetry alignment | Security Guild | `SEC2.PLG`, `SEC3.PLG`, `SEC5.PLG` (BLOCKED pending AUTH DPoP/MTLS tasks) | Resume once upstream auth surfacing stabilises | Low - plugin remains optional; launch uses default Authority configuration. |
|
||||
|
||||
## 4. Approvals & Distribution
|
||||
|
||||
- Record shared in `#launch-readiness` (Mattermost) 2025-10-26 15:15 UTC with DevOps + Guild leads for acknowledgement.
|
||||
- Updates to this document require dual sign-off from DevOps Guild (owner) and impacted module guild lead; retain change log via Git history.
|
||||
- Cutover rehearsal and rollback drills are tracked separately in `docs/modules/devops/runbooks/launch-cutover.md` (see associated Task `DEVOPS-LAUNCH-18-001`). *** End Patch
|
||||
|
||||
@@ -1,64 +1,64 @@
|
||||
# NuGet Preview Bootstrap (Offline-Friendly)
|
||||
|
||||
The StellaOps build relies on .NET 10 RC2 packages (Microsoft.Extensions.*, JwtBearer 10.0 RC).
|
||||
`NuGet.config` now wires three sources:
|
||||
|
||||
1. `local` → `./local-nuget` (preferred, air-gapped mirror)
|
||||
2. `dotnet-public` → `https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json`
|
||||
3. `nuget.org` → fallback for everything else
|
||||
|
||||
Follow the steps below whenever you refresh the repo or roll a new Offline Kit drop.
|
||||
|
||||
## 1. Mirror the preview packages
|
||||
|
||||
```bash
|
||||
./ops/devops/sync-preview-nuget.sh
|
||||
```
|
||||
|
||||
* Reads `ops/devops/nuget-preview-packages.csv`. Each line specifies the package, version, expected SHA-256 hash, and (optionally) the flat-container base URL (we pin to `dotnet-public`).
|
||||
* Downloads the `.nupkg` straight into `./local-nuget/` and re-verifies the checksum. Existing files are skipped when hashes already match.
|
||||
* Use `NUGET_V2_BASE` if you need to temporarily point at a different mirror.
|
||||
|
||||
💡 The script never mutates packages in place—if a checksum changes you will see a “SHA mismatch … refreshing” message.
|
||||
|
||||
## 2. Restore using the shared `NuGet.config`
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
DOTNET_NOLOGO=1 dotnet restore src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/StellaOps.Excititor.Connectors.Abstractions.csproj \
|
||||
--configfile NuGet.config
|
||||
```
|
||||
|
||||
The `packageSourceMapping` section keeps `Microsoft.Extensions.*`, `Microsoft.AspNetCore.*`, and `Microsoft.Data.Sqlite` bound to `local`/`dotnet-public`, so `dotnet restore` never has to reach out to nuget.org when mirrors are populated.
|
||||
|
||||
Before committing changes (or when wiring up a new environment) run:
|
||||
|
||||
```bash
|
||||
python3 ops/devops/validate_restore_sources.py
|
||||
```
|
||||
|
||||
The validator asserts:
|
||||
|
||||
- `NuGet.config` lists `local` → `dotnet-public` → `nuget.org` in that order.
|
||||
- `Directory.Build.props` pins `RestoreSources` so every project prioritises the local mirror.
|
||||
- No stray `NuGet.config` files shadow the repo root configuration.
|
||||
|
||||
CI executes the validator in both the `build-test-deploy` and `release` workflows,
|
||||
so regressions trip before any restore/build begins.
|
||||
|
||||
If you run fully air-gapped, remember to clear the cache between SDK upgrades:
|
||||
|
||||
```bash
|
||||
dotnet nuget locals all --clear
|
||||
```
|
||||
|
||||
## 3. Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
| --- | --- |
|
||||
| `dotnet restore` still hits nuget.org for preview packages | Re-run `sync-preview-nuget.sh` to ensure the `.nupkg` exists locally, then delete `~/.nuget/packages/microsoft.extensions.*` so the resolver picks up the mirrored copy. |
|
||||
| SHA mismatch in the manifest | Update `ops/devops/nuget-preview-packages.csv` with the new version + checksum (from the feed) and re-run the sync script. |
|
||||
| Azure DevOps feed throttling | Set `DOTNET_PUBLIC_FLAT_BASE` env var and point it at your own mirrored flat-container, then add the URL to the 4th column of the manifest. |
|
||||
|
||||
Keep this doc alongside Offline Kit instructions so air-gapped operators know exactly how to refresh the mirror and verify packages before restore.
|
||||
# NuGet Preview Bootstrap (Offline-Friendly)
|
||||
|
||||
The StellaOps build relies on .NET 10 RC2 packages (Microsoft.Extensions.*, JwtBearer 10.0 RC).
|
||||
`NuGet.config` now wires three sources:
|
||||
|
||||
1. `local` → `./local-nuget` (preferred, air-gapped mirror)
|
||||
2. `dotnet-public` → `https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json`
|
||||
3. `nuget.org` → fallback for everything else
|
||||
|
||||
Follow the steps below whenever you refresh the repo or roll a new Offline Kit drop.
|
||||
|
||||
## 1. Mirror the preview packages
|
||||
|
||||
```bash
|
||||
./ops/devops/sync-preview-nuget.sh
|
||||
```
|
||||
|
||||
* Reads `ops/devops/nuget-preview-packages.csv`. Each line specifies the package, version, expected SHA-256 hash, and (optionally) the flat-container base URL (we pin to `dotnet-public`).
|
||||
* Downloads the `.nupkg` straight into `./local-nuget/` and re-verifies the checksum. Existing files are skipped when hashes already match.
|
||||
* Use `NUGET_V2_BASE` if you need to temporarily point at a different mirror.
|
||||
|
||||
💡 The script never mutates packages in place—if a checksum changes you will see a “SHA mismatch … refreshing” message.
|
||||
|
||||
## 2. Restore using the shared `NuGet.config`
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
DOTNET_NOLOGO=1 dotnet restore src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Abstractions/StellaOps.Excititor.Connectors.Abstractions.csproj \
|
||||
--configfile NuGet.config
|
||||
```
|
||||
|
||||
The `packageSourceMapping` section keeps `Microsoft.Extensions.*`, `Microsoft.AspNetCore.*`, and `Microsoft.Data.Sqlite` bound to `local`/`dotnet-public`, so `dotnet restore` never has to reach out to nuget.org when mirrors are populated.
|
||||
|
||||
Before committing changes (or when wiring up a new environment) run:
|
||||
|
||||
```bash
|
||||
python3 ops/devops/validate_restore_sources.py
|
||||
```
|
||||
|
||||
The validator asserts:
|
||||
|
||||
- `NuGet.config` lists `local` → `dotnet-public` → `nuget.org` in that order.
|
||||
- `Directory.Build.props` pins `RestoreSources` so every project prioritises the local mirror.
|
||||
- No stray `NuGet.config` files shadow the repo root configuration.
|
||||
|
||||
CI executes the validator in both the `build-test-deploy` and `release` workflows,
|
||||
so regressions trip before any restore/build begins.
|
||||
|
||||
If you run fully air-gapped, remember to clear the cache between SDK upgrades:
|
||||
|
||||
```bash
|
||||
dotnet nuget locals all --clear
|
||||
```
|
||||
|
||||
## 3. Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
| --- | --- |
|
||||
| `dotnet restore` still hits nuget.org for preview packages | Re-run `sync-preview-nuget.sh` to ensure the `.nupkg` exists locally, then delete `~/.nuget/packages/microsoft.extensions.*` so the resolver picks up the mirrored copy. |
|
||||
| SHA mismatch in the manifest | Update `ops/devops/nuget-preview-packages.csv` with the new version + checksum (from the feed) and re-run the sync script. |
|
||||
| Azure DevOps feed throttling | Set `DOTNET_PUBLIC_FLAT_BASE` env var and point it at your own mirrored flat-container, then add the URL to the 4th column of the manifest. |
|
||||
|
||||
Keep this doc alongside Offline Kit instructions so air-gapped operators know exactly how to refresh the mirror and verify packages before restore.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Excititor agent guide
|
||||
|
||||
## Mission
|
||||
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# Excititor agent guide
|
||||
|
||||
## Mission
|
||||
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,33 +1,33 @@
|
||||
# StellaOps Excititor
|
||||
|
||||
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
|
||||
|
||||
## Responsibilities
|
||||
- Fetch OpenVEX/CSAF/CycloneDX statements via restart-only connectors.
|
||||
- Store immutable VEX observations with full provenance.
|
||||
- Publish linksets and events that drive policy suppression decisions.
|
||||
- Provide deterministic exports for Offline Kit and downstream tooling.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Excititor.WebService` scheduler/API host.
|
||||
- Connector libraries under `StellaOps.Excititor.Connector.*`.
|
||||
- Normalization helpers and exporters in `StellaOps.Excititor.*`.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Policy Engine for evidence queries.
|
||||
- UI/CLI for conflict visibility and explanation.
|
||||
- Notify for VEX-driven alerts.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for observation storage and job metadata.
|
||||
- Offline kit packaging aligned with Concelier merges.
|
||||
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-LNM-22-006 / DOCS-LNM-22-007 (shared with Concelier).
|
||||
- CLI-EXC-25-001..002 follow-up for CLI parity.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** maintain immutable VEX observations, provenance, and AOC verifier coverage.
|
||||
- **Epic 7 – VEX Consensus Lens:** supply trustworthy raw inputs, trust metadata, and consensus hooks for the lens computations.
|
||||
- **Epic 8 – Advisory AI:** expose citation-ready VEX payloads for the advisory assistant pipeline.
|
||||
# StellaOps Excititor
|
||||
|
||||
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
|
||||
|
||||
## Responsibilities
|
||||
- Fetch OpenVEX/CSAF/CycloneDX statements via restart-only connectors.
|
||||
- Store immutable VEX observations with full provenance.
|
||||
- Publish linksets and events that drive policy suppression decisions.
|
||||
- Provide deterministic exports for Offline Kit and downstream tooling.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Excititor.WebService` scheduler/API host.
|
||||
- Connector libraries under `StellaOps.Excititor.Connector.*`.
|
||||
- Normalization helpers and exporters in `StellaOps.Excititor.*`.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Policy Engine for evidence queries.
|
||||
- UI/CLI for conflict visibility and explanation.
|
||||
- Notify for VEX-driven alerts.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for observation storage and job metadata.
|
||||
- Offline kit packaging aligned with Concelier merges.
|
||||
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-LNM-22-006 / DOCS-LNM-22-007 (shared with Concelier).
|
||||
- CLI-EXC-25-001..002 follow-up for CLI parity.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** maintain immutable VEX observations, provenance, and AOC verifier coverage.
|
||||
- **Epic 7 – VEX Consensus Lens:** supply trustworthy raw inputs, trust metadata, and consensus hooks for the lens computations.
|
||||
- **Epic 8 – Advisory AI:** expose citation-ready VEX payloads for the advisory assistant pipeline.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Task board — Excititor
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| EXCITITOR-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| EXCITITOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| EXCITITOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
# Task board — Excititor
|
||||
|
||||
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
|
||||
|
||||
| ID | Status | Owner(s) | Description | Notes |
|
||||
|----|--------|----------|-------------|-------|
|
||||
| EXCITITOR-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
|
||||
| EXCITITOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| EXCITITOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,21 +1,21 @@
|
||||
# Implementation plan — Excititor
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** enforce immutable VEX observation schema, provenance capture, and guardrails.
|
||||
- **Epic 7 – VEX Consensus Lens:** provide lens-ready metadata (issuer trust, temporal scoping) and consensus APIs.
|
||||
- **Epic 8 – Advisory AI:** guarantee citation-ready payloads and normalized context for AI summaries/explainers.
|
||||
- Track DOCS-LNM-22-006/007 and CLI-EXC-25-001..002 in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
# Implementation plan — Excititor
|
||||
|
||||
## Current objectives
|
||||
- Maintain deterministic behaviour and offline parity across releases.
|
||||
- Keep documentation, telemetry, and runbooks aligned with the latest sprint outcomes.
|
||||
|
||||
## Workstreams
|
||||
- Backlog grooming: reconcile open stories in ../../TASKS.md with this module's roadmap.
|
||||
- Implementation: collaborate with service owners to land feature work defined in SPRINTS/EPIC docs.
|
||||
- Validation: extend tests/fixtures to preserve determinism and provenance requirements.
|
||||
|
||||
## Epic milestones
|
||||
- **Epic 1 – AOC enforcement:** enforce immutable VEX observation schema, provenance capture, and guardrails.
|
||||
- **Epic 7 – VEX Consensus Lens:** provide lens-ready metadata (issuer trust, temporal scoping) and consensus APIs.
|
||||
- **Epic 8 – Advisory AI:** guarantee citation-ready payloads and normalized context for AI summaries/explainers.
|
||||
- Track DOCS-LNM-22-006/007 and CLI-EXC-25-001..002 in ../../TASKS.md.
|
||||
|
||||
## Coordination
|
||||
- Review ./AGENTS.md before picking up new work.
|
||||
- Sync with cross-cutting teams noted in ../../implplan/SPRINTS.md.
|
||||
- Update this plan whenever scope, dependencies, or guardrails change.
|
||||
|
||||
@@ -1,164 +1,164 @@
|
||||
# architecture_excititor_mirrors.md — Excititor Mirror Distribution
|
||||
|
||||
> **Status:** Draft (Sprint 7). Complements `docs/modules/excititor/architecture.md` by describing the mirror export surface exposed by `Excititor.WebService` and the configuration hooks used by operators and downstream mirrors.
|
||||
|
||||
---
|
||||
|
||||
## 0) Purpose
|
||||
|
||||
Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:
|
||||
|
||||
* A declarative map of export bundles (`json`, `jsonl`, `openvex`, `csaf`) reachable via signed HTTP endpoints under `/excititor/mirror`.
|
||||
* Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
|
||||
* Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).
|
||||
|
||||
Mirror endpoints are intentionally **read-only**. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 1) Configuration model
|
||||
|
||||
The web service reads mirror configuration from `Excititor:Mirror` (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.
|
||||
|
||||
```yaml
|
||||
Excititor:
|
||||
Mirror:
|
||||
Domains:
|
||||
- id: primary
|
||||
displayName: Primary Mirror
|
||||
requireAuthentication: false
|
||||
maxIndexRequestsPerHour: 600
|
||||
maxDownloadRequestsPerHour: 1200
|
||||
exports:
|
||||
- key: consensus
|
||||
format: json
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
productKey: pkg:test/demo
|
||||
sort:
|
||||
createdAt: false # descending
|
||||
limit: 1000
|
||||
- key: consensus-openvex
|
||||
format: openvex
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
```
|
||||
|
||||
### Root settings
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `outputRoot` | – | Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted. |
|
||||
| `directoryName` | – | Optional subdirectory created under `outputRoot`; defaults to `mirror`. |
|
||||
| `targetRepository` | – | Hint propagated to manifests/index files indicating the operator-visible location (for example `s3://mirror/excititor`). |
|
||||
| `signing` | – | Bundle signing configuration. When enabled, the exporter emits a detached JWS (`bundle.json.jws`) alongside each domain bundle. |
|
||||
|
||||
`signing` supports the following fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `enabled` | – | Toggles detached signing for domain bundles. |
|
||||
| `algorithm` | – | Signing algorithm identifier (default `ES256`). |
|
||||
| `keyId` | ✅ (when `enabled`) | Signing key identifier resolved via the configured crypto provider registry. |
|
||||
| `provider` | – | Optional provider hint when multiple registries are available. |
|
||||
| `keyPath` | – | Optional PEM path used to seed the provider when the key is not already loaded. |
|
||||
|
||||
### Domain field reference
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `id` | ✅ | Stable identifier. Appears in URLs (`/excititor/mirror/domains/{id}`) and download filenames. |
|
||||
| `displayName` | – | Human-friendly label surfaced in the `/domains` listing. Falls back to `id`. |
|
||||
| `requireAuthentication` | – | When `true` the service enforces that the caller is authenticated (Authority token). |
|
||||
| `maxIndexRequestsPerHour` | – | Per-domain quota for index endpoints. `0`/negative disables the guard. |
|
||||
| `maxDownloadRequestsPerHour` | – | Per-domain quota for artifact downloads. |
|
||||
| `exports` | ✅ | Collection of export projections. |
|
||||
|
||||
Export-level fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `key` | ✅ | Unique key within the domain. Used in URLs (`/exports/{key}`) and filenames/bundle entries. |
|
||||
| `format` | ✅ | One of `json`, `jsonl`, `openvex`, `csaf`. Maps to `VexExportFormat`. |
|
||||
| `filters` | – | Key/value pairs executed via `VexQueryFilter`. Keys must match export data source columns (e.g., `vulnId`, `productKey`). |
|
||||
| `sort` | – | Key/boolean map (false = descending). |
|
||||
| `limit`, `offset`, `view` | – | Optional query bounds passed through to the export query. |
|
||||
|
||||
⚠️ **Misconfiguration:** invalid formats or missing keys cause exports to be flagged with `status` in the index response; they are not exposed downstream.
|
||||
|
||||
---
|
||||
|
||||
## 2) HTTP surface
|
||||
|
||||
Routes are grouped under `/excititor/mirror`.
|
||||
|
||||
| Method | Path | Description |
|
||||
| --- | --- | --- |
|
||||
| `GET` | `/domains` | Returns configured domains with quota metadata. |
|
||||
| `GET` | `/domains/{domainId}` | Domain detail (auth/quota + export keys). `404` for unknown domains. |
|
||||
| `GET` | `/domains/{domainId}/index` | Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}` | Returns manifest metadata (single export). `404` if unknown/missing. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}/download` | Streams export content from the artifact store. Applies download quota. |
|
||||
|
||||
Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable ordering. Download responses include a content-disposition header naming the file `<domain>-<export>.<ext>`.
|
||||
|
||||
### Error handling
|
||||
|
||||
* `401` – authentication required (`requireAuthentication=true`).
|
||||
* `404` – domain/export not found or manifest not persisted.
|
||||
* `429` – per-domain quota exceeded (`Retry-After` header set in seconds).
|
||||
* `503` – export misconfiguration (invalid format/query).
|
||||
|
||||
---
|
||||
|
||||
## 3) Rate limiting
|
||||
|
||||
`MirrorRateLimiter` implements a simple rolling 1-hour window using `IMemoryCache`. Each domain has two quotas:
|
||||
|
||||
* `index` scope → `maxIndexRequestsPerHour`
|
||||
* `download` scope → `maxDownloadRequestsPerHour`
|
||||
|
||||
`0` or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.
|
||||
|
||||
---
|
||||
|
||||
## 4) Interaction with export pipeline
|
||||
|
||||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||||
|
||||
Recommended workflow:
|
||||
|
||||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||||
2. Configure mirror domains mapping to those plans.
|
||||
3. Downstream mirror automation:
|
||||
* `GET /domains/{id}/index`
|
||||
* Compare `exportId` / `consensusRevision`
|
||||
* `GET /download` when new
|
||||
* Verify digest + attestation
|
||||
|
||||
When the export engine runs, it materializes the following artefacts under `outputRoot/<directoryName>`:
|
||||
|
||||
- `index.json` – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
|
||||
- `<domain>/manifest.json` – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
|
||||
- `<domain>/bundle.json` – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
|
||||
- `<domain>/bundle.json.jws` – optional detached JWS when signing is enabled.
|
||||
|
||||
Downstream automation reads `manifest.json`/`bundle.json` directly, while `/excititor/mirror` endpoints stream the same artefacts through authenticated HTTP.
|
||||
|
||||
---
|
||||
|
||||
## 5) Operational guidance
|
||||
|
||||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||||
* Ensure the underlying artifact stores (`FileSystem`, `S3`, offline bundle) retain artefacts long enough for mirrors to sync.
|
||||
* For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see `docs/24_OFFLINE_KIT.md`).
|
||||
|
||||
---
|
||||
|
||||
## 6) Future alignment
|
||||
|
||||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||||
* Integrate domain manifests with DevOps mirror profiles (`DEVOPS-MIRROR-08-001`) so helm/compose overlays can enable or disable domains declaratively.
|
||||
|
||||
# architecture_excititor_mirrors.md — Excititor Mirror Distribution
|
||||
|
||||
> **Status:** Draft (Sprint 7). Complements `docs/modules/excititor/architecture.md` by describing the mirror export surface exposed by `Excititor.WebService` and the configuration hooks used by operators and downstream mirrors.
|
||||
|
||||
---
|
||||
|
||||
## 0) Purpose
|
||||
|
||||
Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:
|
||||
|
||||
* A declarative map of export bundles (`json`, `jsonl`, `openvex`, `csaf`) reachable via signed HTTP endpoints under `/excititor/mirror`.
|
||||
* Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
|
||||
* Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).
|
||||
|
||||
Mirror endpoints are intentionally **read-only**. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 1) Configuration model
|
||||
|
||||
The web service reads mirror configuration from `Excititor:Mirror` (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.
|
||||
|
||||
```yaml
|
||||
Excititor:
|
||||
Mirror:
|
||||
Domains:
|
||||
- id: primary
|
||||
displayName: Primary Mirror
|
||||
requireAuthentication: false
|
||||
maxIndexRequestsPerHour: 600
|
||||
maxDownloadRequestsPerHour: 1200
|
||||
exports:
|
||||
- key: consensus
|
||||
format: json
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
productKey: pkg:test/demo
|
||||
sort:
|
||||
createdAt: false # descending
|
||||
limit: 1000
|
||||
- key: consensus-openvex
|
||||
format: openvex
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
```
|
||||
|
||||
### Root settings
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `outputRoot` | – | Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted. |
|
||||
| `directoryName` | – | Optional subdirectory created under `outputRoot`; defaults to `mirror`. |
|
||||
| `targetRepository` | – | Hint propagated to manifests/index files indicating the operator-visible location (for example `s3://mirror/excititor`). |
|
||||
| `signing` | – | Bundle signing configuration. When enabled, the exporter emits a detached JWS (`bundle.json.jws`) alongside each domain bundle. |
|
||||
|
||||
`signing` supports the following fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `enabled` | – | Toggles detached signing for domain bundles. |
|
||||
| `algorithm` | – | Signing algorithm identifier (default `ES256`). |
|
||||
| `keyId` | ✅ (when `enabled`) | Signing key identifier resolved via the configured crypto provider registry. |
|
||||
| `provider` | – | Optional provider hint when multiple registries are available. |
|
||||
| `keyPath` | – | Optional PEM path used to seed the provider when the key is not already loaded. |
|
||||
|
||||
### Domain field reference
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `id` | ✅ | Stable identifier. Appears in URLs (`/excititor/mirror/domains/{id}`) and download filenames. |
|
||||
| `displayName` | – | Human-friendly label surfaced in the `/domains` listing. Falls back to `id`. |
|
||||
| `requireAuthentication` | – | When `true` the service enforces that the caller is authenticated (Authority token). |
|
||||
| `maxIndexRequestsPerHour` | – | Per-domain quota for index endpoints. `0`/negative disables the guard. |
|
||||
| `maxDownloadRequestsPerHour` | – | Per-domain quota for artifact downloads. |
|
||||
| `exports` | ✅ | Collection of export projections. |
|
||||
|
||||
Export-level fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `key` | ✅ | Unique key within the domain. Used in URLs (`/exports/{key}`) and filenames/bundle entries. |
|
||||
| `format` | ✅ | One of `json`, `jsonl`, `openvex`, `csaf`. Maps to `VexExportFormat`. |
|
||||
| `filters` | – | Key/value pairs executed via `VexQueryFilter`. Keys must match export data source columns (e.g., `vulnId`, `productKey`). |
|
||||
| `sort` | – | Key/boolean map (false = descending). |
|
||||
| `limit`, `offset`, `view` | – | Optional query bounds passed through to the export query. |
|
||||
|
||||
⚠️ **Misconfiguration:** invalid formats or missing keys cause exports to be flagged with `status` in the index response; they are not exposed downstream.
|
||||
|
||||
---
|
||||
|
||||
## 2) HTTP surface
|
||||
|
||||
Routes are grouped under `/excititor/mirror`.
|
||||
|
||||
| Method | Path | Description |
|
||||
| --- | --- | --- |
|
||||
| `GET` | `/domains` | Returns configured domains with quota metadata. |
|
||||
| `GET` | `/domains/{domainId}` | Domain detail (auth/quota + export keys). `404` for unknown domains. |
|
||||
| `GET` | `/domains/{domainId}/index` | Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}` | Returns manifest metadata (single export). `404` if unknown/missing. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}/download` | Streams export content from the artifact store. Applies download quota. |
|
||||
|
||||
Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable ordering. Download responses include a content-disposition header naming the file `<domain>-<export>.<ext>`.
|
||||
|
||||
### Error handling
|
||||
|
||||
* `401` – authentication required (`requireAuthentication=true`).
|
||||
* `404` – domain/export not found or manifest not persisted.
|
||||
* `429` – per-domain quota exceeded (`Retry-After` header set in seconds).
|
||||
* `503` – export misconfiguration (invalid format/query).
|
||||
|
||||
---
|
||||
|
||||
## 3) Rate limiting
|
||||
|
||||
`MirrorRateLimiter` implements a simple rolling 1-hour window using `IMemoryCache`. Each domain has two quotas:
|
||||
|
||||
* `index` scope → `maxIndexRequestsPerHour`
|
||||
* `download` scope → `maxDownloadRequestsPerHour`
|
||||
|
||||
`0` or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.
|
||||
|
||||
---
|
||||
|
||||
## 4) Interaction with export pipeline
|
||||
|
||||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||||
|
||||
Recommended workflow:
|
||||
|
||||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||||
2. Configure mirror domains mapping to those plans.
|
||||
3. Downstream mirror automation:
|
||||
* `GET /domains/{id}/index`
|
||||
* Compare `exportId` / `consensusRevision`
|
||||
* `GET /download` when new
|
||||
* Verify digest + attestation
|
||||
|
||||
When the export engine runs, it materializes the following artefacts under `outputRoot/<directoryName>`:
|
||||
|
||||
- `index.json` – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
|
||||
- `<domain>/manifest.json` – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
|
||||
- `<domain>/bundle.json` – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
|
||||
- `<domain>/bundle.json.jws` – optional detached JWS when signing is enabled.
|
||||
|
||||
Downstream automation reads `manifest.json`/`bundle.json` directly, while `/excititor/mirror` endpoints stream the same artefacts through authenticated HTTP.
|
||||
|
||||
---
|
||||
|
||||
## 5) Operational guidance
|
||||
|
||||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||||
* Ensure the underlying artifact stores (`FileSystem`, `S3`, offline bundle) retain artefacts long enough for mirrors to sync.
|
||||
* For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see `docs/24_OFFLINE_KIT.md`).
|
||||
|
||||
---
|
||||
|
||||
## 6) Future alignment
|
||||
|
||||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||||
* Integrate domain manifests with DevOps mirror profiles (`DEVOPS-MIRROR-08-001`) so helm/compose overlays can enable or disable domains declaratively.
|
||||
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
# Export Center agent guide
|
||||
|
||||
## Mission
|
||||
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
# Export Center agent guide
|
||||
|
||||
## Mission
|
||||
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
@@ -1,34 +1,34 @@
|
||||
# StellaOps Export Center
|
||||
|
||||
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
|
||||
|
||||
## Responsibilities
|
||||
- Coordinate export jobs based on profiles and scope selectors.
|
||||
- Assemble manifests, provenance documents, and cosign signatures.
|
||||
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
|
||||
- Expose CLI/API surfaces for automation.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.ExportCenter.WebService` planner.
|
||||
- `StellaOps.ExportCenter.Worker` bundle builder.
|
||||
- Adapters in `StellaOps.ExportCenter.*` for JSON/Trivy/mirror variants.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Concelier/Excititor/Policy data stores for evidence.
|
||||
- Signer/Attestor for provenance signing.
|
||||
- CLI for operator-managed exports.
|
||||
|
||||
## Operational notes
|
||||
- Runbooks in ./operations/ for deployment and monitoring.
|
||||
- Mirror bundle instructions and validation notes.
|
||||
- Telemetry dashboards for export latency and retry rates.
|
||||
|
||||
## Related resources
|
||||
- ./operations/runbook.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-EXPORT-35-001 … DOCS-EXPORT-37-002 in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 cross-team deliverable.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 10 – Export Center:** deliver canonical JSON, Trivy DB, and mirror bundle workflows with provenance, signatures, and offline parity.
|
||||
# StellaOps Export Center
|
||||
|
||||
Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) with provenance metadata and optional signing for offline or mirrored deployments.
|
||||
|
||||
## Responsibilities
|
||||
- Coordinate export jobs based on profiles and scope selectors.
|
||||
- Assemble manifests, provenance documents, and cosign signatures.
|
||||
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
|
||||
- Expose CLI/API surfaces for automation.
|
||||
|
||||
## Key components
|
||||
- `StellaOps.ExportCenter.WebService` planner.
|
||||
- `StellaOps.ExportCenter.Worker` bundle builder.
|
||||
- Adapters in `StellaOps.ExportCenter.*` for JSON/Trivy/mirror variants.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Concelier/Excititor/Policy data stores for evidence.
|
||||
- Signer/Attestor for provenance signing.
|
||||
- CLI for operator-managed exports.
|
||||
|
||||
## Operational notes
|
||||
- Runbooks in ./operations/ for deployment and monitoring.
|
||||
- Mirror bundle instructions and validation notes.
|
||||
- Telemetry dashboards for export latency and retry rates.
|
||||
|
||||
## Related resources
|
||||
- ./operations/runbook.md
|
||||
|
||||
## Backlog references
|
||||
- DOCS-EXPORT-35-001 … DOCS-EXPORT-37-002 in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 cross-team deliverable.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 10 – Export Center:** deliver canonical JSON, Trivy DB, and mirror bundle workflows with provenance, signatures, and offline parity.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user