stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

Browse Source 
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes.
- Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes.
- Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables.
- Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
This commit is contained in:
master
2025-10-30 00:09:39 +02:00
parent 3154c67978
commit 7b5bdcf4d3
503 changed files with 16136 additions and 54638 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
docs/implplan/AGENTS.md
View File

@@ -1,112 +0,0 @@
# 1) What is StellaOps?
**StellaOps** an open, sovereign, modular container-security toolkit built for high-speed, offline operation, released under AGPL-3.0-or-later.
It follows an SBOM-first model—analyzing each container layer or ingesting existing CycloneDX/SPDX SBOMs, then enriching them with vulnerability, licence, secret-leak, and misconfiguration data to produce cryptographically signed reports.
Vulnerability detection maps OS and language dependencies to sources such as NVD, GHSA, OSV, ENISA.
Secrets sweep flags exposed credentials or keys in files or environment variables.
Licence audit identifies potential conflicts, especially copyleft obligations.
Misconfiguration checks detect unsafe Dockerfile patterns (root user, latest tags, permissive modes).
Provenance features include in-toto/SLSA attestations signed with cosign for supply-chain trust.
| Guiding principle | What it means for Concelier |
|-------------------|---------------------------|
| **SBOM-first ingest** | Prefer signed SBOMs or reproducible layer diffs before falling back to raw scraping; connectors treat source docs as provenance, never as mutable truth. |
| **Deterministic outputs** | Same inputs yield identical canonical advisories and exported JSON/Trivy DB artefacts; merge hashes and export manifests are reproducible across machines. |
| **Restart-time plug-ins only** | Connector/exporter plug-ins load at service start, keeping runtime sandboxing simple and avoiding hot-patch attack surface. |
| **Sovereign/offline-first** | No mandatory outbound calls beyond allow-listed advisories; Offline Kit bundles Mongo snapshots and exporter artefacts for air-gapped installs. |
| **Operational transparency** | Every stage logs structured events (fetch, parse, merge, export) with correlation IDs so parallel agents can debug without shared state. |
Performance: warm scans < 5 s, cold scans < 30 s on a 4 vCPU runner.
Deployment: entirely SaaS-free, suitable for air-gapped or on-prem use through its Offline Kit.
Policy: anonymous users → 33 scans/day; verified → 333 /day; nearing 90 % quota triggers throttling but never full blocks.
More documention is available ./docs/*.md files. Read `docs/README.md` to gather information about the available documentation. You could inquiry specific documents as your work requires it
---
# 3) Practices
## 3.1) Naming
All modules are .NET projects based on .NET 10 (preview). Exclussion is the UI. It is based on Angular
All modules are contained by one or more projects. Each project goes in its dedicated folder. Each project starts with StellaOps.<ModuleName>. In case it is common for for all StellaOps modules it is library or plugin and it is named StellaOps.<LibraryOrPlugin>.
## 3.2) Key technologies & integrations
- **Runtime**: .NET 10 (`net10.0`) preview SDK; C# latest preview features.
- **Data**: MongoDB (canonical store and job/export state).
- **Observability**: structured logs, counters, and (optional) OpenTelemetry traces.
- **Ops posture**: offline‑first, allowlist for remote hosts, strict schema validation, gated LLM fallback (only where explicitly configured).
# 4) Modules
StellaOps is contained by different modules installable via docker containers
- Concelier. Responsible for aggregation and delivery of vulnerability database
- Cli. Command line tool to unlock full potential - request database operations, install scanner, request scan, configure backend
- Backend. Configures and Manages scans
- UI. UI to access the backend (and scanners)
- Agent. Installable daemon that does the scanning
- Zastava. Realtime monitor for allowed (verified) installations.
For more information of the architecture see `./docs/*ARCHITECTURE*.md` files.
---
### 4.1.4) Glossary (quick)
- **OVAL** — Vendor/distro security definition format; authoritative for OS packages.
- **NEVRA / EVR** — RPM and Debian version semantics for OS packages.
- **PURL / SemVer** — Coordinates and version semantics for OSS ecosystems.
- **KEV** — Known Exploited Vulnerabilities (flag only).
---
# 5) Your role as StellaOps contributor
You acting as information technology engineer that will take different type of roles in goal achieving StellaOps production implementation
In order you to work - you have to be supplied with directory that contains `AGENTS.md`,`TASKS.md` files. There will you have more information about the role you have, the scope of your work and the tasks you will have.
Boundaries:
- You operate only in the working directories I gave you, unless there is dependencies that makes you to work on dependency in shared directory. Then you ask for confirmation.
You main characteristics:
- Keep endpoints small, deterministic, and cancellation-aware.
- Improve logs/metrics as per tasks.
- Update `TASKS.md` when moving tasks forward.
- When you are done with all task you state explicitly you are done.
- Impersonate the role described on working directory `AGENTS.md` you will read, if role is not available - take role of the CTO of the StellaOps in early stages.
- You always strive for best practices
- You always strive for re-usability
- When in doubt of design decision - you ask then act
- You are autonomus - meaning that you will work for long time alone and achieve maximum without stopping for stupid questions
- You operate on the same directory where other agents will work. In case you need to work on directory that is dependency on provided `AGENTS.md`,`TASKS.md` files you have to ask for confirmation first.
## 5.1) Type of contributions
- **BE‑Base (Platform & Pipeline)**
Owns DI, plugin host, job scheduler/coordinator, configuration binding, minimal API endpoints, and Mongo bootstrapping.
- **BE‑Conn‑X (Connectors)**
One agent per source family (NVD, Red Hat, Ubuntu, Debian, SUSE, GHSA, OSV, PSIRTs, CERTs, KEV, ICS). Implements fetch/parse/map with incremental watermarks.
- **BE‑Merge (Canonical Merge & Dedupe)**
Identity graph, precedence policies, canonical JSON serializer, and deterministic hashing (`merge_event`).
- **BE‑Export (JSON & Trivy DB)**
Deterministic export trees, Trivy DB packaging, optional ORAS push, and offline bundle.
- **QA (Validation & Observability)**
Schema tests, fixture goldens, determinism checks, metrics/logs/traces, e2e reproducibility runs.
- **DevEx/Docs**
Maintains this agent framework, templates, and per‑directory guides; assists parallelization and reviews.
## 5.2) Work rules (important)
- **Directory ownership**: Each agent works **only inside its module directory**. Cross‑module edits require a brief handshake in issues/PR description.
- **Scoping**: Use each module’s `AGENTS.md` and `TASKS.md` to plan; autonomous agents must read `src/AGENTS.md` and the module docs before acting.
- **Determinism**: Sort keys, normalize timestamps to UTC ISO‑8601, avoid non‑deterministic data in exports and tests.
- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md and in ./SPRINTS.md file.
- **Coordination**: In case task is discovered as blocked on other team or task, according TASKS.md files that dependency is on needs to be changed by adding new tasks describing the requirement. the current task must be updated as completed. In case task changes, scope or requirements or rules - other documentations needs be updated accordingly.
- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and the relevant TASKS.md file. Always check the AGENTS.md in the relevant TASKS.md directory.
- **Tests**: Add/extend fixtures and unit tests per change; never regress determinism or precedence.
- **Test layout**: Use module-specific projects in `StellaOps.Concelier.<Component>.Tests`; shared fixtures/harnesses live in `StellaOps.Concelier.Testing`.
- **Execution autonomous**: In case you need to continue with more than one options just continue sequentially, unless the continue requires design decision.
- **EPIC reference**: In case task (on relevant TASKS.md file) references in direct or indirect way an EPIC - then seek for epic document like ./EPIC_*.md. There will be more information how to implement the task.
---

524
docs/implplan/EPIC_1.md
View File

@@ -1,524 +0,0 @@
Here’s the full write‑up you can drop into your repo as the canonical reference for Epic 1. It’s written in clean product‑doc style so it’s safe to check in as Markdown. No fluff, just everything you need to build, test, and police it.
---
# Epic 1: Aggregation‑Only Contract (AOC) Enforcement
> Short name: **AOC enforcement**
> Services touched: **Conseiller (advisory ingestion), Excitator (VEX ingestion), Web API, Workers, Policy Engine, CLI, Console, Authority**
> Data stores: **MongoDB primary, optional Redis/NATS for jobs**
---
## 1) What it is
**Aggregation‑Only Contract (AOC)** is the ingestion covenant for StellaOps. It defines a hard boundary between **collection** and **interpretation**:
* **Ingestion (Conseiller/Excitator)** only **collects** data and preserves it as immutable raw facts with provenance. It does not decide, merge, normalize, prioritize, or assign severity. It may compute **links** that help future joins (aliases, PURLs, CPEs), but never derived judgments.
* **Policy evaluation** is the only place where merges, deduplication, consensus, severity computation, and status folding are allowed. It’s reproducible and traceable.
The AOC establishes:
* **Immutable raw stores**: `advisory_raw` and `vex_raw` documents with full provenance, signatures, checksums, and upstream identifiers.
* **Linksets**: machine‑generated join hints (aliases, PURLs, CPEs, CVE/GHSA IDs) that never change the underlying source content.
* **Invariants**: a strict set of “never do this in ingestion” rules enforced by schema validation, runtime guards, and CI checks.
* **AOC Verifier**: a build‑time and runtime watchdog that blocks non‑compliant code and data writes.
This epic delivers: schemas, guards, error codes, APIs, tests, migration, docs, and ops dashboards to make AOC non‑negotiable across the platform.
---
## 2) Why
AOC makes results **auditable, deterministic, and organization‑specific**. Source vendors disagree; your policies decide. By removing hidden heuristics from ingestion, we avoid unexplainable risk changes, race conditions between collectors, and vendor bias. Policy‑time evaluation yields reproducible deltas with complete “why” traces.
---
## 3) How it should work (deep details)
### 3.1 Core invariants
The following must be true for every write to `advisory_raw` and `vex_raw` and for every ingestion pipeline:
1. **No severity in ingestion**
* Forbidden fields: `severity`, `cvss`, `cvss_vector`, `effective_status`, `effective_range`, `merged_from`, `consensus_provider`, `reachability`, `asset_criticality`, `risk_score`.
2. **No merges or de‑dups in ingestion**
* No combining two upstream advisories into one. No picking a single truth when multiple VEX statements exist.
3. **Provenance is mandatory**
* Every raw doc includes `provenance` and `signature/checksum`.
4. **Idempotent upserts**
* Same upstream document (by `upstream_id` + `source` + `content_hash`) must not create duplicates.
5. **Append‑only versioning**
* Revisions from the source create new immutable documents with `supersedes` pointers; no in‑place edits.
6. **Linkset only**
* Ingestion can compute and store a `linkset` for join performance. Linkset does not alter or infer severity/status.
7. **Policy‑time only for effective findings**
* Only the Policy Engine can write `effective_finding_*` materializations.
8. **Schema safety**
* Strict JSON schema validation at DB level; unknown fields reject writes.
9. **Clock discipline**
* Timestamps are UTC, monotonic within a batch; collectors record `fetched_at` and `received_at`.
### 3.2 Data model
#### 3.2.1 `advisory_raw` (Mongo collection)
```json
{
"_id": "advisory_raw:osv:GHSA-xxxx-....:v3",
"source": {
"vendor": "OSV",
"stream": "github",
"api": "https://api.osv.dev/v1/.../GHSA-...",
"collector_version": "conseiller/1.7.3"
},
"upstream": {
"upstream_id": "GHSA-xxxx-....",
"document_version": "2024-09-01T12:13:14Z",
"fetched_at": "2025-01-02T03:04:05Z",
"received_at": "2025-01-02T03:04:06Z",
"content_hash": "sha256:...",
"signature": {
"present": true,
"format": "dsse",
"key_id": "rekor:.../key/abc",
"sig": "base64..."
}
},
"content": {
"format": "OSV",
"spec_version": "1.6",
"raw": { /* full upstream JSON, unmodified */ }
},
"identifiers": {
"cve": ["CVE-2023-1234"],
"ghsa": ["GHSA-xxxx-...."],
"aliases": ["CVE-2023-1234", "GHSA-xxxx-...."]
},
"linkset": {
"purls": ["pkg:npm/lodash@4.17.21", "pkg:maven/..."],
"cpes": ["cpe:2.3:a:..."],
"references": [
{"type":"advisory","url":"https://..."},
{"type":"fix","url":"https://..."}
],
"reconciled_from": ["content.raw.affected.ranges", "content.raw.pkg"]
},
"supersedes": "advisory_raw:osv:GHSA-xxxx-....:v2",
"tenant": "default"
}
```
> Note: No `severity`, no `cvss`, no `effective_*`. If the upstream payload includes CVSS, it stays inside `content.raw` and is not promoted or normalized at ingestion.
#### 3.2.2 `vex_raw` (Mongo collection)
```json
{
"_id": "vex_raw:vendorX:doc-123:v4",
"source": {
"vendor": "VendorX",
"stream": "vex",
"api": "https://.../vex/doc-123",
"collector_version": "excitator/0.9.2"
},
"upstream": {
"upstream_id": "doc-123",
"document_version": "2025-01-15T08:09:10Z",
"fetched_at": "2025-01-16T01:02:03Z",
"received_at": "2025-01-16T01:02:03Z",
"content_hash": "sha256:...",
"signature": { "present": true, "format": "cms", "key_id": "kid:...", "sig": "..." }
},
"content": {
"format": "CycloneDX-VEX", // or "CSAF-VEX"
"spec_version": "1.5",
"raw": { /* full upstream VEX */ }
},
"identifiers": {
"statements": [
{
"advisory_ids": ["CVE-2023-1234","GHSA-..."],
"component_purls": ["pkg:deb/openssl@1.1.1"],
"status": "not_affected",
"justification": "component_not_present"
}
]
},
"linkset": {
"purls": ["pkg:deb/openssl@1.1.1"],
"cves": ["CVE-2023-1234"],
"ghsas": ["GHSA-..."]
},
"supersedes": "vex_raw:vendorX:doc-123:v3",
"tenant": "default"
}
```
> VEX statuses remain as raw facts. No cross‑provider consensus is computed here.
### 3.3 Database validation
* MongoDB JSON Schema validators on both collections:
* Reject forbidden fields at the top level.
* Enforce presence of `source`, `upstream`, `content`, `linkset`, `tenant`.
* Enforce string formats for timestamps and hashes.
### 3.4 Write paths
1. **Collector fetches upstream**
* Normalize transport (gzip/json), compute `content_hash`, verify signature if available.
2. **Build raw doc**
* Populate `source`, `upstream`, `content.raw`, `identifiers`, `linkset`.
3. **Idempotent upsert**
* Lookup by `(source.vendor, upstream.upstream_id, upstream.content_hash)`. If exists, skip; if new content hash, insert new revision with `supersedes`.
4. **AOC guard**
* Runtime interceptor inspects write payload; if any forbidden field detected, reject with `ERR_AOC_001`.
5. **Metrics**
* Emit `ingestion_write_ok` or `ingestion_write_reject` with reason code.
### 3.5 Read paths (ingestion scope)
* Allow only listing, getting raw docs, and searching by linkset. No endpoints return “effective findings” from ingestion services.
### 3.6 Error codes
| Code | Meaning | HTTP |
| ------------- | ------------------------------------------------------------ | ---- |
| `ERR_AOC_001` | Forbidden field present (severity/consensus/normalized data) | 400 |
| `ERR_AOC_002` | Merge attempt detected (multiple upstreams fused) | 400 |
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes) | 409 |
| `ERR_AOC_004` | Missing provenance fields | 422 |
| `ERR_AOC_005` | Signature/checksum mismatch | 422 |
| `ERR_AOC_006` | Attempt to write effective findings from ingestion context | 403 |
| `ERR_AOC_007` | Unknown top‑level fields (schema violation) | 400 |
### 3.7 AOC Verifier
A build‑time and runtime safeguard:
* **Static checks (CI)**
* Block imports of `*.Policy*` or `*.Merge*` from ingestion modules.
* AST lint rule: any write to `advisory_raw` or `vex_raw` setting a forbidden key fails the build.
* **Runtime checks**
* Repository layer interceptor inspects documents before insert/update; rejects forbidden fields and multi‑source merges.
* **Drift detection job**
* Nightly job scans newest N docs; if violation found, pages ops and blocks new pipeline runs.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.8 Indexing strategy
* `advisory_raw`:
* `{ "identifiers.cve": 1 }`, `{ "identifiers.ghsa": 1 }`, `{ "linkset.purls": 1 }`, `{ "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 }` (unique), `{ "tenant": 1 }`.
* `vex_raw`:
* `{ "identifiers.statements.advisory_ids": 1 }`, `{ "linkset.purls": 1 }`, `{ "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 }` (unique), `{ "tenant": 1 }`.
### 3.9 Interaction with Policy Engine
* Policy Engine pulls raw docs by identifiers/linksets and computes:
* De‑dup/merge per policy
* Consensus for VEX statements
* Severity normalization and risk scoring
* Writes **only** to `effective_finding_{policyId}` collections.
A dedicated write guard refuses `effective_finding_*` writes from any caller that isn’t the Policy Engine service identity.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.10 Migration plan
1. **Freeze ingestion writes** except raw pass‑through.
2. **Backfill**: copy existing ingestion collections to `_backup_*`.
3. **Strip forbidden fields** from raw copies, move them into a temporary `advisory_view_legacy` used only by Policy Engine for parity.
4. **Enable DB schema validators**.
5. **Run collectors** in dry‑run; ensure only allowed keys land.
6. **Switch Policy Engine** to pull exclusively from `*_raw` and to compute everything else.
7. **Delete legacy normalized fields** in ingestion codepaths.
8. **Enable runtime guards** and CI lint.
### 3.11 Observability
* Metrics:
* `aoc_violation_total{code=...}`, `ingestion_write_total{result=ok|reject}`, `ingestion_signature_verified_total{result=ok|fail}`, `ingestion_latency_seconds`, `advisory_revision_count`.
* Tracing: span `ingest.fetch`, `ingest.transform`, `ingest.write`, `aoc.guard`.
* Logs: include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, `correlation_id`.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.12 Security and tenancy
* Every raw doc carries a `tenant` field.
* Authority enforces `advisory:ingest` and `vex:ingest` scopes for ingestion endpoints.
* Cross‑tenant reads/writes are blocked by default.
* Secrets never logged; signatures verified with pinned trust stores.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.13 CLI and Console behavior
* **CLI**
* `stella sources ingest --dry-run` prints would‑write payload and explicitly shows that no severity/status fields are present.
* `stella aoc verify` scans last K documents and reports violations with exit codes.
* **Console**
* Sources dashboard shows AOC pass/fail per job, most recent violation codes, and a drill‑down to the offending document.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 4) API surface (ingestion scope)
### 4.1 Conseiller (Advisories)
* `POST /ingest/advisory`
* Body: raw upstream advisory with metadata; server constructs document, not the client.
* Rejections: `ERR_AOC_00x` per table above.
* `GET /advisories/raw/{id}`
* `GET /advisories/raw?cve=CVE-...&purl=pkg:...&tenant=...`
* `GET /advisories/raw/{id}/provenance`
* `POST /aoc/verify?since=ISO8601` returns summary stats and first N violations.
### 4.2 Excitator (VEX)
* `POST /ingest/vex`
* `GET /vex/raw/{id}`
* `GET /vex/raw?advisory_id=CVE-...&purl=pkg:...`
* `POST /aoc/verify?since=ISO8601`
All endpoints require `tenant` scope and appropriate `:write` or `:read`.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 5) Example: end‑to‑end flow
1. Collector fetches `GHSA-1234` from OSV.
2. Build `advisory_raw` with linkset PURLs.
3. Insert; AOC guard approves.
4. Policy Engine later evaluates SBOM `S-42` under `policy P-7`, reads raw advisory and any VEX raw docs, computes effective findings, and writes to `effective_finding_P-7`.
5. CLI `stella aoc verify --since 24h` returns `0` violations.
---
## 6) Implementation tasks
Breakdown by component with exact work items. Each section ends with the imposed sentence you requested.
### 6.1 Conseiller (advisory ingestion, WS + Worker)
* [ ] Add Mongo JSON schema validation for `advisory_raw`.
* [ ] Implement repository layer with **write interceptors** that reject forbidden fields.
* [ ] Compute `linkset` from upstream using deterministic mappers.
* [ ] Enforce idempotency by unique index on `(source.vendor, upstream.upstream_id, upstream.content_hash, tenant)`.
* [ ] Remove any normalization pipelines; relocate to Policy Engine.
* [ ] Add `POST /ingest/advisory` and `GET /advisories/raw*` endpoints with Authority scope checks.
* [ ] Emit observability metrics and traces.
* [ ] Unit tests: schema violations, idempotency, supersedes chain, forbidden fields.
* [ ] Integration tests: large batch ingest, linkset correctness against golden fixtures.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.2 Excitator (VEX ingestion, WS + Worker)
* [ ] Add Mongo JSON schema validation for `vex_raw`.
* [ ] Implement repository layer guard identical to Conseiller.
* [ ] Deterministic `linkset` extraction for advisory IDs and PURLs.
* [ ] Endpoints `POST /ingest/vex`, `GET /vex/raw*` with scopes.
* [ ] Remove any consensus or folding logic; leave VEX statements as raw.
* [ ] Tests as per Conseiller, with rich fixtures for CycloneDX‑VEX and CSAF.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.3 Web API shared library
* [ ] Define `AOCForbiddenKeys` and export for both services.
* [ ] Provide `AOCWriteGuard` middleware and `AOCError` types.
* [ ] Provide `ProvenanceBuilder` utility.
* [ ] Provide `SignatureVerifier` and `Checksum` helpers.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.4 Policy Engine
* [ ] Block any import/use from ingestion modules by lint rule.
* [ ] Add hard gate on `effective_finding_*` writes that verifies caller identity is Policy Engine.
* [ ] Update readers to pull fields only from `content.raw`, `identifiers`, `linkset`, not any legacy normalized fields.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.5 Authority
* [ ] Introduce scopes: `advisory:ingest`, `advisory:read`, `vex:ingest`, `vex:read`, `aoc:verify`.
* [ ] Add `tenant` claim propagation to ingestion services.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.6 CLI
* [ ] `stella sources ingest --dry-run` and `stella aoc verify` commands.
* [ ] Exit codes mapping to `ERR_AOC_00x`.
* [ ] JSON output schema including violation list.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.7 Console
* [ ] Sources dashboard tiles: last run, AOC violations, top error codes.
* [ ] Drill‑down page rendering offending doc with highlight on forbidden keys.
* [ ] “Verify last 24h” action calling the AOC Verifier endpoint.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.8 CI/CD
* [ ] AST linter to forbid writes of banned keys in ingestion modules.
* [ ] Unit test coverage gates for AOC guard code.
* [ ] Pipeline stage that runs `stella aoc verify` against seeded DB snapshots.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Documentation changes (create/update these files)
1. **`/docs/ingestion/aggregation-only-contract.md`**
* Add: philosophy, invariants, schemas for `advisory_raw`/`vex_raw`, error codes, linkset definition, examples, idempotency rules, supersedes, API references, migration steps, observability, security.
2. **`/docs/architecture/overview.md`**
* Update system diagram to show AOC boundary and raw stores; add sequence diagram: fetch → guard → raw insert → policy evaluation.
3. **`/docs/architecture/policy-engine.md`**
* Clarify ingestion boundary; list inputs consumed from raw; note that any severity/consensus is policy‑time only.
4. **`/docs/ui/console.md`**
* Add Sources dashboard section: AOC tiles and violation drill‑down.
5. **`/docs/cli/cli-reference.md`**
* Add `stella aoc verify` and `stella sources ingest --dry-run` usage and exit codes.
6. **`/docs/observability/observability.md`**
* Document new metrics, traces, logs keys for AOC.
7. **`/docs/security/authority-scopes.md`**
* Add new scopes and tenancy enforcement for ingestion endpoints.
8. **`/docs/deploy/containers.md`**
* Note DB validators must be enabled; environment flags for AOC guards; read‑only user for verify endpoint.
Each file should include a “Compliance checklist” subsection for AOC.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 8) Acceptance criteria
* DB validators are active and reject writes with forbidden fields.
* AOC runtime guards log and reject violations with correct error codes.
* CI linter prevents shipping code that writes forbidden keys to raw stores.
* Ingestion of known fixture sets results in zero normalized fields outside `content.raw`.
* Policy Engine is the only writer of `effective_finding_*` materializations.
* CLI `stella aoc verify` returns success on clean datasets and non‑zero on seeded violations.
* Console shows AOC status and violation drill‑downs.
---
## 9) Risks and mitigations
* **Collector drift**: new upstream fields tempt developers to normalize.
* Mitigation: CI linter + guard + schema validators; require RFC to extend linkset.
* **Performance impact**: extra validation on write.
* Mitigation: guard is O(number of keys) and schema check is bounded; indexes sized appropriately.
* **Migration complexity**: moving legacy normalized fields out.
* Mitigation: temporary `advisory_view_legacy` for parity; stepwise cutover.
* **Tenant leakage**: missing tenant on write.
* Mitigation: schema requires `tenant`; middleware injects and asserts.
---
## 10) Test plan
* **Unit tests**
* Guard rejects forbidden keys; idempotency; supersedes chain; provenance required.
* Signature verification paths: good, bad, absent.
* **Property tests**
* Randomized upstream docs never produce forbidden keys at top level.
* **Integration tests**
* Batch ingest of 50k advisories: throughput, zero violations.
* Mixed VEX sources with contradictory statements remain separate in raw.
* **Contract tests**
* Policy Engine refuses to run without raw inputs; writes only to `effective_finding_*`.
* **End‑to‑end**
* Seed SBOM + advisories + VEX; ensure findings are identical pre/post migration.
---
## 11) Developer checklists
**Definition of Ready**
* Upstream spec reference attached.
* Linkset mappers defined.
* Example fixtures added.
**Definition of Done**
* DB validators deployed and tested.
* Runtime guards enabled.
* CI linter merged and enforced.
* Docs updated (files in section 7).
* Metrics visible on dashboard.
* CLI verify passes.
---
## 12) Glossary
* **Raw document**: exact upstream content plus provenance, with join hints.
* **Linkset**: PURLs/CPEs/IDs extracted to accelerate joins later.
* **Supersedes**: pointer from a newer raw doc to the previous revision of the same upstream doc.
* **Policy‑time**: evaluation phase where merges, consensus, and severity are computed.
* **AOC**: Aggregation‑Only Contract.
---
### Final imposed reminder
**Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

566
docs/implplan/EPIC_10.md
View File

@@ -1,566 +0,0 @@
Fine. Here’s your next brick of “maximum documentation.” Try not to drop it on your foot.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
# Epic 10: Export Center (JSON, Trivy DB, Mirror bundles)
**Short name:** `Export Center`
**Primary service:** `exporter`
**Surfaces:** Console (Web UI), CLI, Web API
**Touches:** Conseiller (Feedser), Excitator (Vexer), SBOM Service, Policy Engine, VEX Consensus Lens, Findings Ledger, Authority (authN/Z), Object Storage, Orchestrator, Signing/Attestation, Telemetry
**AOC ground rule:** Conseiller and Excitator aggregate but never merge. The Export Center serializes evidence and policy results; it does not rewrite or “improve” your data in-flight.
---
## 1) What it is
The Export Center is the unified system for packaging StellaOps data into portable, verifiable bundles:
* **Canonical JSON exports** for advisories, VEX, SBOMs, findings, and policy-evaluation snapshots.
* **Trivy DB compatible bundles** so downstream scanners can use Stella’s curated vulnerability knowledge without custom integrations.
* **Mirror bundles** for air‑gapped or disconnected environments containing raw evidence, normalized records, indexes, and policy snapshots, with provenance, signatures, and optional encryption.
It centralizes format adapters, compliance and provenance, scheduling, versioning, and distribution (download, OCI push, file share). Every export is reproducible by `run_id`, cryptographically signed, and audit‑traceable back to source artifacts.
---
## 2) Why (brief)
Teams need to move results into other scanners, CI systems, and isolated networks without babysitting ten different scripts. The Export Center gives a single, policy‑aware, verifiable exit point that doesn’t surprise compliance or set your ops team on fire.
---
## 3) How it should work (maximum detail)
### 3.1 Capabilities
* **Profiles**
* `json:raw` canonical JSON lines for advisories, VEX, SBOMs.
* `json:policy` adds policy evaluation results (allow/deny/risk, rationales).
* `trivy:db` Trivy DB‑compatible export (core vulnerability DB).
* `trivy:java-db` optional Java ecosystem DB export if enabled.
* `mirror:full` air‑gap bundle with raw + normalized + indexes + policy + VEX consensus + SBOMs.
* `mirror:delta` incremental bundle based on a previous export manifest.
* **Scope selectors**
* By time window, product, ecosystem, package, image digest, repository, tenant, tag.
* Include/exclude SBOMs, advisories, VEX, findings, policy snapshots.
* **Policy awareness**
* Optionally bake a **policy snapshot** into the bundle, including the policy version, inputs and decision traces.
* Can export **raw** evidence only (AOC) or **raw + evaluated**.
* **Provenance & signing**
* Generate attestation metadata with source URIs, artifact hashes, schema versions, export profile and filters.
* Sign manifests and bundle using configured KMS; support detached and in‑bundle signatures.
* **Distribution**
* Download via Console or API (streaming).
* Push to OCI registry as an artifact/image with annotations.
* Write to object storage prefix for batch pickup.
* **Scheduling & automation**
* One‑off, cron, and event‑triggered exports (e.g., after a “VEX consensus recompute” run).
* Retention policies and automatic expiry for old bundles.
* **Observability**
* Export metrics, throughput, size, downstream pull counts (when pushed to registry with report backs).
### 3.2 Architecture
* **exporter (service)**
* Orchestrates export runs, gathers records from the ledger and indexes, calls format adapters, writes bundles, signs, and publishes distribution tasks.
* Stateless workers pull “export.jobs” from the orchestrator, stream data, and write manifests into object storage.
* **format adapters**
* Pluggable adapters:
* `adapter-json`: canonicalized JSONL writers per record type.
* `adapter-trivy`: translates Stella’s normalized advisory model into Trivy DB format (and Java DB if enabled).
* `adapter-mirror`: constructs a portable filesystem/OCI layout with manifests, indexes, and data subtrees.
* **manifesting & provenance**
* `export.json` (export manifest): profile, filters, counts, schema versions, content checksums, start/finish times, inputs list (artifact ids + hashes).
* `provenance.json`: full chain back to source runs and artifacts; linked signatures.
* **distribution engines**
* `dist-http` streaming for downloads.
* `dist-oci` layer writer with descriptors and annotations.
* `dist-objstore` for staging to buckets.
* **security**
* Tenant scoping, RBAC on export creation and retrieval.
* Optional in‑bundle encryption (age/AES‑GCM) with key wrapping using KMS.
### 3.3 Data model (selected tables)
* `export_profiles`
* `id`, `name`, `kind` (`json|trivy|mirror`), `variant` (`raw|policy|db|java-db|full|delta`), `config_json`, `created_by`, `created_at`.
* `export_runs`
* `id`, `profile_id`, `trigger` (`manual|schedule|event|api`), `state`, `filters_json`, `policy_version`, `started_at`, `finished_at`, `artifact_uri`, `size_bytes`, `sig_uri`, `provenance_uri`, `tenant_id`, `error_class`, `error_message`.
* `export_inputs`
* Link table between `export_runs` and source artifacts. `export_run_id`, `artifact_id`, `hash`.
* `export_distributions`
* `export_run_id`, `type` (`download|oci|objstore`), `target`, `state`, `meta_json`, `created_at`, `updated_at`.
### 3.4 Canonical file layouts
**JSON profile output**
Directory layout under export root:
```
/export/
export.json # export manifest
provenance.json # provenance and source artifact chain
signatures/
export.sig # detached signature for export.json
advisories/
normalized.jsonl # normalized advisory records
vex/
normalized.jsonl # normalized VEX records
sboms/
<subject>/sbom.spdx.json # one per subject; SPDX JSON or CycloneDX JSON
findings/
policy_evaluated.jsonl # if profile=json:policy
```
**Trivy DB profile output**
Produced as a compressed artifact:
```
/export/
export.json
provenance.json
trivy/
db.bundle # Trivy DB compatible archive
java-db.bundle # optional Java DB bundle (if enabled)
signatures/
trivy-db.sig
```
Notes:
* The adapter keeps an internal mapping of Stella normalized fields to Trivy’s expected fields and namespaces. The mapping is versioned to track upstream schema evolution.
**Mirror bundle (filesystem layout)**
```
/mirror/
manifest.yaml # high-level bundle manifest (profile, filters, counts)
export.json # same as JSON profile
provenance.json
indexes/
advisories.index.json # quick lookups (pkg -> advisory ids)
vex.index.json
sbom.index.json
advisories/raw/...
advisories/normalized/...
vex/raw/...
vex/normalized/...
sboms/raw/...
sboms/graph/...
policy/
snapshot.yaml # full policy set used for evaluation
evaluations.jsonl # decision outputs if requested
consensus/
vex_consensus.jsonl
signatures/
manifest.sig
export.sig
README.md
```
**Mirror bundle (OCI layout)**
Following standard OCI image artifact layout with annotations (`org.opencontainers.artifact.description`, `com.stella.export.profile`, `com.stella.export.filters`), and manifest lists for large bundles.
### 3.5 Export workflow
1. **Plan**
Exporter computes candidates based on filters. For `mirror:delta`, compares with previous manifest to compute changes.
2. **Stream & write**
Records are streamed from the Findings Ledger and stores. Writers are forward‑only, emitting JSONL or adapter‑specific structures, chunked for memory safety.
3. **Sign & attest**
Once all content hashes are stable, Export Center writes `export.json`, `provenance.json`, and signs using KMS. Optional encryption wraps data layers.
4. **Distribute**
Depending on profile settings, it exposes a download URL, pushes an OCI artifact, or writes to object storage. Distribution metadata is recorded.
5. **Audit & retention**
Run, manifest, and signatures are immutable. Retention policy prunes large data folders after N days with manifests retained longer.
### 3.6 APIs
```
POST /export/profiles
GET /export/profiles?kind=&variant=
GET /export/profiles/{id}
PATCH /export/profiles/{id}
DELETE /export/profiles/{id}
POST /export/runs
GET /export/runs?state=&profile_id=&from=&to=&tenant_id=
GET /export/runs/{run_id}
POST /export/runs/{run_id}/cancel
GET /export/runs/{run_id}/download # presigned URL or streaming
POST /export/runs/{run_id}/distribute # { "type":"oci|objstore", "target":"..." }
GET /export/runs/{run_id}/manifest # export.json
GET /export/runs/{run_id}/provenance
GET /export/runs/{run_id}/signatures
GET /export/metrics/overview
WS /export/streams/updates
```
**Request example (create run):**
```json
{
"profile_id": "prof_json_policy",
"filters": {
"time_from": "2025-01-01T00:00:00Z",
"time_to": "2025-01-31T23:59:59Z",
"ecosystems": ["pypi", "npm"],
"include": ["advisories", "vex", "sboms", "findings"]
},
"distribution": { "type": "download" },
"policy_version": "pol-v1.8.2"
}
```
### 3.7 CLI
```
stella export profiles list --kind mirror
stella export profiles create --file profile.yaml
stella export run create --profile prof_json_policy --from 2025-01-01 --to 2025-01-31 --include advisories,vex,sboms --download
stella export run status <run-id>
stella export run cancel <run-id>
stella export run get <run-id> --manifest
stella export run download <run-id> --out export-jan.tar.zst
stella export distribute <run-id> --oci ghcr.io/org/stella-export:jan2025
stella export verify <bundle> --manifest export.json --sig signatures/export.sig
```
Exit codes: `0` ok, `2` bad args, `4` not found, `5` denied, `6` integrity failed, `8` export error.
### 3.8 RBAC & security
* Roles:
* `Export.Viewer`: list runs, download completed bundles.
* `Export.Operator`: create runs, cancel, schedule, distribute.
* `Export.Admin`: manage profiles, set retention, manage signing keys.
* Tenancy:
* Every run and artifact scoped by tenant; cross‑tenant export is disallowed.
* Secrets:
* KMS references for signing and encryption; never store private keys.
* PII & redaction:
* Exporters must not include secrets or credentials. Redaction rules enforced at writer level with schema‑based allowlists.
### 3.9 Observability
* Metrics:
* `export_bytes_total{profile,tenant}`
* `export_records_total{type}`
* `export_duration_ms{profile}`
* `export_failures_total{error_class}`
* `export_distributions_total{type}`
* `export_verify_fail_total`
* Traces:
* Spans per export phase: plan, stream, write, sign, distribute; baggage includes `export_run_id`.
* Logs:
* Structured JSON with counts, sizes, hashes, and redaction hints.
### 3.10 Performance targets
* Stream throughput ≥ 25k records/sec per worker for JSONL writing with compression.
* Trivy bundle generation for 1M advisories ≤ 8 minutes on a standard worker.
* Mirror delta export for 5% change set ≤ 2 minutes.
### 3.11 Edge cases & behavior
* **Schema drift**: adapter refuses to emit unknown fields without explicit mapping; run fails with `error_class=schema_mismatch`.
* **Oversized bundles**: automatic sharding by time or content type; mirror OCI uses multi‑manifest indices.
* **Missing policy snapshot**: profile `json:policy` will auto‑pull latest version unless pinned; pinning is recommended for reproducibility.
* **Duplicate evidence**: writers dedupe by artifact hash and advisory id; AOC forbids merging.
* **Air‑gap encryption**: if `encrypt=true`, mirror bundles require recipient public key material; decryption tooling documented.
---
## 4) Implementation plan
### 4.1 Modules
* **New service:** `src/ExportCenter/StellaOps.ExportCenter`
* `api/` REST + WS
* `planner/` scope planning, delta computation, sampling
* `adapters/`
* `json/` canonical writers
* `trivy/` db builders and schema mapping
* `mirror/` fs/OCI builders, sharding, delta logic
* `signing/` KMS clients, attestations
* `dist/` download streaming, OCI push, object storage writer
* `state/` repositories, migrations
* `metrics/`, `audit/`, `security/`
* **SDK/CLI**
* `src/Cli/StellaOps.Cli` subcommands with streaming download and verification.
* **Console**
* `console/apps/export-center/` pages:
* Overview, Profiles, Runs, Run Detail, Distributions, Settings.
* Components: `ExportPlanPreview`, `ProfileEditor`, `RunDiff`, `VerifyPanel`.
* **Existing services updates**
* Findings Ledger: new paginated streaming endpoints for advisories/VEX/SBOM/findings by filters and snapshots.
* Policy Engine: “policy snapshot” exportable endpoint.
* VEX Lens: consensus snapshot endpoint.
### 4.2 Packaging & deployment
* Containers:
* `stella/exporter:<ver>`
* `stella/exporter-worker:<ver>` (optional separated worker pool)
* Helm:
* WS replicas, concurrent export limits, default compression (`zstd`), default retention, KMS settings, OCI creds secrets.
* DB migrations:
* Create `export_*` tables with proper indices (tenant, time, state).
### 4.3 Rollout
* Phase 1: JSON (raw, policy) and Mirror (full) as download only.
* Phase 2: Trivy DB adapters, OCI distribution.
* Phase 3: Mirror deltas, encryption, verification tooling, scheduling.
---
## 5) Documentation changes
Create/update the following docs; each page must end with the imposed rule statement.
1. `/docs/export-center/overview.md`
Purpose, profiles, supported targets, AOC alignment, security model.
2. `/docs/export-center/architecture.md`
Service components, adapters, manifests, signing, distribution flows.
3. `/docs/export-center/profiles.md`
Profile schemas, examples, versioning, compatibility notes.
4. `/docs/export-center/api.md`
All endpoints with request/response examples and error codes.
5. `/docs/export-center/cli.md`
Commands with examples, scripts for CI/CD, verification.
6. `/docs/export-center/mirror-bundles.md`
Filesystem and OCI layouts, delta exports, encryption, air‑gap import guide.
7. `/docs/export-center/trivy-adapter.md`
Field mapping, supported ecosystems, compatibility and test matrix.
8. `/docs/export-center/provenance-and-signing.md`
Manifest format, attestation details, verification process.
9. `/docs/operations/export-runbook.md`
Common failures, recovery, tuning, capacity planning.
10. `/docs/security/export-hardening.md`
RBAC, tenant isolation, secret redaction, encryption keys.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 6) Engineering tasks
### Backend: exporter
* [ ] Migrations for `export_profiles`, `export_runs`, `export_inputs`, `export_distributions`.
* [ ] Planner to resolve filters to iterators over advisory/VEX/SBOM/findings datasets with pagination.
* [ ] JSON adapter: canonical JSONL writers with schema normalization and redaction enforcement.
* [ ] Policy snapshot embedder: pull policy version and evaluation outputs when requested.
* [ ] Trivy adapter: implement schema mapping, writer, integrity validation, and compatibility version flag.
* [ ] Mirror adapter: filesystem and OCI writer, sharding, manifest creation, delta computation.
* [ ] Signing/attestation using KMS; detached and embedded options.
* [ ] Distribution engines: download streaming, OCI push, object storage staging.
* [ ] API layer with async export run handling and WebSocket updates.
* [ ] Rate limit and concurrency controls per tenant/profile.
* [ ] Audit logging for all create/cancel/distribute/verify actions.
### Integrations
* [ ] Findings Ledger streaming APIs for each content type.
* [ ] Policy Engine endpoint to return deterministic policy snapshot and decision set by run.
* [ ] VEX Lens endpoint to expose consensus snapshot.
### Console
* [ ] Profiles CRUD with validation and test preview.
* [ ] Create Run wizard with live count estimates and storage footprint prediction.
* [ ] Runs list + detail page with manifest, provenance, and quick verify.
* [ ] Download and distribution actions with progress and logs.
* [ ] Verification panel to check signatures and hashes client‑side.
### CLI
* [ ] `stella export` commands as defined; include `verify` that checks signatures and hashes.
* [ ] Auto‑resume of interrupted downloads with range requests.
* [ ] Friendly error messages for schema mismatch and verification failure.
### Observability
* [ ] Metrics and traces per §3.9; dashboards for throughput, durations, failures, sizes.
* [ ] Alerts for export failure rate and verify failures.
### Security & RBAC
* [ ] Enforce tenant scoping at query level; fuzz tests for leakage.
* [ ] Role matrix checks on each API; Console hides forbidden actions.
* [ ] Encryption test vectors and key rotation procedure.
### Docs
* [ ] Author all files in §5 with concrete examples and diagrams.
* [ ] Cross‑link from Orchestrator, Policy Studio, VEX Lens, and SBOM docs to the Export Center pages.
* [ ] Append imposed rule line at the end of each page.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Implementation notes per profile
### 7.1 JSON: raw
* **Content:** advisories.normalized, vex.normalized, sboms (SPDX/CycloneDX), optional findings.raw.
* **Normalization:** enforce Stella field casing, timestamps in RFC3339, unicode NFC.
* **Compression:** `.jsonl.zst` per file to allow split/merge.
### 7.2 JSON: policy
* **Adds:** `policy_snapshot` and `findings.policy_evaluated.jsonl` with decision, rule_id, rationale, inputs fingerprint.
* **Determinism:** include `policy_version` and `inputs_hash`; replays should match exactly.
### 7.3 Trivy DB
* **Mapping:**
* Package name, ecosystem, version ranges, CVE/CWE/aliases, severity mapping, vendor statements, fixed versions.
* Ensure namespace mapping avoids collisions (e.g., distro vs ecosystem).
* **Compatibility:** version flag in manifest; adapter throws if upstream schema version not supported.
* **Validation:** run post‑build sanity checks (counts, required indexes).
### 7.4 Mirror: full/delta
* **Full:** everything needed to spin up an isolated read‑only Stella mirror with local search.
* **Delta:** compute changed/added/removed advisory ids, VEX statements, SBOM subjects; update indexes and manifest with `base_export_id`.
* **Encryption:** if enabled, encrypt data subtrees; leave `manifest.yaml` unencrypted for discoverability unless `strict=true`.
---
## 8) Acceptance criteria
* Operators can create an export with filters, download it, verify signatures, and trace back to source artifacts via provenance.
* Trivy adapter produces a bundle consumable by Trivy without custom flags (basic validation in CI).
* Mirror bundle imports successfully in an air‑gapped “mirror‑reader” sample app and serves queries from indexes.
* Policy‑aware exports include deterministic decisions matching the specified `policy_version`.
* RBAC prevents a Viewer from creating or canceling exports; tenancy prevents cross‑tenant leakage.
* Metrics and dashboards show per‑profile throughput and error classes; alerts trigger on failure spikes.
* Export retries are idempotent and do not duplicate content; hashes stable across re‑runs with identical inputs.
---
## 9) Risks & mitigations
* **Upstream schema changes break Trivy export.**
Mitigation: versioned adapter with compatibility gate; integration tests against known fixtures; fail early with clear remediation.
* **Bundle size explosion.**
Mitigation: zstd compression, sharding, delta exports, content‑addressed storage reuse for mirror OCI.
* **Data leakage via exports.**
Mitigation: strict allowlist schemas, redaction filters, RBAC, tenant scoping tests, encryption for mirror.
* **Non‑deterministic policy outputs.**
Mitigation: pin policy version and inputs hash; snapshot embedded rules; deterministic evaluation mode only.
* **Slow downloads/timeouts.**
Mitigation: streaming with range support, resumable downloads, CDN integration if needed.
---
## 10) Test plan
* **Unit**
Schema normalization; Trivy mapping; mirror delta computation; manifest hashing; signing.
* **Integration**
End‑to‑end export with each profile; verify bundles; replay determinism of policy exports; OCI push/pull.
* **Compatibility**
Validate Trivy bundle against a matrix of versions; import mirror bundle into a reference reader and run queries.
* **Security**
Tenant isolation fuzzing; redaction checks; encryption round‑trip; signature verification with rotated keys.
* **Performance**
Large dataset generation; parallel writer stress; OCI multi‑manifest publishing; download resume under packet loss.
* **Chaos**
Kill exporter mid‑write; ensure resume or clean failure without partial corrupt bundles.
---
## 11) Philosophy
* **Ports, not prisons.** Exports should free your data to move with integrity and context, not trap it in a proprietary maze.
* **Reproducible or it didn’t happen.** Every bit derived from known inputs, signed, traceable.
* **Air‑gap is a first‑class citizen.** Mirror bundles are not an afterthought; they’re how serious orgs actually run.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

BIN
docs/implplan/EPIC_11.md
View File

Binary file not shown.

BIN
docs/implplan/EPIC_12.md
View File

Binary file not shown.

BIN
docs/implplan/EPIC_13.md
View File

Binary file not shown.

BIN
docs/implplan/EPIC_14.md
View File

Binary file not shown.

BIN
docs/implplan/EPIC_15.md
View File

Binary file not shown.

1
docs/implplan/EPIC_16.md
View File

@@ -1 +0,0 @@
See `docs/airgap/EPIC_16_AIRGAP_MODE.md` for the full Epic 16 specification.

1
docs/implplan/EPIC_17.md
View File

@@ -1 +0,0 @@
See `docs/api/EPIC_17_SDKS_OPENAPI.md` for the complete Epic 17 specification.

1
docs/implplan/EPIC_18.md
View File

@@ -1 +0,0 @@
See `docs/risk/EPIC_18_RISK_PROFILES.md` for the complete Epic 18 specification.

1
docs/implplan/EPIC_19.md
View File

@@ -1 +0,0 @@
See `docs/attestor/EPIC_19_ATTESTOR_CONSOLE.md` for the complete Epic 19 specification.

567
docs/implplan/EPIC_2.md
View File

@@ -1,567 +0,0 @@
Fine. Here’s the next epic, written so you can paste it straight into the repo without having to babysit me. Same structure as before, maximum detail, zero hand‑waving.
---
# Epic 2: Policy Engine & Policy Editor (VEX + Advisory Application Rules)
> Short name: **Policy Engine v2**
> Services touched: **Policy Engine, Web API, Console (Policy Editor), CLI, Conseiller, Excitator, SBOM Service, Authority, Workers/Scheduler**
> Data stores: **MongoDB (policies, runs, effective findings), optional Redis/NATS for jobs**
---
## 1) What it is
This epic delivers the **organization‑specific decision layer** for Stella. Ingestion is now AOC‑compliant (Epic 1). That means advisories and VEX arrive as immutable raw facts. This epic builds the place where those facts become **effective findings** under policies you control.
Core deliverables:
* **Policy Engine**: deterministic evaluator that applies rule sets to inputs:
* Inputs: `advisory_raw`, `vex_raw`, SBOMs, optional telemetry hooks (reachability stubs), org metadata.
* Outputs: `effective_finding_{policyId}` materializations, with full explanation traces.
* **Policy Editor (Console + CLI)**: versioned policy authoring, simulation, review/approval workflow, and change diffs.
* **Rules DSL v1**: safe, declarative language for VEX application, advisory normalization, and risk scoring. No arbitrary code execution, no network calls.
* **Run Orchestrator**: incremental re‑evaluation when new raw facts or SBOM changes arrive; efficient partial updates.
The philosophy is boring on purpose: policy is a **pure function of inputs**. Same inputs and same policy yield the same outputs, every time, on every machine. If you want drama, watch reality TV, not your risk pipeline.
---
## 2) Why
* Vendors disagree, contexts differ, and your tolerance for risk is not universal.
* VEX means nothing until you decide **how** to apply it to **your** assets.
* Auditors care about the “why.” You’ll need consistent, replayable answers, with traces.
* Security teams need **simulation** before rollouts, and **diffs** after.
---
## 3) How it should work (deep details)
### 3.1 Data model
#### 3.1.1 Policy documents (Mongo: `policies`)
```json
{
"_id": "policy:P-7:v3",
"policy_id": "P-7",
"version": 3,
"name": "Default Org Policy",
"status": "approved", // draft | submitted | approved | archived
"owned_by": "team:sec-plat",
"valid_from": "2025-01-15T00:00:00Z",
"valid_to": null,
"dsl": {
"syntax": "stella-dsl@1",
"source": "rule-set text or compiled IR ref"
},
"metadata": {
"description": "Baseline scoring + VEX precedence",
"tags": ["baseline","vex","cvss"]
},
"provenance": {
"created_by": "user:ali",
"created_at": "2025-01-15T08:00:00Z",
"submitted_by": "user:kay",
"approved_by": "user:root",
"approval_at": "2025-01-16T10:00:00Z",
"checksum": "sha256:..."
},
"tenant": "default"
}
```
Constraints:
* `status=approved` is required to run in production.
* Version increments are append‑only. Old versions remain runnable for replay.
#### 3.1.2 Policy runs (Mongo: `policy_runs`)
```json
{
"_id": "run:P-7:2025-02-20T12:34:56Z:abcd",
"policy_id": "P-7",
"policy_version": 3,
"inputs": {
"sbom_set": ["sbom:S-42"],
"advisory_cursor": "2025-02-20T00:00:00Z",
"vex_cursor": "2025-02-20T00:00:00Z"
},
"mode": "incremental", // full | incremental | simulate
"stats": {
"components": 1742,
"advisories_considered": 9210,
"vex_considered": 1187,
"rules_fired": 68023,
"findings_out": 4321
},
"trace": {
"location": "blob://traces/run-.../index.json",
"sampling": "smart-10pct"
},
"status": "succeeded", // queued | running | failed | succeeded | canceled
"started_at": "2025-02-20T12:34:56Z",
"finished_at": "2025-02-20T12:35:41Z",
"tenant": "default"
}
```
#### 3.1.3 Effective findings (Mongo: `effective_finding_P-7`)
```json
{
"_id": "P-7:S-42:pkg:npm/lodash@4.17.21:CVE-2021-23337",
"policy_id": "P-7",
"policy_version": 3,
"sbom_id": "S-42",
"component_purl": "pkg:npm/lodash@4.17.21",
"advisory_ids": ["CVE-2021-23337", "GHSA-..."],
"status": "affected", // affected | not_affected | fixed | under_investigation | suppressed
"severity": {
"normalized": "High",
"score": 7.5,
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
"rationale": "cvss_base(OSV) + vendor_weighting + env_modifiers"
},
"rationale": [
{"rule":"vex.precedence","detail":"VendorX not_affected justified=component_not_present wins"},
{"rule":"advisory.cvss.normalization","detail":"mapped GHSA severity to CVSS 3.1 = 7.5"}
],
"references": {
"advisory_raw_ids": ["advisory_raw:osv:GHSA-...:v3"],
"vex_raw_ids": ["vex_raw:VendorX:doc-123:v4"]
},
"run_id": "run:P-7:2025-02-20T12:34:56Z:abcd",
"tenant": "default"
}
```
Write protection: only the **Policy Engine** service identity may write any `effective_finding_*` collection.
---
### 3.2 Rules DSL v1 (stella‑dsl@1)
**Design goals**
* Declarative, composable, deterministic.
* No loops, no network IO, no non‑deterministic time.
* Policy authors see readable text; the engine compiles to a safe IR.
**Concepts**
* **WHEN** condition matches a tuple `(sbom_component, advisory, optional vex_statements)`
* **THEN** actions set `status`, compute `severity`, attach `rationale`, or `suppress` with reason.
* **Profiles** for severity and scoring; **Maps** for vendor weighting; **Guards** for VEX justification.
**Mini‑grammar (subset)**
```
policy "Default Org Policy" syntax "stella-dsl@1" {
profile severity {
map vendor_weight {
source "GHSA" => +0.5
source "OSV" => +0.0
source "VendorX" => -0.2
}
env base_cvss {
if env.runtime == "serverless" then -0.5
if env.exposure == "internal-only" then -1.0
}
}
rule vex_precedence {
when vex.any(status in ["not_affected","fixed"])
and vex.justification in ["component_not_present","vulnerable_code_not_present"]
then status := vex.status
because "VEX strong justification prevails";
}
rule advisory_to_cvss {
when advisory.source in ["GHSA","OSV"]
then severity := normalize_cvss(advisory)
because "Map vendor severity or CVSS vector";
}
rule reachability_soft_suppress {
when severity.normalized <= "Medium"
and telemetry.reachability == "none"
then status := "suppressed"
because "not reachable and low severity";
}
}
```
**Built‑ins** (non‑exhaustive)
* `normalize_cvss(advisory)` maps GHSA/OSV/CSAF severity fields to CVSS v3.1 numbers when possible; otherwise vendor‑to‑numeric mapping table in policy.
* `vex.any(...)` tests across matching VEX statements for the same `(component, advisory)`.
* `telemetry.*` is an optional input namespace reserved for future reachability data; if absent, expressions evaluate to `unknown` (no effect).
**Determinism**
* Rules are evaluated in **stable order**: explicit `priority` attribute or lexical order.
* **First‑match** semantics for conflicting status unless `combine` is used.
* Severity computations are pure; numeric maps are part of policy document.
---
### 3.3 Evaluation model
1. **Selection**
* For each SBOM component PURL, find candidate advisories from `advisory_raw` via linkset PURLs or identifiers.
* For each pair `(component, advisory)`, load all matching VEX facts from `vex_raw`.
2. **Context assembly**
* Build an evaluation context from:
* `sbom_component`: PURL, licenses, relationships.
* `advisory`: source, identifiers, references, embedded vendor severity (kept in `content.raw`).
* `vex`: list of statements with status and justification.
* `env`: org‑specific env vars configured per policy run (e.g., exposure).
* Optional `telemetry` if available.
3. **Rule execution**
* Compile DSL to IR once per policy version; cache.
* Execute rules per tuple; record which rules fired and the order.
* If no rule sets status, default is `affected`.
* If no rule sets severity, default severity uses `normalize_cvss(advisory)` with vendor defaults.
4. **Materialization**
* Write to `effective_finding_{policyId}` with `rationale` chain and references to raw docs.
* Emit per‑tuple trace events; sample and store full traces per run.
5. **Incremental updates**
* A watch job observes new `advisory_raw` and `vex_raw` inserts and SBOM deltas.
* The orchestrator computes the affected tuples and re‑evaluates only those.
6. **Replay**
* Any `policy_run` is fully reproducible by `(policy_id, version, input set, cursors)`.
---
### 3.4 VEX application semantics
* **Precedence**: a `not_affected` with strong justification (`component_not_present`, `vulnerable_code_not_present`, `fix_not_required`) wins unless another rule explicitly overrides by environment context.
* **Scoping**: VEX statements often specify product/component scope. Matching uses PURL equivalence and version ranges extracted during ingestion linkset generation.
* **Conflicts**: If multiple VEX statements conflict, the default is **most‑specific scope wins** (component > product > vendor), then newest `document_version`. Policies can override with explicit rules.
* **Explainability**: Every VEX‑driven decision records which statement IDs were considered and which one won.
---
### 3.5 Advisory normalization rules
* **Vendor severity mapping**: Map GHSA levels or CSAF product‑tree severities to CVSS‑like numeric bands via policy maps.
* **CVSS vector use**: If a valid vector exists in `content.raw`, parse and compute; apply policy modifiers from `profile severity`.
* **Temporal/environment modifiers**: Optional reductions for network exposure, isolation, or compensating controls, all encoded in policy.
---
### 3.6 Performance and scale
* Partition evaluation by SBOM ID and hash ranges of PURLs.
* Pre‑index `advisory_raw.linkset.purls` and `vex_raw.linkset.purls` (already in Epic 1).
* Use streaming iterators; avoid loading entire SBOM or advisory sets into memory.
* Materialize only changed findings (diff‑aware writes).
* Target: 100k components, 1M advisories considered, 5 minutes incremental SLA on commodity hardware.
---
### 3.7 Error codes
| Code | Meaning | HTTP |
| ------------- | ----------------------------------------------------- | ---- |
| `ERR_POL_001` | Policy syntax error | 400 |
| `ERR_POL_002` | Policy not approved for run | 403 |
| `ERR_POL_003` | Missing inputs (SBOM/advisory/vex fetch failed) | 424 |
| `ERR_POL_004` | Determinism guard triggered (non‑pure function usage) | 500 |
| `ERR_POL_005` | Write denied to effective findings (caller invalid) | 403 |
| `ERR_POL_006` | Run canceled or timed out | 408 |
---
### 3.8 Observability
* Metrics:
* `policy_compile_seconds`, `policy_run_seconds{mode=...}`, `rules_fired_total`, `findings_written_total`, `vex_overrides_total`, `simulate_diff_total{delta=up|down|unchanged}`.
* Tracing:
* Spans: `policy.compile`, `policy.select`, `policy.eval`, `policy.materialize`.
* Logs:
* Include `policy_id`, `version`, `run_id`, `sbom_id`, `component_purl`, `advisory_id`, `vex_count`, `rule_hits`.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
### 3.9 Security and tenancy
* Only users with `policy:write` can create/modify policies.
* `policy:approve` is a separate privileged role.
* Only Policy Engine service identity has `effective:write`.
* Tenancy is explicit on all documents and queries.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 4) API surface
### 4.1 Policy CRUD and lifecycle
* `POST /policies` create draft
* `GET /policies?status=...` list
* `GET /policies/{policyId}/versions/{v}` fetch
* `POST /policies/{policyId}/submit` move draft to submitted
* `POST /policies/{policyId}/approve` approve version
* `POST /policies/{policyId}/archive` archive version
### 4.2 Compilation and validation
* `POST /policies/{policyId}/versions/{v}/compile`
* Returns IR checksum, syntax diagnostics, rule stats.
### 4.3 Runs
* `POST /policies/{policyId}/runs` body: `{mode, sbom_set, advisory_cursor?, vex_cursor?, env?}`
* `GET /policies/{policyId}/runs/{runId}` status + stats
* `POST /policies/{policyId}/simulate` returns **diff** vs current approved version on a sample SBOM set.
### 4.4 Findings and explanations
* `GET /findings/{policyId}?sbom_id=S-42&status=affected&severity=High+Critical`
* `GET /findings/{policyId}/{findingId}/explain` returns ordered rule hits and linked raw IDs.
All endpoints require tenant scoping and appropriate `policy:*` or `findings:*` roles.
---
## 5) Console (Policy Editor) and CLI behavior
**Console**
* Monaco‑style editor with DSL syntax highlighting, lint, quick docs.
* Side‑by‑side **Simulation** panel: show count of affected findings before/after.
* Approval workflow: submit, review comments, approve with rationale.
* Diffs: show rule‑wise changes and estimated impact.
* Read‑only run viewer: heatmap of rules fired, top suppressions, VEX wins.
**CLI**
* `stella policy new --name "Default Org Policy"`
* `stella policy edit P-7` opens local editor -> `submit`
* `stella policy approve P-7 --version 3`
* `stella policy simulate P-7 --sbom S-42 --env exposure=internal-only`
* `stella findings ls --policy P-7 --sbom S-42 --status affected`
Exit codes map to `ERR_POL_*`.
---
## 6) Implementation tasks
### 6.1 Policy Engine service
* [ ] Implement DSL parser and IR compiler (`stella-dsl@1`).
* [ ] Build evaluator with stable ordering and first‑match semantics.
* [ ] Implement selection joiners for SBOM↔advisory↔vex using linksets.
* [ ] Materialization writer with upsert‑only semantics to `effective_finding_{policyId}`.
* [ ] Determinism guard (ban wall‑clock, network, and RNG during eval).
* [ ] Incremental orchestrator listening to advisory/vex/SBOM change streams.
* [ ] Trace emitter with rule‑hit sampling.
* [ ] Unit tests, property tests, golden fixtures; perf tests to target SLA.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.2 Web API
* [ ] Policy CRUD, compile, run, simulate, findings, explain endpoints.
* [ ] Pagination, filters, and tenant enforcement on all list endpoints.
* [ ] Error mapping to `ERR_POL_*`.
* [ ] Rate limits on simulate endpoints.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.3 Console (Policy Editor)
* [ ] Editor with DSL syntax highlighting and inline diagnostics.
* [ ] Simulation UI with pre/post counts and top deltas.
* [ ] Approval workflow UI with audit trail.
* [ ] Run viewer dashboards (rule heatmap, VEX wins, suppressions).
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.4 CLI
* [ ] New commands: `policy new|edit|submit|approve|simulate`, `findings ls|get`.
* [ ] Json/YAML output formats for CI consumption.
* [ ] Non‑zero exits on syntax errors or simulation failures; map to `ERR_POL_*`.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.5 Conseiller & Excitator integration
* [ ] Provide search endpoints optimized for policy selection (batch by PURLs and IDs).
* [ ] Harden linkset extraction to maximize join recall.
* [ ] Add cursors for incremental selection windows per run.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.6 SBOM Service
* [ ] Ensure fast PURL index and component metadata projection for policy queries.
* [ ] Provide relationship graph API for future transitive logic.
* [ ] Emit change events on SBOM updates.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.7 Authority
* [ ] Define scopes: `policy:write`, `policy:approve`, `policy:run`, `findings:read`, `effective:write`.
* [ ] Issue service identity for Policy Engine with `effective:write` only.
* [ ] Enforce tenant claims at gateway.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 6.8 CI/CD
* [ ] Lint policy DSL in PRs; block invalid syntax.
* [ ] Run `simulate` against golden SBOMs to detect explosive deltas.
* [ ] Determinism CI: two runs with identical seeds produce identical outputs.
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Documentation changes (create/update these files)
1. **`/docs/policy/overview.md`**
* What the Policy Engine is, high‑level concepts, inputs, outputs, determinism.
2. **`/docs/policy/dsl.md`**
* Full grammar, built‑ins, examples, best practices, anti‑patterns.
3. **`/docs/policy/lifecycle.md`**
* Draft → submitted → approved → archived, roles, and audit trail.
4. **`/docs/policy/runs.md`**
* Run modes, incremental mechanics, cursors, replay.
5. **`/docs/api/policy.md`**
* Endpoints, request/response schemas, error codes.
6. **`/docs/cli/policy.md`**
* Command usage, examples, exit codes, JSON output contracts.
7. **`/docs/ui/policy-editor.md`**
* Screens, workflows, simulation, diffs, approvals.
8. **`/docs/architecture/policy-engine.md`**
* Detailed sequence diagrams, selection/join strategy, materialization schema.
9. **`/docs/observability/policy.md`**
* Metrics, tracing, logs, sample dashboards.
10. **`/docs/security/policy-governance.md`**
* Scopes, approvals, tenancy, least privilege.
11. **`/docs/examples/policies/`**
* `baseline.pol`, `serverless.pol`, `internal-only.pol`, each with commentary.
12. **`/docs/faq/policy-faq.md`**
* Common pitfalls, VEX conflict handling, determinism gotchas.
Each file includes a **Compliance checklist** for authors and reviewers.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 8) Acceptance criteria
* Policies are versioned, approvable, and compilable; invalid DSL blocks merges.
* Engine produces deterministic outputs with full rationale chains.
* VEX precedence rules work per spec and are overridable by policy.
* Simulation yields accurate pre/post deltas and diffs.
* Only Policy Engine can write to `effective_finding_*`.
* Incremental runs pick up new advisories/VEX/SBOM changes without full re‑runs.
* Console and CLI cover authoring, simulation, approval, and retrieval.
* Observability dashboards show rule hits, VEX wins, and run timings.
---
## 9) Risks and mitigations
* **Policy sprawl**: too many similar policies.
* Mitigation: templates, policy inheritance in v1.1, tagging, ownership metadata.
* **Non‑determinism creep**: someone sneaks wall‑clock or network into evaluation.
* Mitigation: determinism guard, static analyzer, and CI replay check.
* **Join miss‑rate**: weak linksets cause under‑matching.
* Mitigation: linkset strengthening in ingestion, PURL equivalence tables, monitoring for “zero‑hit” rates.
* **Approval bottlenecks**: blocked rollouts.
* Mitigation: RBAC with delegated approvers and time‑boxed SLAs.
---
## 10) Test plan
* **Unit**: parser, compiler, evaluator; conflict resolution; precedence.
* **Property**: random policies over synthetic inputs; ensure no panics and stable outputs.
* **Golden**: fixed SBOM + curated advisories/VEX → expected findings; compare every run.
* **Performance**: large SBOMs with heavy rule sets; assert run times and memory ceilings.
* **Integration**: end‑to‑end simulate → approve → run → diff; verify write protections.
* **Chaos**: inject malformed VEX, missing advisories; ensure graceful degradation and clear errors.
---
## 11) Developer checklists
**Definition of Ready**
* Policy grammar finalized; examples prepared.
* Linkset join queries benchmarked.
* Owner and approvers assigned.
**Definition of Done**
* All APIs live with RBAC.
* CLI and Console features shipped.
* Determinism and golden tests green.
* Observability dashboards deployed.
* Docs in section 7 merged.
* Two real org policies migrated and in production.
---
## 12) Glossary
* **Policy**: versioned rule set controlling status and severity.
* **DSL**: domain‑specific language used to express rules.
* **Run**: a single evaluation execution with defined inputs and outputs.
* **Simulation**: a run that doesn’t write findings; returns diffs.
* **Materialization**: persisted effective findings for fast queries.
* **Determinism**: same inputs + same policy = same outputs. Always.
---
### Final imposed reminder
**Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

409
docs/implplan/EPIC_4.md
View File

@@ -1,409 +0,0 @@
Here’s Epic 4 in the same paste‑into‑repo, implementation‑ready style as the prior epics. It’s exhaustive, formal, and slots directly into the existing AOC model, Policy Engine, and Console.
---
# Epic 4: Policy Studio (author, version, simulate)
> Short name: **Policy Studio**
> Services touched: **Policy Engine**, **Policy Registry** (new), **Web API Gateway**, **Authority** (authN/Z), **Scheduler/Workers**, **SBOM Service**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Telemetry**
> Surfaces: **Console (Web UI)** feature module, **CLI**, **CI hooks**
> Deliverables: Authoring workspace, policy versioning, static checks, simulation at scale, reviews/approvals, signing/publishing, promotion
---
## 1) What it is
**Policy Studio** is the end‑to‑end system for creating, evolving, and safely rolling out the rules that turn AOC facts (SBOM, advisories, VEX) into **effective findings**. It provides:
* A **workspace** where authors write policies in the DSL (Epic 2), with linting, autocompletion, snippets, and templates.
* A **Policy Registry** that stores immutable versions, compiled artifacts, metadata, provenance, and signatures.
* **Simulation** at two levels: quick local samples and large batch simulations across real SBOM inventories with full deltas.
* A **review/approval** workflow with comments, diffs, required approvers, and promotion to environments (dev/test/prod).
* **Publishing** semantics: signed, immutable versions bound to tenants; rollback and deprecation.
* Tight integration with **Explain** traces so any change can show exactly which rules fired and why outcomes shifted.
The Studio respects **AOC enforcement**: policies never edit or merge facts. They only interpret facts and produce determinations consistent with precedence rules defined in the DSL.
---
## 2) Why
* Policy errors are expensive. Authors need safe sandboxes, deterministic builds, and evidence before rollout.
* Auditors require immutability, provenance, and reproducibility from “source policy” to “effective finding.”
* Teams want gradual rollout: simulate, canary, promote, observe, rollback.
* Policy knowledge should be modular, reusable, and testable, not tribal.
---
## 3) How it should work (maximum detail)
### 3.1 Domain model
* **PolicyPackage**: `{name, tenant, description, owners[], tags[], created_at}`
* **PolicyVersion** (immutable): `{package, semver, source_sha, compiled_sha, status: draft|review|approved|published|deprecated|archived, created_by, created_at, signatures[], changelog, metadata{}}`
* **Workspace**: mutable working area for authors; holds unversioned edits until compiled.
* **CompilationArtifact**: `{policy_version, compiler_version, diagnostics[], rule_index[], symbol_table}`
* **SimulationSpec**: `{policy_version|workspace, sbom_selector, time_window?, environment?, sample_size?, severity_floor?, includes{advisories?, vex?}}`
* **SimulationRun**: `{run_id, spec, started_at, finished_at, result{counts_before, counts_after, top_deltas[], by_rule_hit[], sample_explains[]}}`
* **Review**: `{policy_version, required_approvers[], votes[], comments[], files_changed[], diffs[]}`
* **Promotion**: `{policy_version, environment: dev|test|prod, promoted_by, promoted_at, rollout_strategy: All|Percent|TenantSubset}`
* **Attestation**: OIDC‑backed signature metadata binding `source_sha` and `compiled_sha` to an actor and time.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.2 Authoring workflow
1. **Create** a workspace from a template (e.g., “Default Risk Model,” “License Tilted,” “Cloud‑Native SBOM”).
2. **Edit** in the Studio: Monaco editor with DSL grammar, intelligent completion for predicates, policies, attributes.
3. **Lint & compile** locally: semantic checks, forbidden rules detection, policy size limits, constant‑folding.
4. **Unit tests**: run policy test cases on bundled fixtures and golden expectations.
5. **Quick simulate** on selected SBOMs (10–50 items) to preview counts, examples, and rule heatmap.
6. **Propose version**: bump semver, enter changelog; create a **PolicyVersion** in `review` with compiled artifacts.
7. **Review & approval**: side‑by‑side diff, comments, required approvers enforced by RBAC.
8. **Batch simulation**: run at scale across tenant inventory; produce deltas, sample explainer evidence.
9. **Publish**: sign and move to `published`; optional **Promotion** to target environment(s).
10. **Run** evaluation with the selected policy version; verify outcomes; optionally promote to default.
11. **Rollback**: select an older version; promotion updates references without mutating older versions.
### 3.3 Editing experience (Console)
* **Three‑pane layout**: file tree, editor, diagnostics/simulation.
* **Features**: autocomplete from symbol table, in‑editor docs on hover, go‑to definition, rule references, rename symbols across files, snippet library, policy templates.
* **Validations**:
* AOC guardrails: no edit/merge actions on source facts, only interpretation.
* Precedence correctness: if rules conflict, studio shows explicit order and effective winner.
* Severity floor and normalization mapping validated against registry configuration.
* **Diagnostics panel**: errors, warnings, performance hints (e.g., “predicate X loads N advisories per component; consider indexing”).
* **Rule heatmap**: during simulation, bar chart of rule firings and the objects they impact.
* **Explain sampler**: click any delta bucket to open a sampled finding with full trace.
### 3.4 Simulation
* **Quick Sim**: synchronous; runs in browser‑orchestrated job against API, constrained by `sample_size`.
* **Batch Sim**: asynchronous run in workers:
* Input selection: all SBOMs, labels, artifact regex, last N ingests, or a curated set.
* Outputs: counts by severity before/after, by status, top deltas by component and advisory, rule heatmap, top K affected artifacts.
* Evidence: NDJSON of sampled findings with traces; CSV summary; signed result manifest.
* Guardrails: cannot publish if batch sim drift > configurable threshold without an override justification.
### 3.5 Versioning & promotion
* Semver enforced: `major` implies compatibility break (e.g., precedence changes), `minor` adds rules, `patch` fixes.
* **Immutable**: after `published`, the version cannot change; deprecate instead.
* **Environment bindings**: dev/test/prod mapping per tenant; default policy per environment.
* **Canary**: promote to a subset of tenants or artifacts; the Runs page displays A/B comparisons.
### 3.6 Review & approval
* Require N approvers by role; self‑approval optionally prohibited.
* Line and file comments; overall decision with justification.
* Review snapshot captures: diffs, diagnostics, simulation summary.
* Webhooks to notify external systems of review events.
### 3.7 RBAC (Authority)
Roles per tenant:
* **Policy Author**: create/edit workspace, quick sim, propose versions.
* **Policy Reviewer**: comment, request changes, approve/reject.
* **Policy Approver**: final approve, publish.
* **Policy Operator**: promote, rollback, schedule runs.
* **Read‑only Auditor**: view everything, download evidence.
All actions server‑checked; UI only hides affordances.
### 3.8 CLI + CI integration
CLI verbs (examples):
```
stella policy init --template default
stella policy lint
stella policy compile
stella policy test --golden ./tests
stella policy simulate --sboms label:prod --sample 1000
stella policy version bump --level minor --changelog "Normalize GHSA CVSS"
stella policy submit --reviewers alice@example.com,bob@example.com
stella policy approve --version 1.3.0
stella policy publish --version 1.3.0 --sign
stella policy promote --version 1.3.0 --env test --percent 20
stella policy rollback --env prod --to 1.2.1
```
CI usage:
* Lint, compile, and run unit tests on PRs that modify `/policies/**`.
* Optionally trigger **Batch Sim** against a staging inventory and post a Markdown report to the PR.
* Block merge if diagnostics include errors or drift exceeds thresholds.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.9 APIs (representative)
* `POST /policies/workspaces` create from template
* `PUT /policies/workspaces/{id}/files` edit source files
* `POST /policies/workspaces/{id}/compile` get diagnostics + compiled artifact
* `POST /policies/workspaces/{id}/simulate` quick sim
* `POST /policies/versions` create version from workspace with semver + changelog
* `GET /policies/versions/{id}` fetch version + diagnostics + sim summary
* `POST /policies/versions/{id}/reviews` open review
* `POST /policies/versions/{id}/approve` record approval
* `POST /policies/versions/{id}/publish` sign + publish
* `POST /policies/versions/{id}/promote` bind to env/canary
* `POST /policies/versions/{id}/simulate-batch` start batch sim (async)
* `GET /policies/simulations/{run_id}` get sim results and artifacts
* `GET /policies/registry` list packages/versions, status and bindings
All calls require tenant scoping and RBAC.
### 3.10 Storage & data
* **Policy Registry DB** (MongoDB): packages, versions, workspaces, metadata.
* **Object storage**: source bundles, compiled artifacts, simulation result bundles, evidence.
* **Indexing**: compound indexes by `{tenant, package}`, `{tenant, status}`, `{tenant, environment}`.
* **Retention**: configurable retention for workspaces and simulation artifacts; versions never deleted, only archived.
### 3.11 Evidence & provenance
* Every published version has:
* `source_sha` (content digest of the policy source bundle)
* `compiled_sha` (digest of compiled artifact)
* Attestation: signed envelope binding digests to an identity, time, and tenant.
* Links to the exact compiler version, inputs, and environment.
### 3.12 Observability
* Metrics: compile time, diagnostics rate, simulation queue depth, delta magnitude distribution, approval latencies.
* Logs: structured events for lifecycle transitions.
* Traces: long simulations emit span per shard.
### 3.13 Performance & scale
* Compilation should complete under 3 seconds for typical policies; warn at 10s.
* Batch sim uses workers with partitioning by SBOM id; results reduced by the API.
* Memory guardrails on rule execution; deny policies that exceed configured complexity limits.
### 3.14 Security
* OIDC‑backed signing and attestation.
* Policy sources are scanned on upload for secrets; blocked if found.
* Strict CSP in Studio pages; tokens stored in memory, not localStorage.
* Tenant isolation in buckets and DB collections.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 4) Implementation plan
### 4.1 Services
* **Policy Registry (new microservice)**
* REST API and background workers for batch simulation orchestration.
* Stores workspaces, versions, metadata, bindings, reviews.
* Generates signed attestations at publish time.
* Coordinates with **Policy Engine** for compile/simulate invocations.
* **Policy Engine (existing)**
* Expose compile and simulate endpoints with deterministic outputs.
* Provide rule coverage, symbol table, and explain traces for samples.
* **Web API Gateway**
* Routes requests; injects tenant context; enforces RBAC.
### 4.2 Console (Web UI) feature module
* `packages/features/policies` (shared with Epic 3):
* **Studio** routes: `/policies/studio`, `/policies/:id/versions/:v/edit`, `/simulate`, `/review`.
* Monaco editor wrapper for DSL with hover docs, autocomplete.
* Diff viewer, diagnostics, heatmap, explain sampler, review UI.
### 4.3 CLI
* New commands under `stella policy *`; typed client generated from OpenAPI.
* Outputs machine‑readable JSON and pretty tables.
### 4.4 Workers
* **Simulation workers**: pull shards of SBOMs, run policy, emit partials, reduce into result bundle.
* **Notification worker**: sends webhooks on review, approval, publish, promote.
---
## 5) Documentation changes (create/update)
1. **`/docs/policy/studio-overview.md`**
* Concepts, roles, lifecycle, glossary.
2. **`/docs/policy/authoring.md`**
* Workspace, templates, snippets, lint rules, best practices.
3. **`/docs/policy/versioning-and-publishing.md`**
* Semver, immutability, deprecation, rollback, attestations.
4. **`/docs/policy/simulation.md`**
* Quick vs batch sim, selection strategies, thresholds, evidence artifacts.
5. **`/docs/policy/review-and-approval.md`**
* Required approvers, comments, webhooks, audit trail.
6. **`/docs/policy/promotion.md`**
* Environments, canary, default policy binding, rollback.
7. **`/docs/policy/cli.md`**
* Command reference with examples and JSON outputs.
8. **`/docs/policy/api.md`**
* REST endpoints, request/response schemas, error codes.
9. **`/docs/security/policy-attestations.md`**
* Signatures, digests, verifier steps.
10. **`/docs/architecture/policy-registry.md`**
* Service design, schemas, queues, failure modes.
11. **`/docs/observability/policy-telemetry.md`**
* Metrics, logs, tracing, dashboards.
12. **`/docs/runbooks/policy-incident.md`**
* Rolling back a bad policy, freezing publishes, forensic steps.
13. **`/docs/examples/policy-templates.md`**
* Ready‑made templates and snippet catalog.
14. **`/docs/aoc/aoc-guardrails.md`**
* How Studio enforces AOC in authoring and review.
Each doc ends with a “Compliance checklist.”
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 6) Tasks
### 6.1 Backend: Policy Registry
* [ ] Define OpenAPI spec for Registry (workspaces, versions, reviews, sim).
* [ ] Implement workspace storage and file CRUD.
* [ ] Integrate with Policy Engine compile endpoint; return diagnostics, symbol table.
* [ ] Implement quick simulation with request limits.
* [ ] Implement batch simulation orchestration: enqueue shards, collect results, reduce deltas, store artifacts.
* [ ] Implement review model: comments, required approvers, decisions.
* [ ] Implement publish: sign, persist attestation, set status=published.
* [ ] Implement promotion bindings per tenant/environment; canary subsets.
* [ ] RBAC checks for all endpoints.
* [ ] Unit/integration tests; load tests for batch sim.
### 6.2 Policy Engine enhancements
* [ ] Return rule coverage and firing counts with compile/simulate.
* [ ] Return symbol table and inline docs for editor autocomplete.
* [ ] Expose deterministic Explain traces for sampled findings.
* [ ] Enforce complexity/time limits and report breaches.
### 6.3 Console (Web UI)
* [ ] Build Studio editor wrapper with Monaco + DSL language server hooks.
* [ ] Implement file tree, snippets, templates, hotkeys, search/replace.
* [ ] Diagnostics panel with jump‑to‑line, quick fixes.
* [ ] Simulation panel: quick sim UI, charts, heatmap, sample explains.
* [ ] Review UI: diff, comments, approvals, status badges.
* [ ] Publish & Promote flows with confirmation and post‑actions.
* [ ] Batch sim results pages with export buttons.
* [ ] Accessibility audits and keyboard‑only authoring flow.
### 6.4 CLI
* [ ] Implement commands listed in 3.8 with rich help and examples.
* [ ] Add `--json` flag for machine consumption; emit stable schemas.
* [ ] Exit codes aligned with CI usage (lint errors → non‑zero).
### 6.5 CI/CD & Security
* [ ] Add CI job that runs `stella policy lint/compile/test` on PRs.
* [ ] Optional job that triggers batch sim against staging inventory; post summary to PR.
* [ ] Policy source secret scanning; block on findings.
* [ ] Signing keys configuration; verify pipeline for attestation on publish.
### 6.6 Docs
* [ ] Write all docs in section 5 with screenshots and CLI transcripts.
* [ ] Add cookbook examples and templates in `/docs/examples/policy-templates.md`.
* [ ] Wire contextual Help links from Studio to relevant docs.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* Authors can create, edit, lint, compile policies with inline diagnostics and autocomplete.
* Quick simulation produces counts, rule heatmap, and sample explains within UI.
* Batch simulation scales across large SBOM sets, producing deltas and downloadable evidence.
* Review requires configured approvers; comments and diffs are preserved.
* Publish generates immutable, signed versions with attestations.
* Promotion binds versions to environments and supports canary and rollback.
* CLI supports full lifecycle and is usable in CI.
* All actions are tenant‑scoped, RBAC‑enforced, and logged.
* AOC guardrails prevent any mutation of raw facts.
* Documentation shipped and linked contextually from the Studio.
---
## 8) Risks & mitigations
* **Policy complexity causes timeouts** → compile‑time complexity scoring, execution limits, early diagnostics.
* **Simulation cost at scale** → sharding and streaming reducers; sampling; configurable quotas.
* **RBAC misconfiguration** → server‑enforced checks, defense‑in‑depth tests, deny‑by‑default.
* **Attestation key management** → OIDC‑backed signatures; auditable verifier tool; time‑boxed credentials.
* **Editor usability** → language server with accurate completions; docs on hover; snippet library.
---
## 9) Test plan
* **Unit**: compiler adapters, registry models, reviewers workflow, CLI options.
* **Integration**: compile→simulate→publish→promote on seeded data.
* **E2E**: Playwright flows for author→review→batch sim→publish→promote→rollback.
* **Performance**: load test batch simulation with 100k components spread across SBOMs.
* **Security**: RBAC matrix tests; secret scanning; signing and verification.
* **Determinism**: same inputs produce identical `compiled_sha` and simulation summaries.
---
## 10) Feature flags
* `policy.studio` (enables editor and quick sim)
* `policy.batch-sim`
* `policy.canary-promotion`
* `policy.signature-required` (enforce signing on publish)
Flags documented in `/docs/observability/policy-telemetry.md`.
---
## 11) Non‑goals (this epic)
* Building a general IDE for arbitrary languages; the editor is purpose‑built for the DSL.
* Auto‑generated policies from AI without human approval.
* Cross‑tenant policies; all policies are tenant‑scoped.
---
## 12) Philosophy
* **Safety first**: it’s cheaper to prevent a bad policy than to fix its fallout.
* **Determinism**: same inputs, same outputs, verifiably.
* **Immutability**: versions and evidence are forever; we deprecate, not mutate.
* **Transparency**: every change is explainable with traces and proofs.
* **Reusability**: templates, snippets, and tests turn policy from art into engineering.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

431
docs/implplan/EPIC_5.md
View File

@@ -1,431 +0,0 @@
Here’s Epic 5 in the same paste‑into‑repo, implementation‑ready format as the prior epics. It’s exhaustive, formal, and designed to slot into AOC, Policy Engine, Conseiller/Excitator, and the Console.
---
# Epic 5: SBOM Graph Explorer
> Short name: **Graph Explorer**
> Services touched: **SBOM Service**, **Graph Indexer** (new), **Graph API** (new), **Policy Engine**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Web API Gateway**, **Authority** (authN/Z), **Workers/Scheduler**, **Telemetry**
> Surfaces: **Console (Web UI)** graph module, **CLI**, **Exports**
> Deliverables: Interactive graph UI with semantic zoom, saved queries, policy/VEX/advisory overlays, diff views, impact analysis, exports
---
## 1) What it is
**SBOM Graph Explorer** is the interactive, tenant‑scoped view of all supply‑chain relationships the platform knows about, rendered as a navigable graph. It connects:
* **Artifacts** (applications, images, libs), **Packages/Versions**, **Files/Paths**, **Licenses**, **Advisories** (from Conseiller), **VEX statements** (from Excitator), **Provenance** (builds, sources), and **Policies** (overlays of determinations)
* **Edges** like `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`
* **Time/version** dimension: multiple SBOM snapshots with diffs
It’s built for investigation and review: find where a vulnerable package enters; see which apps are impacted; understand why a finding exists; simulate a policy version and see the delta. The explorer observes **AOC enforcement**: it never mutates facts; it aggregates and visualizes them. Only the Policy Engine may classify, and classification is displayed as overlays.
---
## 2) Why
* SBOMs are graphs. Tables flatten what matters and hide transitive risk.
* Engineers, security, and auditors need impact answers quickly: “What pulls in `log4j:2.17` and where is it at runtime?”
* Policy/VEX/advisory interactions are nuanced. A visual overlay makes precedence and outcomes obvious.
* Review is collaborative; you need saved queries, deep links, exports, and consistent evidence.
---
## 3) How it should work (maximum detail)
### 3.1 Domain model
**Nodes** (typed, versioned, tenant‑scoped):
* `Artifact`: application, service, container image, library, module
* `Package`: name + ecosystem (purl), `PackageVersion` with resolved version
* `File`: path within artifact or image layer
* `License`: SPDX id
* `Advisory`: normalized advisory id (GHSA, CVE, vendor), source = Conseiller
* `VEX`: statement with product context, status, justification, source = Excitator
* `SBOM`: ingestion unit; includes metadata (tool, sha, build info)
* `PolicyDetermination`: materialized view of Policy Engine results (read‑only overlay)
* `Build`: provenance, commit, workflow run
* `Source`: repo, tag, commit
**Edges** (directed):
* `declared_in` (PackageVersion → SBOM)
* `contains` (Artifact → PackageVersion | File)
* `depends_on` (PackageVersion → PackageVersion) with scope attr (prod|dev|test|optional)
* `built_from` (Artifact → Build), `provenance_of` (Build → Source)
* `affected_by` (PackageVersion → Advisory) with range semantics
* `vex_exempts` (Advisory ↔ VEX) scoped by product/component
* `licensed_under` (Artifact|PackageVersion → License)
* `governs_with` (Artifact|PackageVersion → PolicyDetermination)
* `derived_from` (SBOM → SBOM) for superseding snapshots
**Identity & versioning**
* Every node has a stable key: `{tenant}:{type}:{natural_id}` (e.g., purl for packages, digest for images).
* SBOM snapshots are immutable; edges carry `valid_from`/`valid_to` for time travel and diffing.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.2 User capabilities (end‑to‑end)
* **Search & Navigate**: global search (purls, CVEs, repos, licenses), keyboard nav, breadcrumbs, semantic zoom.
* **Lenses**: toggle views (Security, License, Provenance, Runtime vs Dev, Policy effect).
* **Overlays**:
* **Advisory overlay**: show affected nodes/edges with source, severity, ranges.
* **VEX overlay**: show suppressions/justifications; collapse exempted paths.
* **Policy overlay**: choose a policy version; nodes/edges reflect determinations (severity, status) with explain sampling.
* **Impact analysis**: pick a vulnerable node; highlight upstream/downstream dependents, scope filters, shortest/all paths with constraints.
* **Diff view**: compare SBOM A vs B; show added/removed nodes/edges, changed versions, changed determinations.
* **Saved queries**: visual builder + JSON query; shareable permalinks scoped by tenant and environment.
* **Exports**: GraphML, CSV edge list, NDJSON of findings, PNG/SVG snapshot.
* **Evidence details**: side panel with raw facts, advisory links, VEX statements, policy explain trace, provenance.
* **Accessibility**: tab‑navigable, high‑contrast, screen‑reader labels for nodes and sidebars.
### 3.3 Query model
* **Visual builder** for common queries:
* “Show all paths from Artifact X to Advisory Y up to depth 6.”
* “All runtime dependencies with license = GPL‑3.0.”
* “All artifacts affected by GHSA‑… with no applicable VEX.”
* “Which SBOMs introduced/removed `openssl` between build 120 and 130?”
* **JSON query** (internal, POST body) with:
* `start`: list of node selectors (type + id or attributes)
* `expand`: edge types and depth, direction, scope filters
* `where`: predicates on node/edge attributes
* `overlay`: policy version id, advisory sources, VEX filters
* `limit`: nodes, edges, timebox, cost budget
* **Cost control**: server estimates cost, denies or pages results; UI streams partial graph tiles.
### 3.4 UI architecture (Console)
* **Canvas**: WebGL renderer with level‑of‑detail, edge bundling, and label culling; deterministic layout when possible (seeded).
* **Semantic zoom**:
* Far: clusters by artifact/repo/ecosystem, color by lens
* Mid: package groups, advisory badges, license swatches
* Near: concrete versions, direct edges, inline badges for policy determinations
* **Panels**:
* Left: search, filters, lens selector, saved queries
* Right: details, explain trace, evidence tabs (Advisory/VEX/Policy/Provenance)
* Bottom: query expression, diagnostics, performance/stream status
* **Diff mode**: split or overlay, color legend (add/remove/changed), filter by node type.
* **Deep links**: URL encodes query + viewport; shareable respecting RBAC.
* **Keyboard**: space drag, +/- zoom, F to focus, G to expand neighbors, P to show paths.
### 3.5 Back‑end architecture
**Graph Indexer (new)**
* Consumes SBOM ingests, Conseiller advisories, Excitator VEX statements, Policy Engine determinations (read‑only).
* Projects facts into a **property graph** persisted in:
* Primary: document store + adjacency sets (e.g., Mongo collections + compressed adjacency lists)
* Optional driver for graph DB backends if needed (pluggable)
* Maintains materialized aggregates: degree, critical paths cache, affected artifact counts, license distribution.
* Emits **graph snapshots** per SBOM with lineage to original ingestion.
**Graph API (new)**
* Endpoints for search, neighbor expansion, path queries, diffs, overlays, exports.
* Streaming responses for large graphs (chunked NDJSON tiles).
* Cost accounting + quotas per tenant.
**Workers**
* **Centrality & clustering** precompute on idle: betweenness approximations, connected components, Louvain clusters.
* **Diff compute** on new SBOM ingestion pairs (previous vs current).
* **Overlay materialization** cache for popular policy versions.
**Policy Engine integration**
* Graph API requests can specify a policy version.
* For sampled nodes, the API fetches explain traces; for counts, uses precomputed overlay materializations where available.
**AOC enforcement**
* Graph Indexer never merges or edits advisories/VEX; it links them and exposes overlays that the Policy Engine evaluates.
* Conseiller and Excitator remain authoritative sources; severities come from Policy‑governed normalization.
### 3.6 APIs (representative)
* `GET /graph/search?q=...&type=package|artifact|advisory|license`
* `POST /graph/query` ⇒ stream tiles `{nodes[], edges[], stats, cursor}`
* `POST /graph/paths` body: `{from, to, depth<=6, constraints{scope, runtime_only}}`
* `POST /graph/diff` body: `{sbom_a, sbom_b, filters}`
* `GET /graph/snapshots/{sbom_id}` ⇒ graph metadata, counts, top advisories
* `POST /graph/export` body: `{format: graphml|csv|ndjson|png|svg, query|snapshot}`
* `GET /graph/saved` / `POST /graph/saved` save and list tenant queries
* `GET /graph/overlays/policy/{version_id}` ⇒ summary stats for caching
All endpoints tenant‑scoped, RBAC‑checked. Timeouts and pagination by server. Errors return structured diagnostics.
### 3.7 CLI
```
stella sbom graph search "purl:pkg:maven/org.apache.logging.log4j/log4j-core"
stella sbom graph query --file ./query.json --export graphml > graph.graphml
stella sbom graph impacted --advisory GHSA-xxxx --runtime-only --limit 100
stella sbom graph paths --from artifact:service-a --to advisory:GHSA-xxxx --depth 5 --policy 1.3.0
stella sbom graph diff --sbom-a 2025-03-15T10:00Z --sbom-b 2025-03-22T10:00Z --export csv > diff.csv
stella sbom graph save --name "openssl-runtime" --file ./query.json
```
Exit codes: 0 ok, 2 query validation error, 3 over‑budget, 4 not found, 5 RBAC denied.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
### 3.8 Performance & scale
* **Progressive loading**: server pages tiles by BFS frontier; client renders incrementally.
* **Viewport culling**: only visible nodes/edges in canvas; offscreen demoted to aggregates.
* **Level‑of‑detail**: simplified glyphs and collapsed clusters at distance.
* **Query budgets**: per‑tenant rate + node/edge caps; interactive paths limited to depth ≤ 6.
* **Caching**: hot queries memoized per tenant + overlay version; diffs precomputed for consecutive SBOMs.
### 3.9 Security
* Multi‑tenant isolation at storage and API layers.
* RBAC roles:
* **Viewer**: browse graphs, saved queries
* **Investigator**: run queries, export data
* **Operator**: configure budgets, purge caches
* **Auditor**: download evidence bundles
* Input validation for query JSON; deny disallowed edge traversals; strict CSP in web app.
### 3.10 Observability
* Metrics: tile latency, nodes/edges per tile, cache hit rate, query denials, memory pressure.
* Logs: structured, include query hash, cost, truncation flags.
* Traces: server spans per stage (parse, plan, fetch, overlay, stream).
### 3.11 Accessibility & UX guarantees
* Keyboard complete, ARIA roles for graph and panels, high‑contrast theme.
* Deterministic layout on reload for shareable investigations.
### 3.12 Data retention
* Graph nodes derived from SBOMs share retention with SBOM artifacts; overlays are ephemeral caches.
* Saved queries retained until deleted; references to missing objects show warnings.
---
## 4) Implementation plan
### 4.1 Services
* **Graph Indexer (new microservice)**
* Subscribes to SBOM ingest events, Conseiller advisory updates, Excitator VEX updates, Policy overlay materializations.
* Builds adjacency lists and node documents; computes aggregates and clusters.
* **Graph API (new microservice)**
* Validates and executes queries; streams tiles; composes overlays; serves diffs and exports.
* Integrates with Policy Engine for explain sample retrieval.
* **SBOM Service (existing)**
* Emits ingestion events with SBOM ids and lineage; exposes SBOM metadata to Graph API.
* **Web API Gateway**
* Routes `/graph/*`, injects tenant context, enforces RBAC.
### 4.2 Console (Web UI) feature module
* `packages/features/graph-explorer`
* Canvas renderer (WebGL), panels, query builder, diff mode, overlays, exports.
* Deep‑link router and viewport state serializer.
### 4.3 Workers
* Centrality/clustering worker, diff worker, overlay materialization worker.
* Schedules on low‑traffic windows; backpressure aware.
### 4.4 Data model (storage)
* Collections:
* `graph_nodes`: `{_id, tenant, type, natural_id, attrs, degree, created_at, updated_at}`
* `graph_edges`: `{_id, tenant, from_id, to_id, type, attrs, valid_from, valid_to}`
* `graph_snapshots`: per‑SBOM node/edge references
* `graph_saved_queries`: `{_id, tenant, name, query_json, created_by}`
* `graph_overlays_cache`: keyed by `{tenant, policy_version, hash(query)}`
* Indexes: compound on `{tenant, type, natural_id}`, `{tenant, from_id}`, `{tenant, to_id}`, time bounds.
---
## 5) Documentation changes (create/update)
1. **`/docs/sbom/graph-explorer-overview.md`**
* Concepts, node/edge taxonomy, lenses, overlays, roles, limitations.
2. **`/docs/sbom/graph-using-the-console.md`**
* Walkthroughs: search, navigate, impact, diff, export; screenshots and keyboard cheatsheet.
3. **`/docs/sbom/graph-query-language.md`**
* JSON schema, examples, constraints, cost/budget rules.
4. **`/docs/sbom/graph-api.md`**
* REST endpoints, request/response examples, streaming and pagination.
5. **`/docs/sbom/graph-cli.md`**
* CLI command reference and example pipelines.
6. **`/docs/policy/graph-overlays.md`**
* How policy versions render in Graph; explain sampling; AOC guardrails.
7. **`/docs/vex/graph-integration.md`**
* How VEX suppressions appear and how to validate product scoping.
8. **`/docs/advisories/graph-integration.md`**
* Advisory linkage and severity normalization by policy.
9. **`/docs/architecture/graph-services.md`**
* Graph Indexer, Graph API, storage choices, failure modes.
10. **`/docs/observability/graph-telemetry.md`**
* Metrics, logs, tracing, dashboards.
11. **`/docs/runbooks/graph-incidents.md`**
* Handling runaway queries, cache poisoning, degraded render.
12. **`/docs/security/graph-rbac.md`**
* Permissions matrix, multi‑tenant boundaries.
Every doc should end with a “Compliance checklist.”
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 6) Tasks
### 6.1 Backend: Graph Indexer
* [ ] Define node/edge schemas and attribute dictionaries for each type.
* [ ] Implement event consumers for SBOM ingests, Conseiller updates, Excitator updates.
* [ ] Build ingestion pipeline that populates nodes/edges and maintains `valid_from/valid_to`.
* [ ] Implement aggregate counters and degree metrics.
* [ ] Implement clustering job and persist cluster ids per node.
* [ ] Implement snapshot materialization per SBOM and lineage tracking.
* [ ] Unit tests for each node/edge builder; property‑based tests for identity stability.
### 6.2 Backend: Graph API
* [ ] Implement `/graph/search` with prefix and exact match across node types.
* [ ] Implement `/graph/query` with validation, planning, cost estimation, and streaming tile results.
* [ ] Implement `/graph/paths` with constraints and depth limits; shortest path heuristic.
* [ ] Implement `/graph/diff` computing adds/removes/changed versions; stream results.
* [ ] Implement overlays: advisory join, VEX join, policy materialization and explain sampling.
* [ ] Implement exports: GraphML, CSV edge list, NDJSON findings, PNG/SVG snapshots.
* [ ] RBAC middleware integration; multi‑tenant scoping.
* [ ] Load tests with synthetic large SBOMs; define default budgets.
### 6.3 Policy Engine integration
* [ ] Add endpoint to fetch explain traces for specific node ids in batch.
* [ ] Add materialization export that Graph API can cache per policy version.
### 6.4 Console (Web UI)
* [ ] Create `graph-explorer` module with routes `/graph`, `/graph/diff`, `/graph/q/:id`.
* [ ] Implement WebGL canvas with LOD, culling, edge bundling, deterministic layout seed.
* [ ] Build search, filter, lens, and overlay toolbars.
* [ ] Side panels: details, evidence tabs, explain viewer.
* [ ] Diff mode: split/overlay toggles and color legend.
* [ ] Saved queries: create, update, run; deep links.
* [ ] Export UI: formats, server round‑trip, progress indicators.
* [ ] a11y audit and keyboard‑only flow.
### 6.5 CLI
* [ ] Implement `stella sbom graph *` subcommands with JSON IO and piping support.
* [ ] Document examples and stable output schemas for CI consumption.
### 6.6 Observability & Ops
* [ ] Dashboards for tile latency, query denials, cache hit rate, memory.
* [ ] Alerting on query error spikes, OOM risk, cache churn.
* [ ] Runbooks in `/docs/runbooks/graph-incidents.md`.
### 6.7 Docs
* [ ] Author all docs in section 5, link from Console contextual help.
* [ ] Add end‑to‑end tutorial: “Investigate GHSA‑XXXX across prod artifacts.”
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* Console renders large SBOM graphs with semantic zoom, overlays, and responsive interactions.
* Users can run impact and path queries with bounded depth and get results within budget.
* VEX suppressions and advisory severities appear correctly and are consistent with policy.
* Diff view clearly shows added/removed/changed nodes/edges between two SBOMs.
* Saved queries and deep links reproduce the same view deterministically (given same data).
* Exports produce valid GraphML/CSV/NDJSON and image snapshots.
* CLI supports search, query, paths, impacted, diff, and export with stable schemas.
* AOC guardrails: explorer never mutates facts; overlays reflect Policy Engine decisions.
* RBAC enforced; all actions logged and observable.
---
## 8) Risks & mitigations
* **Graph explosion on large monorepos** → tiling, clustering, budgets, and strict depth limits.
* **Inconsistent identities across tools** → canonicalize purls/digests; property‑based tests for identity stability.
* **Policy overlay latency** → precompute materializations for hot policy versions; sample explains only on focus.
* **User confusion** → strong lens defaults, deterministic layouts, legends, in‑context help.
---
## 9) Test plan
* **Unit**: node/edge builders, identity normalization, cost estimator.
* **Integration**: ingest SBOM + advisories + VEX, verify overlays and counts.
* **E2E**: Playwright flows for search→impact→diff→export; deep link determinism.
* **Performance**: simulate 500k nodes/2M edges; measure tile latency and memory.
* **Security**: RBAC matrix; tenant isolation tests; query validation fuzzing.
* **Determinism**: snapshot round‑trip: same query and seed produce identical layout and stats.
---
## 10) Feature flags
* `graph.explorer` (UI feature module)
* `graph.paths` (advanced path queries)
* `graph.diff` (SBOM diff mode)
* `graph.overlays.policy` (policy overlay + explain sampling)
* `graph.export` (exports enabled)
Documented in `/docs/observability/graph-telemetry.md`.
---
## 11) Non‑goals (this epic)
* Real‑time process/runtime call graphs.
* Full substitution for text reports; Explorer complements Reports.
* Cross‑tenant graphs; all queries are tenant‑scoped.
---
## 12) Philosophy
* **See the system**: security and license risk are structural. If you cannot see structure, you will miss risk.
* **Evidence over assertion**: every colored node corresponds to raw facts and explainable determinations.
* **Bounded interactivity**: fast, partial answers beat slow “complete” ones.
* **Immutability**: graphs mirror SBOM snapshots and are never rewritten; we add context, not edits.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

650
docs/implplan/EPIC_6.md
View File

@@ -1,650 +0,0 @@
Below is the expanded, “maximum documentation” package for Epic 6. It is paste‑ready for your repo and deliberately formal so engineering, docs, and audit folks can work from the same source of truth.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
# Epic 6: Vulnerability Explorer (policy‑aware)
**Short name:** `Vuln Explorer`
**Services touched:** Conseiller (Feedser), Excitator (Vexer), SBOM Service, Policy Engine, Findings Ledger (new), Web API Gateway, Authority (authN/Z), Workers/Scheduler, Telemetry/Analytics, Console (Web UI), CLI
**AOC ground rule:** Conseiller/Excitator aggregate but never merge or rewrite source documents. The Explorer only renders **effective** results as decided by Policy Engine and records human workflow as immutable ledger events.
---
## 1) What it is
The Vulnerability Explorer is the API and UI for operational triage, investigation, and reporting of vulnerabilities across all artifacts tracked by StellaOps. It correlates SBOM inventory with advisory and VEX evidence, then displays the **effective** status and obligations per the currently selected **Policy version**. It never edits source evidence. It provides:
* Policy‑aware lists, pivots, and detail views
* Triage workflow with immutable audit ledger
* Risk acceptance with expiry and evidence
* SLA tracking by severity and business tier
* Simulation against other policy versions
* Exports and cryptographically signed “evidence bundles”
**Identity principle:** one **Finding** per tuple `(artifact_id, purl, version, advisory_key)`, with links to every contributing AdvisoryEvidence and VexEvidence.
---
## 2) Why (concise)
* Prioritization must reflect **policy and VEX**, not raw feeds.
* Audit requires complete, reproducible lineage: what changed, why, and who decided.
* Operators need consistent APIs, CLI, and a UI that explain determinations, not just list CVEs.
---
## 3) How it should work (maximum detail)
### 3.1 Domain model and contracts
**Evidence (immutable)**
* `AdvisoryEvidence` (from Conseiller)
```json
{
"id": "adv_evd:...uuid...",
"tenant": "acme",
"source": "ghsa|nvd|vendor|ossindex|... ",
"source_id": "GHSA-xxxx",
"schema": "GHSA|CVE|CSAF-ADV",
"advisory_key": "CVE-2024-12345",
"affected": [{"ecosystem":"npm","purl":"pkg:npm/lodash","ranges":[{"type":"semver","events":[{"introduced":"0.0.0"},{"fixed":"4.17.21"}]}]}],
"cvss": {"version":"3.1","baseScore":7.5,"vectorString":"AV:N/AC:L/..."},
"severity": "HIGH",
"urls": ["https://..."],
"published": "2024-06-10T12:00:00Z",
"withdrawn": null,
"ingested_at": "2024-06-11T08:43:21Z"
}
```
* `VexEvidence` (from Excitator)
```json
{
"id": "vex_evd:...uuid...",
"tenant": "acme",
"source": "vendor-csaf",
"schema": "CSAF-VEX",
"advisory_key": "CVE-2024-12345",
"product_scope": [{"purl":"pkg:npm/lodash@4.17.20"}],
"status": "not_affected|affected|fixed|under_investigation",
"justification": "component_not_present|vulnerable_code_not_in_execute_path|... ",
"impact_statement": "Not reachable in Acme Payment Service.",
"timestamp": "2024-06-12T10:00:00Z",
"ingested_at": "2024-06-12T10:10:02Z"
}
```
* `InventoryEvidence` (from SBOM Service)
```json
{
"id": "inv_evd:...uuid...",
"tenant": "acme",
"artifact_id": "svc:payments@1.14.0",
"sbom_id": "sbom:sha256:...",
"purl": "pkg:npm/lodash@4.17.20",
"scope": "runtime|dev|test",
"runtime_flag": true,
"paths": [["root", "pkg:npm/a", "pkg:npm/lodash"]],
"discovered_at": "2024-06-12T11:00:00Z"
}
```
**PolicyDetermination** (read‑only from Policy Engine)
```json
{
"key": {
"artifact_id": "svc:payments@1.14.0",
"purl": "pkg:npm/lodash",
"version": "4.17.20",
"advisory_key": "CVE-2024-12345",
"policy_version": "1.3.0"
},
"applicable": true,
"effective_severity": "HIGH",
"exploitability": "ACTIVE|LIKELY|UNKNOWN|UNLIKELY",
"signals": {"epss": 0.86, "kev": true, "maturity": "weaponized"},
"suppression_state": "none|policy|vex",
"obligations": [{"type":"fix_by","due":"2024-07-15"},{"type":"document_risk"}],
"sla": {"due":"2024-07-15","tier":"gold"},
"rationale": [
{"rule":"sev.base=nvd","detail":"NVD base:7.5"},
{"rule":"exploit.upsell.kev","detail":"KEV flag → HIGH"},
{"rule":"env.weight.prod","detail":"Env=prod no downgrade"}
]
}
```
**Finding identity**
```
finding_id = hash(tenant, artifact_id, purl, version, advisory_key)
```
**Ledger event** (append‑only, tamper‑evident)
```json
{
"event_id": "led:...uuid...",
"finding_id": "f:...hash...",
"tenant": "acme",
"type": "assign|comment|attach|change_state|accept_risk|set_target_fix|verify_fix|reopen",
"payload": {"to":"team:platform","reason":"oncall triage"},
"actor": {"user_id":"u:42","display":"Dana S."},
"ts": "2024-06-12T14:01:02Z",
"prev_event_hash": "sha256:...",
"event_hash": "sha256:sha256(canonical_json(event) + prev_event_hash)"
}
```
**Projection** (materialized current state for fast lists)
```json
{
"finding_id":"f:...hash...",
"tenant":"acme",
"artifact_id":"svc:payments@1.14.0",
"purl":"pkg:npm/lodash",
"version":"4.17.20",
"advisory_key":"CVE-2024-12345",
"effective_severity":"HIGH",
"exploitability":"ACTIVE",
"suppression_state":"none",
"status":"UNDER_INVESTIGATION|REMEDIATING|ACCEPTED|RESOLVED|NEW|SUPPRESSED",
"sla_due":"2024-07-15",
"owner":"team:platform",
"kev":true,
"epss":0.86,
"new_since":"2024-06-12",
"has_fix":true,
"envs":["prod"],
"runtime_flag":true,
"updated_at":"2024-06-12T14:01:02Z"
}
```
**Advisory key normalization**
* Input identifiers: `CVE-*`, `GHSA-*`, vendor IDs.
* Preference order: CVE, then GHSA, else vendor id prefixed with namespace.
* Canonicalization: uppercase, trim, map withdrawn to same key but mark `withdrawn=true` in evidence.
* Conseiller must publish `links: [all source ids]` for provenance.
### 3.2 Resolver algorithm (candidate findings)
**Goal:** produce tuples `(artifact_id, purl, version, advisory_key)` where inventory intersects affected ranges and policy deems the path relevant.
**Pseudocode**
```
for each artifact sbom S:
inv = inventory(S)
for each advisory evidence A:
for each affected package spec in A.affected:
for each inv_item in inv where inv_item.purl package == spec.purl package:
if version_in_ranges(inv_item.version, spec.ranges, ecosystem):
if policy.path_scope_allows(inv_item.scope, inv_item.runtime_flag, inv_item.paths):
yield candidate (artifact_id, inv_item.purl, inv_item.version, A.advisory_key)
```
**Version semantics per ecosystem**
* npm: semver, pre-release excluded unless explicitly in range.
* Maven: Maven version rules, handle `-SNAPSHOT`, use maven‑resolver semantics.
* PyPI: PEP 440 versioning.
* Go: semver with `+incompatible` handling.
* OS packages: RPM/DEB epoch:version‑release ordering.
**Edge cases**
* Multiple paths: store shortest path and count.
* Dev/test scope: policy may exclude or downgrade.
* Withdrawn advisories: keep as evidence; Policy can set severity to `NONE`.
### 3.3 VEX precedence and scoping
* If any matching **VEX** says `not_affected` scoped to the artifact product/component per CSAF product tree, set `suppression_state="vex"` and `applicable=false`.
* If VEX says `fixed` and inventory version is >= fixed version, mark as **Resolved (verified)** after SBOM recrawl confirms.
* If VEX `under_investigation`, no suppression; may add a policy grace period obligation.
### 3.4 Policy evaluation
* Inputs: candidate tuple + context (artifact env, business tier, ownership, signals, fix availability).
* Determinations: applicability, effective severity, exploitability, obligations, SLA, suppression.
* Suppression by policy examples: test‑scope only; path through optional deps; package vendored but not linked at runtime.
* Simulation: identical input, alternate `policy_version`; returns determinations without side effects.
### 3.5 API surface (authoritative)
**List**
```
GET /vuln/findings?policy=1.3.0&sev=high,critical&group_by=artifact&exploit=kev&env=prod&page=1&page_size=100
```
Response
```json
{
"page": 1,
"page_size": 100,
"total": 740,
"group_by": "artifact",
"results": [
{"group":"svc:payments","counts":{"CRITICAL":3,"HIGH":12,"MEDIUM":8},"sla_breaches":2},
...
]
}
```
**Query (complex filters)**
```
POST /vuln/findings/query
```
```json
{
"policy": "1.3.0",
"filter": {
"severity": [">=MEDIUM"],
"exploit": ["kev", "epss>=0.8"],
"artifact": ["svc:payments", "svc:checkouts"],
"status": ["NEW","UNDER_INVESTIGATION"],
"env": ["prod"]
},
"sort": [{"field":"effective_severity","dir":"desc"},{"field":"epss","dir":"desc"}],
"page": {"number":1,"size":200}
}
```
**Detail**
```
GET /vuln/findings/{finding_id}?policy=1.3.0
```
Returns projection, evidence links, policy rationale, paths, history summary.
**Workflow**
```
POST /vuln/findings/{id}/assign { "to": "team:platform" }
POST /vuln/findings/{id}/comment { "text": "triage notes..." }
POST /vuln/findings/{id}/accept-risk { "until":"2025-06-30","reason":"vendor patch pending","evidence":["url|upload_id"] }
POST /vuln/findings/{id}/verify-fix { "sbom_id": "sbom:sha256:..." }
POST /vuln/findings/{id}/target-fix { "version": "4.17.21" }
```
**Simulation**
```
POST /vuln/simulate
{
"policy_from": "1.3.0",
"policy_to": "1.4.0",
"query": { "severity":[">=MEDIUM"], "env":["prod"] }
}
```
Response includes per‑finding delta `{before, after, diff}`.
**Export**
```
POST /vuln/export { "format":"ndjson","scope":{"query":{...}} }
```
Returns a signed bundle (see §3.10).
**Errors**
* `400` validation, `403` RBAC, `404` not found, `409` state conflict (idempotency), `429` rate limited, `5xx` server.
### 3.6 Console (Web UI)
**Routes**
* `/vuln` list with saved views
* `/vuln/:id` detail drawer state
* `/vuln/simulate/:policyVersion` diff mode
**State shape (client)**
```ts
interface VulnListState {
policyVersion: string;
filters: {...};
sort: [...];
columns: string[];
viewId?: string;
page: {number: number; size: number};
}
```
**UX**
* Virtualized grid with server paging; column chooser; density toggle.
* Quick filters: severity, exploit signals, status, env, owner, fix availability.
* Detail tabs: Summary, Evidence (raw docs with provenance), Policy (rationale chain), Paths (deep link to Graph Explorer), Fixes, History.
* Simulation bar shows delta chips: `+21 HIGH`, `-9 Suppressed by VEX` etc.
* Evidence bundle dialog previews scope and size.
* a11y: ARIA roles on grid, keyboard shortcuts: `A` assign, `C` comment, `R` accept risk, `V` verify fix.
### 3.7 CLI
Commands
```
stella vuln list --policy 1.3.0 --sev high,critical --group-by artifact --env prod --json
stella vuln show --id <finding-id> --policy 1.3.0
stella vuln simulate --from 1.3.0 --to 1.4.0 --sev '>=medium' --delta --json
stella vuln assign --filter 'advisory:CVE-2024-12345 artifact:payments' --to team:platform
stella vuln accept-risk --id <finding-id> --until 2025-06-30 --reason "vendor patch pending" --evidence url:https://ticket/123
stella vuln verify-fix --id <finding-id> --sbom <sbom-id>
```
Return codes: `0 ok`, `2 invalid args`, `3 budget exceeded`, `4 not found`, `5 denied`.
### 3.8 Storage schema (illustrative)
**Tables**
```sql
-- Evidence
CREATE TABLE evidence_advisory (...);
CREATE INDEX ea_tenant_key ON evidence_advisory(tenant, advisory_key);
CREATE TABLE evidence_vex (...);
CREATE INDEX ev_tenant_key ON evidence_vex(tenant, advisory_key);
CREATE TABLE evidence_inventory (...);
CREATE INDEX ei_artifact_purl ON evidence_inventory(tenant, artifact_id, purl);
-- Ledger
CREATE TABLE findings_ledger_events (
event_id uuid PRIMARY KEY,
finding_id bytea NOT NULL,
tenant text NOT NULL,
type text NOT NULL,
payload jsonb NOT NULL,
actor jsonb NOT NULL,
ts timestamptz NOT NULL,
prev_event_hash bytea,
event_hash bytea NOT NULL
);
CREATE INDEX fle_find_ts ON findings_ledger_events(tenant, finding_id, ts);
-- Projection
CREATE TABLE findings_projection (
finding_id bytea PRIMARY KEY,
tenant text NOT NULL,
artifact_id text NOT NULL,
purl text NOT NULL,
version text NOT NULL,
advisory_key text NOT NULL,
policy_version text NOT NULL,
effective_severity text NOT NULL,
exploitability text,
suppression_state text,
status text NOT NULL,
sla_due date,
owner text,
kev boolean,
epss double precision,
envs text[],
runtime_flag boolean,
updated_at timestamptz NOT NULL
);
CREATE INDEX fp_query ON findings_projection(tenant, policy_version, effective_severity, status);
```
**Tamper‑evidence**
* Ledger events use chained SHA‑256 hashes over canonical JSON + previous hash.
* Daily Merkle root of all event hashes is anchored to the audit store (and optionally external timestamping service).
### 3.9 Performance and scaling
* P95 list endpoint under 600 ms for 100‑row pages at 5M findings/tenant.
* Projections denormalize heavy joins; background projector uses idempotent jobs keyed by `(tenant,finding_id,policy_version)`.
* Rate limits per tenant and per API key; backpressure on export jobs; exponential retry for projector.
### 3.10 Evidence bundle format
* **Container:** ZIP with `manifest.json`, `findings.ndjson`, `advisory_evidence.ndjson`, `vex_evidence.ndjson`, `inventory_evidence.ndjson`, `policy_version.json`, `ledger_events.ndjson`, `CHECKSUMS`.
* **Signing:** Detached signature `bundle.sig` using tenant’s org key (Ed25519).
* **Manifest**
```json
{"generated_at":"2024-06-12T15:00:00Z","tenant":"acme","policy_version":"1.3.0","scope":{"query":{...}},"counts":{"findings":421}}
```
### 3.11 Observability
* Metrics (OpenTelemetry):
* `vuln_findings_list_latency_ms` (histogram)
* `vuln_projection_lag_seconds` (gauge)
* `vuln_new_findings_total` (counter)
* `vuln_sla_breaches_total` (counter by sev, owner)
* `vuln_simulation_latency_ms` (histogram)
* Logs: structured JSON with `tenant`, `policy_version`, `query_hash`, `result_count`.
* Traces: spans for resolver, policy calls, projection builds, export assembly.
* PII: redact comments in logs; store attachments encrypted at rest (KMS).
### 3.12 Security & RBAC
**Roles**
* Viewer: GET list/detail/export read scope.
* Investigator: Viewer + workflow actions except risk acceptance.
* Operator: Investigator + risk acceptance, verify fix, bulk actions.
* Auditor: Viewer + evidence bundles and ledger integrity checks.
**ABAC**
* Attribute constraints: by `artifact.owner`, `env`, and `business_tier`.
* CSRF protection for Console; all POST require anti‑forgery tokens.
* Attachments stored with envelope encryption; signed URLs for limited time access.
### 3.13 Rollout and migrations
* Feature flags: `vuln.explorer.ui`, `vuln.explorer.simulation`, `vuln.explorer.bulk_actions`, `vuln.explorer.evidence_bundle`.
* Phase 1: dark launch API and projections.
* Phase 2: UI read‑only list and detail.
* Phase 3: workflow actions and exports.
* Data backfill: replay advisory/VEX/SBOM events to seed projections.
* Compatibility: maintain projection v1 schema for two releases; migration scripts in `/migrations/vuln/`.
---
## 4) Implementation plan
### 4.1 Services
* **Findings Ledger (new)**
* Append‑only event store with projector to `findings_projection`.
* Event validation and canonicalization; hashing and Merkle root anchoring.
* **Vuln Explorer API (new)**
* Query/filter engine with policy parameterization and grouping.
* Simulation endpoint.
* Export job orchestrator.
* **Conseiller / Excitator (updates)**
* Guarantee canonical `advisory_key` and publish `links[]`.
* No merges; maintain raw payload snapshots.
* **Policy Engine (updates)**
* Batch evaluation endpoint `POST /policy/eval/batch` with `simulate` support.
* Return rationale chain with rule IDs.
* **SBOM Service (updates)**
* Publish inventory deltas; include `scope`, `runtime_flag`, `paths`.
* Nearest safe version hints.
* **Workers/Scheduler**
* Resolver job keyed by `(tenant, artifact_id, sbom_id)`; emits candidate tuples.
* Recompute on policy activation and evidence changes.
### 4.2 Code structure
```
/src/Findings/StellaOps.Findings.Ledger
/api
/projector
/storage
/src/VulnExplorer/StellaOps.VulnExplorer.Api
/routes
/query
/simulation
/export
/packages/console/features/vuln-explorer
/components
/pages
/state
/src/Cli/StellaOps.Cli
```
### 4.3 Performance tasks
* Projection indexes and covering queries; explain plans in `/docs/vuln/perf-notes.md`.
* Cache hot groupings per tenant with TTL and invalidation on ledger projector tick.
---
## 5) Documentation changes (create/update)
1. `/docs/vuln/explorer-overview.md`
Conceptual model, identities, evidence vs determinations, AOC guarantees.
2. `/docs/vuln/explorer-using-console.md`
Workflows with screenshots, keyboard shortcuts, saved views, deep links.
3. `/docs/vuln/explorer-api.md`
Endpoint specs, query language, grouping, pagination, errors, rate limits.
4. `/docs/vuln/explorer-cli.md`
Commands, flags, examples, exit codes.
5. `/docs/vuln/findings-ledger.md`
Event schema, state machine, hashing, Merkle roots, integrity checks.
6. `/docs/policy/vuln-determinations.md`
Inputs, outputs, precedence rules, simulation semantics.
7. `/docs/vex/explorer-integration.md`
CSAF mapping, scoping to product tree, precedence.
8. `/docs/advisories/explorer-integration.md`
Advisory key normalization, provenance, withdrawn handling.
9. `/docs/sbom/vuln-resolution.md`
Ecosystem version semantics, path sensitivity, scope rules.
10. `/docs/observability/vuln-telemetry.md`
Metrics, logs, traces, dashboards, SLOs.
11. `/docs/security/vuln-rbac.md`
Role mapping, ABAC, attachment encryption, CSRF.
12. `/docs/runbooks/vuln-ops.md`
Recompute storms, projector lag, policy activation drains, export failures.
13. `/docs/install/containers.md`
Add `findings-ledger`, `vuln-explorer-api` images, compose/k8s manifests, resource sizing, health checks.
> Each doc ends with: **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 6) Engineering tasks
### Backend: Findings & API
* [ ] Define evidence and ledger schemas; migrations.
* [ ] Implement resolver for npm, Maven, PyPI, Go, OS packages with property‑based tests for version comparisons.
* [ ] Implement ledger API and projector with idempotency and hashing.
* [ ] Implement list/detail/grouping endpoints with server‑side paging.
* [ ] Implement simulation and export (bundle assembly, signing).
* [ ] Integrate Policy Engine batch eval with rationale traces.
* [ ] RBAC via Authority with ABAC filters.
* [ ] Load tests at 5M findings/tenant; tune indexes.
### Conseiller/Excitator
* [ ] Normalize `advisory_key` and persist `links[]`.
* [ ] Ensure raw payload snapshots are retrievable by Explorer for Evidence tab.
### SBOM Service
* [ ] Emit `scope`, `runtime_flag`, `paths`; safe version hints.
* [ ] Inventory delta events to trigger resolver.
### Console
* [ ] Build grid with virtualization, saved views, deep link serializer.
* [ ] Implement detail tabs and path deep‑links to Graph Explorer.
* [ ] Add simulation bar and delta chips.
* [ ] Evidence bundle dialog.
* [ ] a11y keyboard flow and ARIA labeling; unit and E2E tests.
### CLI
* [ ] `stella vuln list|show|simulate|assign|accept-risk|verify-fix` with `--json` and CSV export.
* [ ] Stable output schemas; pipe‑friendly defaults.
### Observability/Ops
* [ ] Dashboards for list latency, projection lag, new/reopened, SLA breaches.
* [ ] Alerts on projector backlog, API 5xx spikes, export failures.
* [ ] Runbooks in `/docs/runbooks/vuln-ops.md`.
### Docs
* [ ] Author files listed in §5 with cross‑links to Policy Studio and SBOM Graph Explorer.
* [ ] Update `/docs/install/containers.md` with new images and compose/k8s snippets.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* List and detail views reflect **effective** policy outcomes and update instantly when switching policy versions.
* Evidence tab shows all raw advisory/VEX documents with provenance; no source merging.
* Resolver respects ecosystem semantics, scope, and paths; path tab round‑trips to Graph Explorer.
* Ledger events are immutable and reconstruct historical list states accurately.
* Simulation returns diffs without side effects and matches Policy Engine outputs.
* CLI/API support paging, grouping, export, simulation; contracts stable and documented.
* RBAC and tenant isolation validated by tests; attachments encrypted.
* P95 performance budgets met; dashboards green for SLOs.
---
## 8) Risks and mitigations
* **Advisory identity collisions** → strict canonicalization; preserve `links[]`; never merge raw docs.
* **Projection lag** → backpressure, worker autoscaling, health checks; alerting on lag.
* **Resolver false positives** → path evidence required; dev/test scope rules explicit; ecosystem‑specific tests.
* **User confusion over suppression** → explicit badges; Policy tab with rationale and “why changed.”
* **Export size** → NDJSON streaming, size estimator in UI, scope previews.
---
## 9) Test plan
* **Unit:** version comparators, resolver per ecosystem, policy mapping, ledger state machine.
* **Integration:** SBOM + advisories + VEX ingestion, candidate generation, policy application, suppression precedence.
* **E2E Console:** triage, bulk assign, simulation, evidence bundle download; keyboard‑only flow.
* **Performance:** list/grouping at target scale; projector rebuild; export assembly.
* **Security:** RBAC matrix, ABAC filters, CSRF, signed URL lifetimes, tamper‑evidence verification.
* **Determinism:** time‑travel snapshots reproduce prior states byte‑for‑byte.
---
## 10) Philosophy
* **Facts first, decisions second.** Evidence is immutable; decisions and workflow sit on top in a ledger.
* **Policy is the lens.** The same facts can imply different obligations; the system must make that explicit and reproducible.
* **Audit > convenience.** Every state change is justified, signed, and verifiable.
* **No hidden magic.** If anything is suppressed, the UI shows the rule or VEX that did it, with documents attached.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

545
docs/implplan/EPIC_7.md
View File

@@ -1,545 +0,0 @@
Below is the expanded, “maximum documentation” package for Epic 7. It is paste‑ready for your repo and deliberately formal so engineering, docs, governance, and audit can work from the same source of truth.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
# Epic 7: VEX Consensus Lens
**Short name:** `VEX Lens`
**Services touched:** Excitator (Vexer), Conseiller (Feedser), SBOM Service, Policy Engine, Findings Ledger, Web API Gateway, Authority (authN/Z), Console (Web UI), CLI, Telemetry/Analytics
**AOC ground rule:** Excitator and Conseiller aggregate but never merge or rewrite source documents. The Lens only computes a **derived, reproducible** consensus view while preserving all raw evidence and provenance.
---
## 1) What it is
The VEX Consensus Lens is a deterministic computation and presentation layer that ingests all available VEX statements about a given advisory key (e.g., CVE) and product/context, then produces an **explicit consensus state** for each `(artifact_id, purl, version, advisory_key)` tuple. It answers: “Given multiple, possibly conflicting VEX statements from different issuers, what is the most trustworthy, policy‑consistent interpretation for this artifact and version?”
It never edits or merges VEX documents. It normalizes them, aligns their **product scope** to SBOM inventory, scores issuers via a tenant‑configurable **trust model**, applies time and version scoping, and outputs:
* `consensus_state`: `NOT_AFFECTED | AFFECTED | FIXED | UNDER_INVESTIGATION | INCONCLUSIVE | DISPUTED`
* `confidence`: 0.0–1.0 numeric score
* `rationale`: structured explanation of evidence, weights, and rules applied
* `quorum`: weighted vote breakdown by issuer, timestamp, justification, and version applicability
The Lens plugs into the Policy Engine. Policy remains the source of truth for applicability, severity, and obligations; the Lens supplies structured signals and provenance that policy can use to suppress findings, downgrade risk, or set SLAs.
Key properties:
* **Reproducible:** Same inputs and policy yield same outputs.
* **Explainable:** Every output carries a machine‑readable rationale chain.
* **Scoped:** Product trees and version ranges are resolved against actual SBOM inventory.
* **Immutable evidence:** All raw VEX docs remain intact, retrievable, and linkable.
---
## 2) Why (concise)
Real ecosystems have overlapping VEX statements from upstream maintainers, vendors, distros, and third parties. Operators need a single, defensible view without losing provenance. The Lens reduces noise, flags conflicts, and turns fragmented VEX evidence into auditable signals that policy and humans can act on.
---
## 3) How it should work (maximum detail)
### 3.1 Input model
**From Excitator (Vexer):**
* Raw `VexEvidence` documents (CSAF VEX, OpenVEX, CycloneDX VEX) with:
* `advisory_key` (canonicalized)
* product tree or component coordinates (purl, CPE, vendor IDs)
* status (`not_affected`, `affected`, `fixed`, `under_investigation`)
* justifications (CSAF, OpenVEX enumerations)
* version ranges or fixed versions if available
* timestamps (issued, last_updated), source URLs
* cryptographic metadata (signature, issuer, certificate chain) if present
**From SBOM Service:**
* Inventory for each artifact: purls, versions, dependency paths, scopes, env flags.
**From Issuer Directory (new)**
* Directory of known VEX issuers:
* identity, organization, domain, CSAF publisher metadata
* public keys and trust anchors (Ed25519/X.509/PKIX/PKCS7/DSSE)
* default trust weight
* tenancy‑specific overrides
**From Policy Engine:**
* Policy version and options relevant to VEX evaluation:
* trust model parameters
* confidence thresholds per environment/tier
* justification whitelist/blacklist
* recency requirements and expiry windows
* precedence rules for status conflicts
### 3.2 Normalization
1. **Canonical advisory key:** CVE preferred; GHSA mapped; vendor IDs namespaced.
2. **Status mapping:** Normalize all encodings to `NOT_AFFECTED | AFFECTED | FIXED | UNDER_INVESTIGATION`.
3. **Product scope alignment:** Convert CSAF product tree/CPE to purl variants using deterministic mappings; store mapping evidence.
4. **Version scoping:** For each evidence, compute `applies_to(version)` using ecosystem comparators (npm semver, Maven, PEP 440, Go semver, RPM/DEB EVR).
5. **Signature verification:** If signed, verify against Issuer Directory trust anchors; attach `sig_verified=true/false` and chain metadata.
6. **Temporal fields:** Compute `effective_at` (issued), `observed_at` (ingested), and staleness.
### 3.3 Trust model
Each `VexEvidence` gets a base weight `w_base` from issuer type and verification:
* Maintainer/Upstream signed: 0.9
* Maintainer/Upstream unsigned: 0.7
* Distro/SIG signed: 0.8
* Vendor of downstream product signed: 0.8
* Third‑party scanner VEX: 0.4
* Unknown/unsigned with weak provenance: 0.2
Weights are then adjusted:
```
w = w_base
* f_signature(sig_verified) // e.g., 1.1 if verified, 0.8 if unverifiable
* f_recency(age_days) // decay after T days (policy)
* f_justification(type) // e.g., "component_not_present" lower if SBOM shows present
* f_scope_match(score) // quality of product match: exact purl > family > CPE wildcard
* f_env(app_env) // optional env-specific multipliers
```
All `f_*` are bounded to [0.5, 1.2] by default to prevent runaway effects. Tenants can override per Policy Studio.
### 3.4 Consensus algorithm
For a tuple `(artifact_id, purl, version, advisory_key)`:
1. **Collect** all normalized `VexEvidence` whose product scope maps to `purl` and `applies_to(version)==true`.
2. **Bucket** by normalized status; compute `W(status) = Σ w(evidence)` per status.
3. **Apply precedence** rules from policy:
* If `W(NOT_AFFECTED)` exceeds threshold `T_na` and there is no `FIXED` evidence contradictory for the same version, propose `NOT_AFFECTED`.
* If any `FIXED` applies and inventory version ≥ fixed version, propose `FIXED`.
* If `W(AFFECTED)` ≥ `T_aff` and no dominating `NOT_AFFECTED`, propose `AFFECTED`.
* If `W(UNDER_INVESTIGATION)` dominates and others below thresholds, propose `UNDER_INVESTIGATION`.
* If both `AFFECTED` and `NOT_AFFECTED` exceed thresholds within a small margin `δ`, mark `DISPUTED`.
* Otherwise `INCONCLUSIVE`.
4. **Confidence score:**
```
confidence = W(winning_status) / (W(AFFECTED)+W(NOT_AFFECTED)+W(FIXED)+W(UNDER_INVESTIGATION) + ε)
```
Clip to [0.0, 1.0].
5. **Rationale chain:**
* Winning status, thresholds used, top 5 contributing evidences with weights and reasons, product mapping quality, version scoping evidence, and policy knobs that influenced the result.
6. **Quorum summary:**
* List issuers and their votes, signature state, timestamps, and justifications.
### 3.5 Policy interaction
* The Lens returns `consensus_state`, `confidence`, and structured `signals`.
* Policy Engine consumes these and may:
* Suppress findings automatically on `NOT_AFFECTED` confidence ≥ `P_na`.
* Downgrade severity or extend SLA when `UNDER_INVESTIGATION` from high‑trust issuers.
* Require human approval when `DISPUTED`.
* Treat `FIXED` as resolved only when SBOM crawl verifies the fixed version or a `verify_fix` ledger event exists.
**Simulation:** Policy Studio can simulate different trust weights and thresholds and see consensus deltas without side effects.
### 3.6 Data contracts
**ConsensusRecord (materialized)**
```json
{
"id": "cons:sha256(tenant|artifact|purl|version|advisory|policy)",
"tenant": "acme",
"artifact_id": "svc:payments@1.14.0",
"purl": "pkg:npm/lodash@4.17.20",
"advisory_key": "CVE-2024-12345",
"policy_version": "1.3.0",
"consensus_state": "NOT_AFFECTED",
"confidence": 0.87,
"weights": {"AFFECTED":0.31,"NOT_AFFECTED":1.25,"FIXED":0.00,"UNDER_INVESTIGATION":0.12},
"top_evidence": ["vex_evd:...","vex_evd:..."],
"quorum": [
{"issuer":"lodash-maintainers","status":"NOT_AFFECTED","w":0.9,"sig_verified":true,"just":"vulnerable_code_not_present","issued":"2024-06-08"},
{"issuer":"vendorX-distro","status":"AFFECTED","w":0.25,"sig_verified":false,"just":"generic_advisory","issued":"2024-06-07"}
],
"rationale": [
{"rule":"trust.weight.issuer","detail":"maintainer signed evidence 0.9"},
{"rule":"scope.match.exact","detail":"exact purl match"},
{"rule":"justification.vcnp","detail":"supported by SBOM callgraph hint"}
],
"updated_at": "2024-06-12T10:00:00Z"
}
```
**Issuer Directory entry**
```json
{
"issuer_id": "iss:lodash",
"org": "Lodash Maintainers",
"domains": ["lodash.com"],
"keys": [{"type":"ed25519","pub":"...","expires":"2026-12-31"}],
"default_weight": 0.9,
"metadata": {"csaf_publisher": true}
}
```
### 3.7 APIs
**Compute/Query**
```
GET /vex/consensus?artifact=svc:payments@1.14.0&purl=pkg:npm/lodash@4.17.20&advisory=CVE-2024-12345&policy=1.3.0
POST /vex/consensus/query { "policy":"1.3.0", "filter": { "state":["DISPUTED","INCONCLUSIVE"], "confidence":"<0.6", "env":["prod"] }, "page":{...} }
GET /vex/consensus/{id} // full record
POST /vex/consensus/simulate // override trust knobs and thresholds for what-if
```
**Issuer Directory**
```
GET /vex/issuers
POST /vex/issuers (admin)
POST /vex/issuers/{id}/keys (admin)
```
**Exports**
```
POST /vex/consensus/export { "format":"ndjson","scope":{"filter":{...}} }
```
**Errors**
* `400` invalid mapping, `403` RBAC, `404` not found, `409` conflict, `429` rate limit.
### 3.8 Console (Web UI)
Routes:
* `/vex/consensus` overview with filters: state, confidence, issuer, advisory, artifact, env.
* `/vex/consensus/:id` detail: Evidence pane, Quorum graph, Policy impact, SBOM path links.
UX elements:
* **Quorum bar:** stacked bar showing weights per status; hover reveals issuer contributions.
* **Confidence chip:** numeric and qualitative band (Low/Med/High).
* **Evidence table:** paged list of VEX docs with signature icon, scope match quality tag, justification tag, issued/updated timestamps.
* **Conflict view:** for `DISPUTED`, show side-by-side issuer rationales and suggested next steps.
* **Deep link** into Vulnerability Explorer detail, preselecting the Policy version used for the consensus.
A11y:
* ARIA roles on grid and bars; keyboard shortcuts `S` switch policy, `T` trust presets, `E` export.
### 3.9 CLI
Commands:
```
stella vex consensus list --policy 1.3.0 --state disputed,inconclusive --confidence '<0.6' --artifact payments --json
stella vex consensus show --id <cons-id> --policy 1.3.0
stella vex simulate --policy 1.3.0 --trust 'issuer:lodash=1.0,issuer:vendorX=0.5' --thresholds 'na=1.0,aff=0.6' --json
stella vex issuers list
stella vex export --filter 'artifact:payments advisory:CVE-2024-12345' --out vex-consensus.ndjson
```
Exit codes: `0` ok, `2` invalid args, `4` not found, `5` denied.
### 3.10 Storage schema (illustrative)
```sql
-- Normalized VEX (reference Excitator id; do not alter raw)
CREATE TABLE vex_normalized (
id uuid PRIMARY KEY,
tenant text NOT NULL,
evidence_id text NOT NULL, -- link to Excitator
advisory_key text NOT NULL,
issuer_id text,
status text NOT NULL, -- NOT_AFFECTED|AFFECTED|FIXED|UNDER_INVESTIGATION
justification text,
purl text, -- normalized mapping target
version_range jsonb, -- ecosystem-specific encoding
sig_verified boolean,
scope_score real, -- 0..1 quality of mapping
issued timestamptz,
updated timestamptz,
w_base real,
UNIQUE (tenant, evidence_id)
);
-- Issuer Directory
CREATE TABLE vex_issuers (
issuer_id text PRIMARY KEY,
tenant text NOT NULL,
org text NOT NULL,
default_weight real NOT NULL,
metadata jsonb,
UNIQUE(tenant, org)
);
CREATE TABLE vex_issuer_keys (
id uuid PRIMARY KEY,
issuer_id text NOT NULL REFERENCES vex_issuers(issuer_id),
key_type text NOT NULL,
pubkey text NOT NULL,
expires timestamptz
);
-- Consensus projection
CREATE TABLE vex_consensus (
id bytea PRIMARY KEY,
tenant text NOT NULL,
artifact_id text NOT NULL,
purl text NOT NULL,
version text NOT NULL,
advisory_key text NOT NULL,
policy_version text NOT NULL,
consensus_state text NOT NULL,
confidence real NOT NULL,
weights jsonb NOT NULL,
top_evidence text[] NOT NULL,
updated_at timestamptz NOT NULL
);
CREATE INDEX vc_query ON vex_consensus(tenant, policy_version, consensus_state, confidence);
```
### 3.11 Integration with Findings Ledger and Vuln Explorer
* The Vuln Explorer reads `vex_consensus` for each finding and renders:
* Consensus chip, confidence, and a link to full quorum.
* For `NOT_AFFECTED` with confidence ≥ policy threshold, show “Suppressed by VEX (Consensus)” badge.
* For `DISPUTED`, open a triage banner prompting manual review and optional ledger comment/assignment.
* Ledger receives no new event type from lens computation itself. Human actions triggered by lens views produce standard events (`comment`, `assign`, `change_state`).
### 3.12 Security & RBAC
Roles:
* Viewer: query consensus read‑only.
* Investigator: Viewer + export.
* Operator: Investigator + trust simulation.
* Admin: manage Issuer Directory entries and keys.
CSRF for Console; ABAC scoping by artifact ownership and environment.
### 3.13 Observability
Metrics:
* `vex_consensus_compute_latency_ms` (histogram)
* `vex_consensus_records_total` (counter)
* `vex_consensus_disputed_total` (counter by issuer combinations)
* `vex_consensus_staleness_seconds` (gauge)
* `vex_signature_verification_rate` (gauge)
Logs: structured events with `tenant`, `policy_version`, `advisory_key`, `quorum_summary`.
Traces: spans for normalization, mapping, trust weighting, consensus decision, DB writes.
### 3.14 Performance & scaling
Targets:
* P95 query under 500 ms for 100‑row pages at 10M consensus records/tenant.
* Projection jobs are idempotent and keyed by `(tenant, artifact, purl, version, advisory, policy)`; backpressure with work queues.
* Cache popular queries with tenant‑scoped TTL; invalidate on Excitator or policy changes.
### 3.15 Edge cases
* **Ambiguous product mapping:** mark low `scope_score`, cap weight, surface warning in UI.
* **VEX “not present” vs SBOM shows present:** down‑weight with `f_justification`, require manual check.
* **Withdrawn or superseded VEX:** decay to near zero; keep provenance.
* **Partial fixes:** if fixed version applies to subset of platforms, map to env or arch dimension when available.
* **Time travel:** consensus recalculated as of a timestamp using only evidence ≤ `as_of` and the corresponding policy version.
---
## 4) Implementation plan
### 4.1 Services
* **VEX Lens Service (new)**
* Normalization pipeline, trust weighting, consensus computation, and projections.
* Batch recompute on policy activation and Excitator deltas.
* **Excitator (updates)**
* Ensure all VEX evidence carries issuer hints and raw signature blobs when present.
* Publish product trees and original coordinates intact.
* **Policy Engine (updates)**
* Add VEX trust knobs, thresholds, recency decay, and status precedence.
* Batch eval endpoint accepts `consensus inputs` where needed.
* **Issuer Directory (new)**
* Manage issuer metadata and keys; tenant overrides; audit logs.
### 4.2 Code structure
```
/src/VexLens/StellaOps.VexLens
/normalizer
/mapping # CPE/purl translators
/trust # weighting functions
/consensus # algorithm and projections
/api
/src/Excititor # updates
/src/Policy/__Libraries/StellaOps.Policy # updates
/src/IssuerDirectory/StellaOps.IssuerDirectory
/packages/console/features/vex-consensus
/src/Cli/StellaOps.Cli
```
### 4.3 Rollout
* Phase 1: API read‑only with basic trust model, Console list/detail, no simulation.
* Phase 2: Policy Studio integrations and simulation.
* Phase 3: Issuer Directory admin flows, exports, and advanced mapping diagnostics.
---
## 5) Documentation changes (create/update)
1. `/docs/vex/consensus-overview.md`
Purpose, scope, terminology, evidence vs derived view, AOC guarantees.
2. `/docs/vex/consensus-algorithm.md`
Normalization, mapping, weighting, thresholds, precedence, formulas, examples.
3. `/docs/vex/issuer-directory.md`
Managing issuers, keys, trust overrides, security model.
4. `/docs/vex/consensus-api.md`
Endpoints, request/response schemas, errors, pagination, rate limits.
5. `/docs/vex/consensus-console.md`
Screens, filters, conflict workflows, a11y, deep links.
6. `/docs/policy/vex-trust-model.md`
Policy knobs, thresholds, decay, simulation.
7. `/docs/sbom/vex-mapping.md`
Product tree mapping to purl/version, ecosystem comparators, edge cases.
8. `/docs/security/vex-signatures.md`
Signature verification flows, key management, auditing.
9. `/docs/runbooks/vex-ops.md`
Recompute storms, mapping failures, signature errors, lag, quotas.
All docs end with the imposed rule statement.
---
## 6) Engineering tasks
### Backend core
* [ ] Implement normalization for CSAF VEX, OpenVEX, CycloneDX VEX.
* [ ] Build product mapping library (CPE→purl, vendor tokens→purl families).
* [ ] Implement signature verification (Ed25519/PKIX/DSSE) using Issuer Directory keys.
* [ ] Implement trust weighting functions and configurable parameters.
* [ ] Implement consensus algorithm with unit tests and property tests.
* [ ] Materialize `vex_consensus` projection with indexes and idempotent workers.
* [ ] Batch recompute on policy activation and Excitator deltas.
### APIs & Integrations
* [ ] `/vex/consensus` query, detail, simulate, export.
* [ ] Policy Engine: consume consensus signals; add thresholds and precedence.
* [ ] Vuln Explorer: show consensus chip and triage banners; deep link to Lens.
### Issuer Directory
* [ ] CRUD for issuers and keys, audit logs, RBAC.
* [ ] Import common CSAF publishers; seed with sane defaults.
### Console
* [ ] Build list grid with filters and saved views.
* [ ] Quorum bar and Evidence table with signature icons and scope quality tags.
* [ ] Conflict view for `DISPUTED`.
* [ ] Simulation drawer integrated with Policy Studio.
### CLI
* [ ] `stella vex consensus list|show|simulate|export` with JSON/CSV.
* [ ] Stable schemas; tests for piping and scripting.
### Observability/Perf
* [ ] Metrics, logs, traces as specified; dashboards.
* [ ] Load tests at 10M consensus records/tenant; optimize indexes and caches.
### Docs
* [ ] Author and cross‑link all docs listed in §5.
* [ ] Add examples and screenshots to Console doc.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* Normalization supports CSAF VEX, OpenVEX, CycloneDX VEX with product tree mapping to purls across npm, Maven, PyPI, Go, RPM/DEB.
* Signature verification works and affects weights; unverifiable signatures do not crash flows.
* Consensus outputs are reproducible, explainable, and queryable at scale.
* Vuln Explorer displays consensus state and affects policy outcomes per thresholds.
* Simulation reflects policy trust changes without side effects and returns rationale deltas.
* CLI/API feature parity; evidence and quorum are exportable.
* P95 performance budgets met; dashboards reflect health.
---
## 8) Risks and mitigations
* **Mapping errors (CPE→purl):** use conservative scope scores, cap weights, surface warnings, manual override hooks in Policy Studio.
* **Malicious or mistaken issuer:** signature verification plus trust weighting and tenancy overrides.
* **Evidence storms:** debounce recompute; batch and shard workers; backpressure and prioritization.
* **User confusion with conflicting VEX:** clear conflict UI, rationale chains, suggested actions, and policy banners.
* **Stale statements:** recency decay and expiry windows in policy.
---
## 9) Test plan
* **Unit:** status mapping, comparators per ecosystem, trust weighting, threshold math.
* **Property tests:** invariants such as monotonicity with added supporting evidence and idempotent recompute.
* **Integration:** Excitator→Lens→Policy→Vuln Explorer pipeline with signed/unsigned, conflicting, and stale evidence.
* **E2E Console:** list filters, detail with quorum, conflict workflows, export, simulation.
* **Security:** RBAC on Issuer Directory, CSRF for Console, signature verification path traversal guards.
* **Performance:** cold/hot query latencies, recompute throughput, cache hit ratios.
* **Determinism:** time‑travel snapshots reproduce prior consensus states.
---
## 10) Philosophy
* **Consensus, not replacement.** The Lens summarizes without erasing dissent.
* **Trust is contextual.** Tenants must tune weights and thresholds to their environments.
* **Proof over prose.** Every decision comes with math, provenance, and a rationale chain.
* **Safety by design.** When in doubt, surface conflicts instead of silently suppressing.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

440
docs/implplan/EPIC_8.md
View File

@@ -1,440 +0,0 @@
Below is the “maximum documentation” bundle for Epic 8. It’s engineered to be pasted into your repo without turning into yet another unread wiki tomb. Slight sarcasm included to keep blood flowing.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
# Epic 8: Advisory AI Assistant (summaries, conflict explain, remediation hints)
**Short name:** `Advisory AI`
**Services touched:** Conseiller (Feedser), Excitator (Vexer), VEX Lens, SBOM Service, Policy Engine, Findings Ledger, Web API Gateway, Authority (authN/Z), Console (Web UI), CLI, Telemetry/Analytics
**AOC ground rule:** Conseiller and Excitator aggregate but never merge or mutate source docs. Advisory AI produces derived summaries and plans with strict provenance and citations. No silent rewriting of evidence. Ever.
---
## 1) What it is
Advisory AI is a tenant‑scoped, retrieval‑augmented assistant that turns noisy security advisories and VEX statements into three consumable artifacts:
1. **Advisory Summary**
Condenses one or more advisories (CSAF, OSV, GHSA, vendor PDFs, distro notices) into a concise brief with key facts: affected ranges, exploit status, impact, known workarounds, fixed versions, and links. Always cites the exact sources and sections used.
2. **Conflict Explain**
Explains why VEX statements or advisories disagree for a specific artifact and version. Uses the VEX Consensus Lens outputs and issuer trust model to produce a human‑readable, step‑by‑step explanation: who said what, where the product scoping diverges, and what policy thresholds caused the final state.
3. **Remediation Hints**
Suggests practical next steps: upgrade paths compatible with your dependency graph, backports, config toggles, temporary policy suppressions, or compensating controls. Every hint is grounded in SBOM, environment, and policy. It ships as structured JSON plus a human summary, ready to paste into a ticket.
It lives in the Console as a side panel, in the CLI for batch runs, and via APIs for automation. It does not change scanner results or consensus on its own. Humans remain in charge. The machine does the skimming and the math so humans can keep the judgment and the coffee.
---
## 2) Why (brief)
Advisories are long, inconsistent, and sometimes contradictory. Teams waste cycles reconciling PDFs with package manifests. The assistant eliminates that sludge: fast summaries, explicit conflict explanations, and remediation hints that are actually applicable to your software, not to an imaginary ideal project from 2013.
---
## 3) How it should work (maximum detail)
### 3.1 Capabilities
* **Summaries**
* Input: one advisory or a bundle linked by the same advisory key (CVE, GHSA, vendor‑ID), product scope, and environment.
* Output:
* 150 to 300 words summary
* `AdvisorySummary JSON` (schema below)
* Citations with paragraph anchors
* Confidence label and coverage score (how much of the advisory set is represented)
* **Conflict Explain**
* Input: `(artifact_id, purl, version, advisory_key)` tuple.
* Output: narrative plus a structured breakdown of consensus math, issuer votes, product mapping mismatches, and the exact policy knobs that tipped the result.
* **Remediation Hints**
* Input: same tuple plus SBOM context and environment.
* Output: ranked list of remediation options with feasibility score, blast radius estimate (derived from dependency paths), effort class, and links to fixed versions. Includes “do nothing” when the VEX consensus is not affected.
### 3.2 System design
**Architecture diagram in words (because ASCII art is a crime):**
1. **Retrievers**
* Structured retriever over Conseiller’s normalized advisory fields.
* Vector retriever over advisory text chunks with paragraph anchors.
* VEX retriever over Excitator evidence and VEX Lens consensus.
* SBOM retriever for purl, version, dependency paths, env flags.
2. **Deterministic resolvers**
* Version comparators per ecosystem.
* Range satisfaction checks.
* Dependency path scorers and blast radius estimator.
3. **Orchestrator**
* Task‑specific prompt templates for Summary, Conflict, Remediation.
* Tool calls to deterministics (version check, graph crawl) with results injected into the prompt as structured context.
* Strict token budgets and truncation rules to avoid model babble.
4. **Models**
* Default: on‑prem inference container with mid‑sized model.
* Optional: tenant‑enabled remote inference. Disabled by default.
* Temperature locked low for summary and conflict. Slightly higher for remediation narrative phrasing. No creativity in facts.
5. **Guardrails**
* Prompt injection defense by stripping or quarantining advisory text that tries to instruct the model.
* Fact boundary tagger. The assistant must only state facts that appear in structured inputs or cited chunks.
* Redaction of secrets before prompts.
* Output validator checks: required JSON fields, numeric ranges, valid version strings.
### 3.3 Data contracts
**AdvisorySummary JSON**
```json
{
"advisory_key": "CVE-2025-12345",
"sources": [
{"id":"csaf:vendorA:2025-001","uri":"...","sections":["2.1","3.4"]},
{"id":"osv:pkg:npm/lodash","uri":"...","sections":["affected","references"]}
],
"affected_ranges": [
{"ecosystem":"npm","purl_family":"pkg:npm/lodash","introduced":"<4.17.15","fixed": "4.17.21"}
],
"exploit_status": "no_known_exploit | poc_public | exploited_in_the_wild | n/a",
"impact": {"cvss":[{"vector":"CVSS:3.1/AV:N/...","score":7.5}], "cwes":["CWE-79"]},
"workarounds": ["Disable feature X", "Set flag Y=false"],
"fixed_versions": ["4.17.21"],
"notes": "Vendor states not affected on platform Z due to build option W.",
"coverage_score": 0.86,
"generated_at": "2025-10-25T12:00:00Z"
}
```
**ConflictExplanation JSON**
```json
{
"tuple": {"artifact_id":"svc:checkout@1.9.0","purl":"pkg:npm/lodash@4.17.20","advisory_key":"CVE-2025-12345"},
"consensus": {"state":"NOT_AFFECTED","confidence":0.82},
"quorum": [
{"issuer":"lodash-maintainers","status":"NOT_AFFECTED","weight":0.9,"sig":true,"justification":"component_not_present"},
{"issuer":"vendorX-distro","status":"AFFECTED","weight":0.25,"sig":false,"justification":"generic"}
],
"policy_factors": {"na_threshold":1.0,"aff_threshold":0.6,"recency_decay_days":90},
"mapping_issues": [{"kind":"cpe_to_purl","score":0.6,"detail":"CPE wildcard matched multiple purls"}],
"explanation_steps": [
"Exact purl match found for maintainer VEX; weight 0.9",
"Distro advisory generic; scope score 0.5; effective weight 0.25",
"NA threshold met. Result set to NOT_AFFECTED"
]
}
```
**RemediationPlan JSON**
```json
{
"tuple": {"artifact_id":"svc:checkout@1.9.0","purl":"pkg:npm/lodash@4.17.20","advisory_key":"CVE-2025-12345"},
"options": [
{
"kind": "upgrade",
"target_version": "4.17.21",
"feasibility": 0.92,
"blast_radius": {"direct_callers":3,"transitive_depth":2},
"effort": "low | medium | high",
"rationale": "Semver patch, no breaking APIs in release notes",
"links": ["release_notes_uri"]
},
{
"kind": "workaround",
"action": "Set SAFE_MODE=true",
"feasibility": 0.6,
"blast_radius": {"feature_flags":["SAFE_MODE"]},
"effort": "low",
"rationale": "Vendor states mitigation reduces attack surface on feature X"
}
],
"preferred": 0,
"policy_effects": {"sla_days": 7, "severity_override": "medium_if_not_fixed"},
"generated_at": "2025-10-25T12:00:00Z"
}
```
### 3.4 APIs
```
POST /advisory/ai/summary
{
"advisory_key":"CVE-2025-12345",
"artifact_id":"svc:checkout@1.9.0",
"purl":"pkg:npm/lodash@4.17.20",
"sources":["csaf:*","osv:*"], // optional filters
"policy_version":"1.3.0",
"lang":"en"
}
-> 200 { "summary_text":"...", "summary": {AdvisorySummary}, "citations":[...] }
POST /advisory/ai/conflict
{
"artifact_id":"svc:checkout@1.9.0",
"purl":"pkg:npm/lodash@4.17.20",
"advisory_key":"CVE-2025-12345",
"policy_version":"1.3.0"
}
-> 200 { "explanation_text":"...", "explanation": {ConflictExplanation} }
POST /advisory/ai/remediation
{
"artifact_id":"svc:checkout@1.9.0",
"purl":"pkg:npm/lodash@4.17.20",
"advisory_key":"CVE-2025-12345",
"policy_version":"1.3.0",
"max_options":5,
"strategy_preference":["upgrade","backport","workaround"]
}
-> 200 { "plan_text":"...", "plan": {RemediationPlan} }
POST /advisory/ai/batch
{
"items":[ {tuple}, {tuple}, ... ],
"task":"summary | conflict | remediation",
"policy_version":"1.3.0"
}
-> 207 multi-status
```
Status codes: `400` invalid, `403` RBAC, `404` missing evidence, `409` conflict lock, `422` output validation failed, `429` rate limit.
### 3.5 Console (Web UI)
* Surfaces:
* Vuln Explorer detail: “Advisory AI” side panel with 3 tabs: Summary, Conflict, Remediation.
* Consensus Lens detail: prominent “Explain conflict” button.
* Policy Studio sim: “Show effect on assistant output” preview.
* UX details:
* Citations are footnotes with hover to show source paragraph.
* “Copy as ticket” produces Markdown and JSON.
* Plan options show feasibility bar, blast radius chips, and required approvals per policy.
* Injection warnings appear if advisory text included unsafe instructions.
* A11y: ARIA tags for tabs, keyboard shortcuts `G` to generate, `R` to refresh, `C` to copy JSON.
### 3.6 CLI
```
stella advise summarize --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0 --json
stella advise explain --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0
stella advise remediate --advisory CVE-2025-12345 --artifact svc:checkout@1.9.0 --purl pkg:npm/lodash@4.17.20 --policy 1.3.0 --strategy upgrade,workaround --out plan.json
stella advise batch --file tuples.json --task remediation --policy 1.3.0
```
Exit codes: `0` ok, `2` invalid args, `4` not found, `5` denied, `7` validation fail.
### 3.7 RBAC and security
* Roles:
* Viewer can run summaries and read explanations.
* Operator can run remediation and export plans.
* Admin can toggle model endpoints and guardrail settings.
* Defaults:
* Remote model calls disabled.
* Redaction on.
* Prompt logging anonymized.
* Outputs stored as derived artifacts with TTL (default 30 days) unless pinned to a ticket.
### 3.8 Observability
* Metrics:
* `advisory_ai_latency_ms` by task type.
* `advisory_ai_guardrail_blocks_total`.
* `advisory_ai_output_validation_fail_total`.
* `advisory_ai_citation_coverage` gauge.
* Traces: retriever spans, tool calls, model inference, validator.
* Logs: include tuple key, token usage, truncation events, and guardrail outcomes.
### 3.9 Performance
* Targets:
* P95 under 1.5 s for Summary and Conflict with warm caches.
* P95 under 2.5 s for Remediation on medium SBOMs (1000 packages).
* Batch throughput 10 tuples per second per worker.
### 3.10 Edge cases
* Advisory missing fixed versions: produce workaround‑only plan and mark feasibility low.
* Conflicts with near‑tie weights: declare “DISPUTED” and require human approval, no auto plan preferred.
* Exotic version schemes: fallback to string compare with warning and feasibility cap.
* Private packages: no public release notes. Prefer internal changelog links if attached to artifact metadata.
* Multi‑env differences: render per‑env deltas when policy knobs differ (dev vs prod).
---
## 4) Implementation plan
### 4.1 Services and components
* **New:** `src/AdvisoryAI/StellaOps.AdvisoryAI`
* `retriever/` wrappers for Conseiller, Excitator, VEX Lens, SBOM.
* `deterministic/` version and path analyzers.
* `orchestrator/` task routers and prompt builders.
* `guardrails/` injection, redaction, output validator.
* `api/` REST endpoints and schema enforcement.
* **Updates:**
* Conseiller: expose paragraph‑level anchors for advisories.
* Excitator: expose justifications and product trees in normalized form.
* VEX Lens: stable API for quorum and rationale.
* SBOM Service: efficient path queries and versions timeline per purl.
### 4.2 Packaging
* Container images:
* `stella/advisory-ai:<<version>>`
* `stella/inference:<<version>>` (if using on‑prem model)
* Helm values to toggle remote inference and GPU.
### 4.3 Rollout
* Phase 1: Summary and Conflict read‑only.
* Phase 2: Remediation with “Copy as ticket”.
* Phase 3: Batch APIs, CLI, and Policy Studio simulation hooks.
---
## 5) Documentation changes
Create or update the following files. Each doc ends with the imposed rule statement.
1. `/docs/advisory-ai/overview.md`
What it is, capabilities, guardrails, AOC alignment, RBAC.
2. `/docs/advisory-ai/architecture.md`
RAG design, retrievers, orchestrator, deterministics, models, caching.
3. `/docs/advisory-ai/api.md`
Endpoint specs, payload schemas, error codes, examples.
4. `/docs/advisory-ai/console.md`
Screens, actions, a11y, how citations work, copy‑as‑ticket.
5. `/docs/advisory-ai/cli.md`
Command usage, exit codes, piping examples.
6. `/docs/policy/assistant-parameters.md`
Temperature, max tokens, plan ranking weights, TTLs.
7. `/docs/security/assistant-guardrails.md`
Redaction rules, injection defense, output validation, logging.
8. `/docs/sbom/remediation-heuristics.md`
Feasibility scoring, blast radius, effort classes.
9. `/docs/runbooks/assistant-ops.md`
Warmup, cache priming, model outages, scaling, on‑call steps.
---
## 6) Engineering tasks
### Backend core
* [ ] Implement structured and vector retrievers with paragraph anchors from Conseiller.
* [ ] Implement VEX retriever using Lens APIs with caching.
* [ ] Build deterministics: ecosystem comparators, range checks, dependency path scorer.
* [ ] Implement orchestrator with task‑specific templates and tool call pipeline.
* [ ] Implement guardrails and validators with hard failure on invalid JSON.
* [ ] Add RBAC to endpoints and anonymized prompt logging.
* [ ] Add caching layer with tuple‑keyed entries and policy version scoping.
### Integrations
* [ ] Conseiller: expose advisory chunk API and metadata needed for citations.
* [ ] Excitator: ensure justifications and product trees are queryable.
* [ ] VEX Lens: add “policy factors” endpoint for explanation rendering.
* [ ] SBOM Service: implement `GET /sbom/paths?purl=...` and version timeline.
### Console
* [ ] Build Advisory AI panel with 3 tabs and citation tooltips.
* [ ] Implement “Copy as ticket” (Markdown + JSON) and download.
* [ ] Add injection warning banner when triggered.
* [ ] Respect a11y requirements and shortcuts.
### CLI
* [ ] `stella advise summarize|explain|remediate|batch` with JSON output.
* [ ] Add `--out` option to save plans and summaries.
* [ ] Tests for piping and jq workflows.
### Observability
* [ ] Emit metrics and traces listed in §3.8.
* [ ] Dashboards: latency, guardrail blocks, validation fails, coverage.
### Docs
* [ ] Write all files in §5 with examples and screenshots.
* [ ] Cross‑link to VEX Lens and Vulnerability Explorer docs.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* Summaries cite specific source sections and reflect affected ranges and fixed versions correctly for at least 95% of a validation set.
* Conflict explanations enumerate issuers, weights, justifications, mapping issues, and policy thresholds that caused the consensus state.
* Remediation plans output at least one feasible option when a fixed version exists and correctly flag “no public fix” cases.
* JSON schemas validate for all outputs.
* Console shows the panel with citations, copy‑as‑ticket, and a11y passes.
* CLI produces identical JSON to API responses.
* Guardrails block injection attempts and redact secrets in prompts.
* P95 latency targets are met with warm caches.
* No mutation of raw advisory or VEX evidence occurs anywhere in the pipeline.
---
## 8) Risks and mitigations
* **Prompt injection in advisory text.** Strip instructions, sandbox chunks, and highlight to user when removed.
* **Hallucinated facts.** Hard validation requires facts to appear in structured inputs or cited text. Fail closed if not provable.
* **Mapping errors produce bad hints.** Depend on SBOM Graph and VEX Lens scope scores; cap feasibility when scope is weak.
* **Model outage.** Degrade to deterministic summaries (shorter, but accurate).
* **Privacy concerns.** Default on‑prem inference, remote endpoints opt‑in with clear flags and audit logs.
---
## 9) Test plan
* **Unit:** version comparators, range checks, feasibility scoring, output validators.
* **Golden files:** advisory sets mapped to expected summaries and plans; diff on each build.
* **Injection tests:** adversarial advisories with “ignore prior instructions” payloads must be neutralized.
* **Integration:** Conseiller→Advisory AI→Console loop, with VEX Lens conflicts and SBOM graph lookups.
* **E2E:** generate summary, explanation, and plan for representative ecosystems (npm, Maven, PyPI, Go, RPM/DEB).
* **Perf:** soak tests with 5k tuples batch; observe cache hit ratios and P95.
* **A11y:** keyboard navigation and screen reader labels.
---
## 10) Philosophy
* **Facts first.** If it is not in structured inputs or citations, it does not exist.
* **Explain everything.** Humans should see exactly why the tool said what it said.
* **Helpful by default.** Plans must consider the real dependency graph and environment, not fantasy.
* **No silent merges.** Evidence is sacred. Summaries and plans are separate, auditable derivatives.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

523
docs/implplan/EPIC_9.md
View File

@@ -1,523 +0,0 @@
Below is the “maximum documentation” bundle for Epic 9. Paste it into your repo and pretend the ingestion chaos was always under control.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
# Epic 9: Source & Job Orchestrator Dashboard
**Short name:** `Orchestrator Dashboard`
**Primary service:** `orchestrator` (scheduler, queues, rate‑limits, job state)
**Surfaces:** Console (Web UI), CLI, Web API
**Touches:** Conseiller (Feedser), Excitator (Vexer), VEX Consensus Lens, SBOM Service, Policy Engine, Findings Ledger, Authority (authN/Z), Telemetry/Analytics, Object Storage, Relational DB, Message Bus
**AOC ground rule:** Conseiller and Excitator aggregate but never merge. The orchestrator schedules, tracks and recovers jobs; it does not transform evidence beyond transport and storage. No “smart” merging in flight.
---
## 1) What it is
The Source & Job Orchestrator Dashboard is the control surface for every data source and pipeline run across StellaOps. It gives operators:
* Live health of all advisory/VEX/SBOM sources and derived jobs.
* End‑to‑end pipeline visibility as DAGs and timelines.
* Controls for pausing, backfilling, replaying, throttling and retrying.
* Error pattern analysis, rate‑limit observability and backpressure insights.
* Provenance and audit trails from initial fetch through parse, normalize, index and policy evaluation.
The dashboard sits over the `orchestrator` service, which maintains job state, schedules runs, enforces quotas and rate limits, and collects metrics from worker pools embedded in Conseiller, Excitator, SBOM and related services.
---
## 2) Why (brief)
Ingestion breaks quietly and then loudly. Without a unified control plane, you learn about it from angry users or empty indexes. This dashboard shortens incident MTTR, enables safe backfills, and makes compliance reviewers stop sending emails with twelve attachments and one emoji.
---
## 3) How it should work (maximum detail)
### 3.1 Capabilities
* **Source registry**
* Register, tag and version connectors (OSV, GHSA, CSAF endpoints, vendor PDF scrapers, distro feeds, RSS, S3 drops, internal registries).
* Store connection details, secrets (via KMS), rate‑limit policy, schedules, and ownership metadata.
* Validate and “test connection” safely.
* **Job orchestration**
* Create DAGs composed of job types: `fetch`, `parse`, `normalize`, `dedupe`, `index`, `consensus_compute`, `policy_eval`, `crosslink`, `sbom_ingest`, `sbom_index`.
* Priorities, queues, concurrency caps, exponential backoff, circuit breakers.
* Idempotency keys and output artifact hashing to avoid duplicate work.
* Event‑time watermarks for backfills without double counting.
* **Observability & control**
* Gantt timeline and real‑time DAG view with critical path highlighting.
* Backpressure and queue depth heatmaps.
* Error clustering by class (HTTP 429, TLS, schema mismatch, parse failure, upstream 5xx).
* Per‑source SLOs and SLA budgets with burn‑rate alerts.
* One‑click actions: retry, replay range, pause/resume, throttle/unthrottle, reroute to canary workers.
* **Provenance & audit**
* Immutable run ledger linking input artifact → every job → output artifact.
* Schema version tracking and drift detection.
* Operator actions recorded with reason and ticket reference.
* **Safety**
* Secret redaction everywhere.
* Tenant isolation at API, queue and storage layers.
* AOC: no in‑flight merges of advisory or VEX content.
### 3.2 Core architecture
* **orchestrator (service)**
* Maintains job state in Postgres (`sources`, `runs`, `jobs`, `artifacts`, `dag_edges`, `quotas`, `schedules`).
* Publishes work to a message bus (e.g., `topic.jobs.ready.<queue>`).
* Distributed token‑bucket rate limiter per source/tenant/host.
* Watchdog for stuck jobs and circuit breakers for flapping sources.
* Watermark manager for backfills (event‑time windows).
* **worker SDK**
* Lightweight library embedded in Conseiller/Excitator/SBOM workers to:
* Claim work, heartbeat, update progress, report metrics.
* Emit artifact metadata and checksums.
* Enforce idempotency via orchestrator‑supplied key.
* **object store**
* Raw payloads and intermediate artifacts organized by schema and hash:
* `advisory/raw/<source_id>/<event_time>/<sha256>.json|pdf`
* `advisory/normalized/<schema_ver>/<hash>.json`
* `vex/raw|normalized/...`
* `sbom/raw|graph/...`
* **web API**
* CRUD for sources, runs, jobs, schedules, quotas.
* Control actions (retry, cancel, pause, backfill).
* Streaming updates via WebSocket/SSE for the Console.
* **console**
* React app consuming Orchestrator APIs, rendering DAGs, timelines, health charts and action panels with RBAC.
### 3.3 Data model (selected tables)
* `sources`
* `id`, `kind` (`advisory|vex|sbom|internal`), `subtype` (e.g., `osv`, `ghsa`, `csaf`, `vendor_pdf`), `display_name`, `owner_team`, `schedule_cron`, `rate_policy`, `enabled`, `secrets_ref`, `tags`, `schema_hint`, `created_at`, `updated_at`.
* `runs`
* `id`, `source_id`, `trigger` (`schedule|manual|event|backfill`), `window_start`, `window_end`, `state`, `started_at`, `finished_at`, `stats_json`.
* `jobs`
* `id`, `run_id`, `type`, `queue`, `priority`, `state` (`pending|running|succeeded|failed|canceled|deadletter`), `attempt`, `max_attempt`, `idempotency_key`, `input_artifact_id`, `output_artifact_id`, `worker_id`, `created_at`, `started_at`, `finished_at`, `error_class`, `error_message`, `metrics_json`.
* `dag_edges`
* `from_job_id`, `to_job_id`, `edge_kind` (`success_only|always`).
* `artifacts`
* `id`, `kind` (`raw|normalized|index|consensus`), `schema_ver`, `hash`, `uri`, `bytes`, `meta_json`, `created_at`.
* `quotas`
* `tenant_id`, `resource` (`requests_per_min`, `concurrent_jobs`), `limit`, `window_sec`.
* `schedules`
* Per‑source cron plus jitter, timezone, blackout windows.
### 3.4 Job lifecycle
1. **Plan**
Scheduler creates a `run` for a source and plans a DAG: e.g., `fetch → parse → normalize → dedupe → index → policy_eval` (advisory) or `fetch → parse → normalize → consensus_compute` (VEX).
2. **Enqueue**
Ready nodes become `jobs` with queue, priority, idempotency key and optional rate‑limit tokens reserved.
3. **Execute**
Worker claims job, heartbeats every N seconds. Output artifacts are stored and linked. Failures are classified and retried with exponential backoff and jitter, up to `max_attempt`.
4. **Complete**
Downstream nodes unblock. On run completion, orchestrator computes SLO deltas and emits run summary.
5. **Dead‑letter**
Jobs exceeding attempts move to a DLQ with structured context and suggested remediation.
### 3.5 Scheduling, backpressure, rate‑limits
* **Token bucket** per `{tenant, source.host}` with adaptive refill if upstream 429/503 seen.
* **Concurrency caps** per source and per job type to avoid thundering herd.
* **Backpressure signals** from queue depth, worker CPU, and upstream error rates; scheduler reduces inflight issuance accordingly.
* **Backfills** use event‑time windows with immutable watermarks to avoid re‑processing.
* **Blackout windows** for vendor maintenance periods.
### 3.6 APIs
```
POST /orchestrator/sources
GET /orchestrator/sources?kind=&tag=&q=
GET /orchestrator/sources/{id}
PATCH /orchestrator/sources/{id}
POST /orchestrator/sources/{id}/actions:test|pause|resume|sync-now
POST /orchestrator/sources/{id}/backfill { "from":"2024-01-01", "to":"2024-03-01" }
GET /orchestrator/runs?source_id=&state=&from=&to=
GET /orchestrator/runs/{run_id}
GET /orchestrator/runs/{run_id}/dag
POST /orchestrator/runs/{run_id}/cancel
GET /orchestrator/jobs?state=&type=&queue=&source_id=
GET /orchestrator/jobs/{job_id}
POST /orchestrator/jobs/{job_id}/actions:retry|cancel|prioritize
GET /orchestrator/metrics/overview
GET /orchestrator/errors/top?window=1h
GET /orchestrator/quotas
PATCH /orchestrator/quotas/{tenant_id}
WS /orchestrator/streams/updates
```
### 3.7 Console (Web UI)
* **Overview**
* KPI tiles: sources healthy, runs in progress, queue depth, error rate, burn‑rate to SLO.
* Heatmap of source health by last 24h success ratio.
* **Sources**
* Grid with filters, inline status (active, paused, throttled), next run eta, last error class.
* Detail panel: config, secrets status (redacted), schedule, rate limits, ownership, run history, action buttons.
* **Runs**
* Timeline (Gantt) with critical path, duration distribution, and per‑stage breakdown.
* Run detail: DAG view with node metrics, artifacts, logs, action menu (cancel).
* **Jobs**
* Live table with state filters and “tail” view.
* Job detail: payload preview (redacted), worker, attempts, stack traces, linked artifacts.
* **Errors**
* Clusters by class and signature, suggested remediations (pause source, lower concurrency, patch parser).
* **Queues & Backpressure**
* Per‑queue depth, service rate, inflight, age percentiles.
* Rate‑limit tokens graphs per source host.
* **Controls**
* Backfill wizard with event‑time preview and safety checks.
* Canary routing: route 5% of next 100 runs to a new worker pool.
* **A11y**
* Keyboard nav, ARIA roles for DAG nodes, live regions for updates, color‑blind friendly graphs.
### 3.8 CLI
```
stella orch sources list --kind advisory --tag prod
stella orch sources add --file source.yaml
stella orch sources test <source-id>
stella orch sources pause <source-id> # or resume
stella orch sources sync-now <source-id>
stella orch sources backfill <source-id> --from 2024-01-01 --to 2024-03-01
stella orch runs list --source <id> --state running
stella orch runs show <run-id> --dag
stella orch runs cancel <run-id>
stella orch jobs list --state failed --type parse --limit 100
stella orch jobs retry <job-id>
stella orch jobs cancel <job-id>
stella orch jobs tail --queue normalize --follow
stella orch quotas get --tenant default
stella orch quotas set --tenant default --concurrent-jobs 50 --rpm 1200
```
Exit codes: `0` success, `2` invalid args, `4` not found, `5` denied, `7` precondition failed, `8` rate‑limited.
### 3.9 RBAC & security
* **Roles**
* `Orch.Viewer`: read‑only sources/runs/jobs/metrics.
* `Orch.Operator`: perform actions on sources and jobs, launch backfills.
* `Orch.Admin`: manage quotas, schedules, connector versions, and delete sources.
* **Secrets**
* Stored only as references to your KMS; never persisted in cleartext.
* Console shows redact badges and last rotated timestamp.
* **Tenancy**
* Source, run, job rows scoped by tenant id.
* Queue names and token buckets namespaced per tenant.
* **Compliance**
* Full audit log for every operator action with “reason” and optional ticket link.
* Exportable run ledger for audits.
### 3.10 Observability
* **Metrics (examples)**
* `orch_jobs_inflight{type,queue}`
* `orch_jobs_latency_ms{type,percentile}`
* `orch_rate_tokens_available{source}`
* `orch_error_rate{source,error_class}`
* `orch_slo_burn_rate{source,slo}`
* `orch_deadletter_total{source,type}`
* **Traces**
* Span per job with baggage: `run_id`, `source_id`, `artifact_id`.
* Links across services to Conseiller/Excitator/SBOM workers.
* **Logs**
* Structured JSON with correlation ids, attempt numbers and redacted payload previews.
### 3.11 Performance targets
* Job dispatch P95 < 150 ms after dependency satisfied.
* Scheduler loop P95 < 500 ms for 10k pending jobs.
* Console live updates sub‑second at 1k events/sec per tenant.
* Backfill throughput ≥ 200 jobs/sec per worker pool with zero dupes.
### 3.12 Edge cases & behaviors
* **Upstream 429 storms:** auto‑throttle, pause optional, recommend extended jitter.
* **Schema drift:** parser moves job to DLQ with `error_class=schema_mismatch` and opens a change ticket via webhook.
* **Flapping source:** circuit breaker opens after N consecutive failures; requires human “resume”.
* **Clock skew:** watermark logic uses upstream event time; large skews flagged.
* **Idempotency collisions:** new attempt yields no‑op if artifact hash already exists.
---
## 4) Implementation plan
### 4.1 Modules (new and updated)
* New service: `src/Orchestrator/StellaOps.Orchestrator`
* `api/` REST + WS handlers
* `scheduler/` run planner, DAG builder, watermark/backfill logic
* `queues/` publisher and consumer abstractions
* `ratelimit/` token bucket and adaptive controller
* `state/` Postgres repositories and migrations
* `audit/` action logging and export
* `metrics/` Prometheus exporters
* `security/` tenant scoping, KMS client, secret refs
* Worker SDKs:
* `src/Orchestrator/StellaOps.Orchestrator.WorkerSdk.Go` and `src/Orchestrator/StellaOps.Orchestrator.WorkerSdk.Python` with job claim, heartbeat, progress, artifact publish, and structured error reporting.
* Console:
* `console/apps/orch/` pages: Overview, Sources, Runs, Jobs, Errors, Queues.
* `components/dag-view/`, `components/gantt/`, `components/health-heatmap/`.
* Updates to existing services:
* Conseiller/Excitator/SBOM workers adopt SDK and emit artifacts with schema/version/fingerprint.
* VEX Lens exposes `consensus_compute` as a jobable operation.
* Policy Engine exposes `policy_eval` as a job type for scheduled recalcs.
### 4.2 Packaging & deployment
* Containers:
* `stella/orchestrator:<ver>`
* `stella/worker-sdk-examples:<ver>` for canary pools
* Helm values:
* Queues/topics, per‑tenant concurrency, rate‑limit defaults, WS replica count.
* KMS integration secrets.
* Migrations:
* Flyway/Goose migrations for new tables and indexes.
### 4.3 Rollout strategy
* Phase 1: Read‑only dashboard fed by existing job tables; no controls.
* Phase 2: Control actions enabled for non‑prod tenants.
* Phase 3: Backfills and quota management, then GA.
---
## 5) Documentation changes
Create/update the following, each ending with the imposed rule statement.
1. `/docs/orchestrator/overview.md`
Concepts, roles, responsibilities, AOC alignment.
2. `/docs/orchestrator/architecture.md`
Scheduler, DAGs, watermarks, queues, rate‑limits, data model.
3. `/docs/orchestrator/api.md`
Endpoints, WebSocket events, error codes, examples.
4. `/docs/orchestrator/console.md`
Screens, actions, a11y, live updates.
5. `/docs/orchestrator/cli.md`
Commands, examples, exit codes, scripting patterns.
6. `/docs/orchestrator/run‑ledger.md`
Provenance and audit export format.
7. `/docs/security/secrets‑handling.md`
KMS references, redaction rules, operator hygiene.
8. `/docs/operations/orchestrator‑runbook.md`
Common failures, backfill guide, circuit breakers, tuning.
9. `/docs/schemas/artifacts.md`
Artifact kinds, schema versions, hashing, storage layout.
10. `/docs/slo/orchestrator‑slo.md`
SLO definitions, measurement, alerting.
---
## 6) Engineering tasks
### Backend (orchestrator)
* [ ] Stand up Postgres schemas and indices for sources, runs, jobs, dag_edges, artifacts, quotas, schedules.
* [ ] Implement scheduler: DAG planner, dependency resolver, critical path computation.
* [ ] Implement rate limiter with adaptive behavior on 429/503 and per‑tenant tokens.
* [ ] Implement watermark/backfill manager with event‑time windows and idempotency keys.
* [ ] Implement API endpoints + OpenAPI spec + request validation.
* [ ] Implement WebSocket/SSE event stream for live updates.
* [ ] Implement audit logging and export.
* [ ] Implement dead‑letter store and replay.
### Worker SDKs and integrations
* [ ] Build Go/Python SDKs with claim/heartbeat/progress API.
* [ ] Integrate SDK into Conseiller, Excitator, SBOM workers; ensure artifact emission with schema ver.
* [ ] Add `consensus_compute` and `policy_eval` as job types with deterministic inputs/outputs.
### Console
* [ ] Overview tiles and health heatmap.
* [ ] Source list/detail with actions and config view.
* [ ] Runs timeline (Gantt) and DAG visualization with node inspector.
* [ ] Jobs “tail” with live updates and filters.
* [ ] Errors clustering and suggested remediations.
* [ ] Queues/backpressure dashboard.
* [ ] Backfill wizard with safety checks.
### Observability
* [ ] Emit metrics listed in §3.10 and wire traces across services.
* [ ] Dashboards: health, queue depth, error classes, burn‑rate, dispatch latency.
* [ ] Alerts for SLO burn and circuit breaker opens.
### Security & RBAC
* [ ] Enforce tenant scoping on all endpoints; test leakage.
* [ ] Wire KMS for secret refs and redact everywhere.
* [ ] Implement `Orch.Viewer|Operator|Admin` roles and check in Console and API.
### Docs
* [ ] Author all files in §5 with examples and screenshots.
* [ ] Cross‑link from Conseiller/Excitator/SBOM pages to the dashboard docs.
* [ ] Append imposed rule to each page.
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
---
## 7) Acceptance criteria
* Operators can: pause/resume a source, run “sync‑now,” initiate a backfill for a date range, and retry/cancel individual jobs from Console and CLI.
* DAG and timeline reflect reality within 1 second of job state changes at P95.
* Backfills do not create duplicate artifacts; idempotency proven by hash equality.
* Rate limiter reduces 429s by ≥80% under simulated throttle tests.
* Audit log includes who/when/why for every operator action.
* Provenance ledger exports a complete chain for any artifact.
* RBAC prevents non‑admins from quota changes; tenancy isolation proven via automated tests.
* SLO dashboard shows burn‑rate and triggers alerts under injected failure.
---
## 8) Risks & mitigations
* **Orchestrator becomes a single bottleneck.**
Horizontal scale stateless workers; DB indexes tuned; job state updates batched; cache hot paths.
* **Secret spillage.**
Only KMS references stored; aggressive redaction; log scrubbing in SDK.
* **Over‑eager backfills overwhelm upstream.**
Enforce per‑source quotas and sandbox previews; dry‑run backfills first.
* **Schema drift silently corrupts normalization.**
Hard‑fail on mismatch; DLQ with clear signatures; schema registry gating.
* **Flapping sources cause alert fatigue.**
Circuit breaker with cool‑down and deduped alerts; error budget policy.
---
## 9) Test plan
* **Unit**
Scheduler DAG building, topological sort, backoff math, token bucket, watermark math.
* **Integration**
Orchestrator ↔ worker SDK, artifact store wiring, DLQ replay, audit pipeline.
* **Chaos**
Inject 429 storms, packet loss, worker crashes; verify throttling and recovery.
* **Backfill**
Simulate overlapping windows and verify idempotency and watermark correctness.
* **Perf**
10k concurrent jobs: dispatch latency, DB contention, WebSocket fan‑out.
* **Security**
Multi‑tenant isolation tests; KMS mock tests for secret access; RBAC matrix.
* **UX/A11y**
Screen reader labels on DAG, keyboard navigation, live region updates.
---
## 10) Philosophy
* **Make the invisible visible.** Pipelines should be legible at a glance.
* **Prefer reproducibility to heroics.** Idempotency and provenance over “we think it ran.”
* **Safeguards before speed.** Throttle first, retry thoughtfully, never melt upstreams.
* **No silent merges.** Evidence remains immutable; transformations are explicit, logged and reversible.
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**

2
docs/implplan/EXECPLAN.md
View File

@@ -1244,7 +1244,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster
5. [TODO] DOCS-POLICY-20-005 — `/docs/api/policy.md` endpoints + schemas.
• Prereqs: WEB-POLICY-20-001 (Wave 7)
• Current: TODO
6. [TODO] DOCS-POLICY-20-006 — `/docs/cli/policy.md` with command usage.
6. [TODO] DOCS-POLICY-20-006 — `/docs/modules/cli/guides/policy.md` with command usage.
• Prereqs: CLI-POLICY-20-002 (Wave 7)
• Current: TODO
7. [TODO] DOCS-POLICY-20-007 — `/docs/ui/policy-editor.md` flows + screenshots.

24
docs/implplan/SPRINTS.md
View File

@@ -535,9 +535,9 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 35 | EPDR Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md | TODO | Scanner EPDR Guild | SCANNER-ANALYZERS-LANG-11-001 | Build entrypoint resolver (identity + environment profiles) and emit normalized entrypoint records. |
| Sprint 35 | EPDR Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md | TODO | Scanner EPDR Guild | SCANNER-ANALYZERS-LANG-11-002 | Static IL/reflection/ALC heuristics producing dependency edges with reason codes and confidence. |
| Sprint 35 | EPDR Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md | TODO | Scanner EPDR Guild, Signals Guild | SCANNER-ANALYZERS-LANG-11-003 | Runtime loader/PInvoke signal ingestion merged with static/declared edges (confidence & explain). |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-001 | Author `/docs/export-center/overview.md` with purpose, profiles, security, and imposed rule reminder. |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-002 | Author `/docs/export-center/architecture.md` detailing service components, adapters, manifests, signing, and distribution. |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-003 | Publish `/docs/export-center/profiles.md` covering schemas, examples, and compatibility. |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-001 | Author `/docs/modules/export-center/overview.md` with purpose, profiles, security, and imposed rule reminder. |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-002 | Author `/docs/modules/export-center/architecture.md` detailing service components, adapters, manifests, signing, and distribution. |
| Sprint 35 | Export Center Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-35-003 | Publish `/docs/modules/export-center/profiles.md` covering schemas, examples, and compatibility. |
| Sprint 35 | Export Center Phase 1 | ops/deployment/TASKS.md | TODO | Deployment Guild | DEPLOY-EXPORT-35-001 | Package exporter service/worker containers, Helm overlays (download-only), and rollout guide. |
| Sprint 35 | Export Center Phase 1 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-EXPORT-35-001 | Create exporter CI pipeline (lint/test/perf smoke), object storage fixtures, and initial Grafana dashboards. |
| Sprint 35 | Export Center Phase 1 | src/ExportCenter/StellaOps.ExportCenter/TASKS.md | TODO | Exporter Service Guild | EXPORT-SVC-35-001 | Bootstrap exporter service, configuration, and migrations for export profiles/runs/inputs/distributions with tenant scopes. |
@@ -553,9 +553,9 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 35 | Export Center Phase 1 | src/Web/StellaOps.Web/TASKS.md | TODO | BE-Base Platform Guild | WEB-EXPORT-35-001 | Route Export Center APIs through gateway with tenant scoping, viewer/operator scopes, and streaming downloads. |
| Sprint 36 | EPDR Observations | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md | TODO | Scanner EPDR Guild, SBOM Service Guild | SCANNER-ANALYZERS-LANG-11-004 | Normalize EPDR output to Scanner observation writer (entrypoints + edges + env profiles). |
| Sprint 36 | EPDR Observations | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md | TODO | Scanner EPDR Guild, QA Guild | SCANNER-ANALYZERS-LANG-11-005 | End-to-end fixtures/benchmarks covering publish modes, RIDs, trimming, NativeAOT with explain traces. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-004 | Author `/docs/export-center/api.md` with endpoint examples and imposed rule note. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-005 | Publish `/docs/export-center/cli.md` covering commands, scripts, verification, and imposed rule reminder. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-006 | Write `/docs/export-center/trivy-adapter.md` detailing mappings, compatibility, and test matrix. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-004 | Author `/docs/modules/export-center/api.md` with endpoint examples and imposed rule note. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-005 | Publish `/docs/modules/export-center/cli.md` covering commands, scripts, verification, and imposed rule reminder. |
| Sprint 36 | Export Center Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-36-006 | Write `/docs/modules/export-center/trivy-adapter.md` detailing mappings, compatibility, and test matrix. |
| Sprint 36 | Export Center Phase 2 | ops/deployment/TASKS.md | TODO | Deployment Guild | DEPLOY-EXPORT-36-001 | Document registry credentials, OCI push workflows, and automation for export distributions. |
| Sprint 36 | Export Center Phase 2 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-EXPORT-36-001 | Integrate Trivy compatibility validation, OCI push smoke tests, and metrics dashboards for export throughput. |
| Sprint 36 | Export Center Phase 2 | src/Cli/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI Guild | CLI-EXPORT-36-001 | Add `stella export distribute` (OCI/objstore), `run download --resume`, and status polling enhancements. |
@@ -565,8 +565,8 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 36 | Export Center Phase 2 | src/ExportCenter/StellaOps.ExportCenter/TASKS.md | TODO | Exporter Service Guild | EXPORT-SVC-36-004 | Extend planner/run lifecycle for OCI/object storage distributions with retry + idempotency. |
| Sprint 36 | Export Center Phase 2 | src/Orchestrator/StellaOps.Orchestrator/TASKS.md | TODO | Orchestrator Service Guild | ORCH-SVC-36-101 | Add distribution job follow-ups, retention metadata, and metrics for export runs. |
| Sprint 36 | Export Center Phase 2 | src/Web/StellaOps.Web/TASKS.md | TODO | BE-Base Platform Guild | WEB-EXPORT-36-001 | Expose distribution endpoints (OCI/object storage) and manifest/provenance download proxies with RBAC. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-001 | Publish `/docs/export-center/mirror-bundles.md` detailing layouts, deltas, encryption, imposed rule reminder. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-002 | Publish `/docs/export-center/provenance-and-signing.md` covering manifests, attestation, verification. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-001 | Publish `/docs/modules/export-center/mirror-bundles.md` detailing layouts, deltas, encryption, imposed rule reminder. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-002 | Publish `/docs/modules/export-center/provenance-and-signing.md` covering manifests, attestation, verification. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-003 | Publish `/docs/operations/export-runbook.md` for failures, tuning, capacity, with imposed rule note. |
| Sprint 37 | Export Center Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-EXPORT-37-004 | Publish `/docs/security/export-hardening.md` covering RBAC, isolation, encryption, and imposed rule. |
| Sprint 37 | Export Center Phase 3 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-EXPORT-37-001 | Finalize dashboards/alerts for exports (failure, verify), retention jobs, and chaos testing harness. |
@@ -631,7 +631,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 40 | Notifications Studio Phase 3 | src/Notifier/StellaOps.Notifier/TASKS.md | TODO | Notifications Service Guild | NOTIFY-SVC-40-003 | Harden security: signed ack links, webhook HMAC/IP allowlists, tenant isolation fuzzing, localization fallback. |
| Sprint 40 | Notifications Studio Phase 3 | src/Notifier/StellaOps.Notifier/TASKS.md | TODO | Notifications Service Guild | NOTIFY-SVC-40-004 | Finalize observability (incident metrics, escalation latency) and chaos tests for channel outages. |
| Sprint 40 | Notifications Studio Phase 3 | src/Web/StellaOps.Web/TASKS.md | TODO | BE-Base Platform Guild | WEB-NOTIFY-40-001 | Expose escalation, localization, channel health endpoints and verification of signed links. |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-CLI-41-001 | Publish `/docs/cli/overview.md`, `/cli/configuration.md`, `/cli/output-and-exit-codes.md` (with imposed rule). |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | docs/TASKS.md | TODO | Docs Guild | DOCS-CLI-41-001 | Publish `/docs/modules/cli/guides/overview.md`, `/cli/configuration.md`, `/cli/output-and-exit-codes.md` (with imposed rule). |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | ops/deployment/TASKS.md | TODO | Deployment Guild | DEPLOY-CLI-41-001 | Package CLI release artifacts (tarballs, completions, container image) with distribution docs. |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-CLI-41-001 | Establish CLI build pipeline (multi-platform binaries, SBOM, checksums) and parity matrix CI enforcement. |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | src/Authority/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-PACKS-41-001 | Define CLI SSO scopes and Packs (`Packs.Read/Write/Run/Approve`) roles; update discovery/offline defaults. |
@@ -641,7 +641,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 41 | CLI Parity & Task Packs Phase 1 | src/Orchestrator/StellaOps.Orchestrator/TASKS.md | TODO | Orchestrator Service Guild | ORCH-SVC-41-101 | Register `pack-run` job type, integrate logs/artifacts, expose pack run metadata. |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | src/PacksRegistry/StellaOps.PacksRegistry/TASKS.md | TODO | Packs Registry Guild | PACKS-REG-41-001 | Implement packs index API, signature verification, provenance storage, and RBAC. |
| Sprint 41 | CLI Parity & Task Packs Phase 1 | src/TaskRunner/StellaOps.TaskRunner/TASKS.md | TODO | Task Runner Guild | TASKRUN-41-001 | Bootstrap Task Runner service, migrations, run API, local executor, approvals pause, artifact capture. |
| Sprint 42 | CLI Parity & Task Packs Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-CLI-42-001 | Publish `/docs/cli/parity-matrix.md`, `/cli/commands/*.md`, `/docs/task-packs/spec.md` (imposed rule). |
| Sprint 42 | CLI Parity & Task Packs Phase 2 | docs/TASKS.md | TODO | Docs Guild | DOCS-CLI-42-001 | Publish `/docs/modules/cli/guides/parity-matrix.md`, `/cli/commands/*.md`, `/docs/task-packs/spec.md` (imposed rule). |
| Sprint 42 | CLI Parity & Task Packs Phase 2 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-CLI-42-001 | Add CLI golden output tests, parity diff automation, and pack run CI harness. |
| Sprint 42 | CLI Parity & Task Packs Phase 2 | src/Cli/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI Guild | CLI-PACKS-42-001 | Implement Task Pack CLI commands (`pack plan/run/push/pull/verify`) with plan/simulate engine and expression sandbox. |
| Sprint 42 | CLI Parity & Task Packs Phase 2 | src/Cli/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI Guild | CLI-PARITY-41-001..002 | Close parity gaps for Notifications, Policy Studio advanced features, SBOM graph, Vuln Explorer; parity matrix green. |
@@ -695,7 +695,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 48 | Authority-Backed Scopes & Tenancy Phase 2 | src/Policy/StellaOps.Policy.Engine/TASKS.md | TODO | Policy Guild | POLICY-TEN-48-001 | Add `tenant_id`/`project_id` to policy data, enable Postgres RLS, and expose rationale IDs with tenant context. |
| Sprint 48 | Authority-Backed Scopes & Tenancy Phase 2 | src/TaskRunner/StellaOps.TaskRunner/TASKS.md | TODO | Task Runner Guild | TASKRUN-TEN-48-001 | Propagate tenant/project to all steps, enforce object store prefix, and validate before execution. |
| Sprint 48 | Authority-Backed Scopes & Tenancy Phase 2 | src/Web/StellaOps.Web/TASKS.md | TODO | BE-Base Platform Guild | WEB-TEN-48-001 | Enforce tenant context through persistence (DB GUC, object store prefix), add request annotations, and emit audit events. |
| Sprint 49 | Authority-Backed Scopes & Tenancy Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-TEN-49-001 | Publish `/docs/cli/authentication.md`, `/docs/api/authentication.md`, `/docs/policy/examples/abac-overlays.md`, `/docs/install/configuration-reference.md` updates (imposed rule). |
| Sprint 49 | Authority-Backed Scopes & Tenancy Phase 3 | docs/TASKS.md | TODO | Docs Guild | DOCS-TEN-49-001 | Publish `/docs/modules/cli/guides/authentication.md`, `/docs/api/authentication.md`, `/docs/policy/examples/abac-overlays.md`, `/docs/install/configuration-reference.md` updates (imposed rule). |
| Sprint 49 | Authority-Backed Scopes & Tenancy Phase 3 | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-TEN-49-001 | Implement audit log pipeline, monitor scope usage, chaos tests for JWKS outage, and tenant load/perf tests. |
| Sprint 49 | Authority-Backed Scopes & Tenancy Phase 3 | src/Authority/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-TEN-49-001 | Implement service accounts, delegation tokens (`act` chain), per-tenant quotas, and audit log streaming. |
| Sprint 49 | Authority-Backed Scopes & Tenancy Phase 3 | src/Cli/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI Guild | CLI-TEN-49-001 | Add service account token minting, delegation, and `--impersonate` banner/controls. |
@@ -707,7 +707,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | docs/TASKS.md | TODO | Docs Guild | DOCS-OBS-50-004 | Publish tracing guide `/docs/observability/tracing.md` covering context propagation and sampling. |
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | docs/TASKS.md | TODO | Docs Guild | DOCS-SEC-OBS-50-001 | Update `/docs/security/redaction-and-privacy.md` for telemetry privacy controls. |
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | ops/devops/TASKS.md | DOING (2025-10-26) | DevOps Guild | DEVOPS-OBS-50-002 | Stand up multi-tenant metrics/logs/traces backends with retention and isolation. |
> Staging rollout plan recorded in `docs/ops/telemetry-storage.md`; waiting on Authority-issued tokens and namespace bootstrap.
> Staging rollout plan recorded in `docs/modules/telemetry/operations/storage.md`; waiting on Authority-issued tokens and namespace bootstrap.
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | src/Authority/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-OBS-50-001 | Introduce observability/timeline/evidence/attestation scopes and update discovery metadata. |
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | src/Cli/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI Guild | CLI-OBS-50-001 | Propagate trace headers from CLI commands and print correlation IDs. |
| Sprint 50 | Observability & Forensics Phase 1 – Baseline Telemetry | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-OBS-50-001 | Replace ad-hoc logging with telemetry core across advisory ingestion/linking. |

16
docs/implplan/SPRINTS_PRIOR_20251019.md
View File

@@ -36,7 +36,7 @@ Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation<br>Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep<br>Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling<br>`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.<br>Instructions to work:<br>DONE Read ./AGENTS.md and src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift |
@@ -72,14 +72,14 @@ Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/ops/concelier-cccs-operations.md` with fixtures validating EN/FR list handling. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/ops/concelier-certbund-operations.md` captures locale guidance and offline packaging. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/modules/concelier/operations/connectors/cccs.md` with fixtures validating EN/FR list handling. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/modules/concelier/operations/connectors/certbund.md` captures locale guidance and offline packaging. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/ops/concelier-msrc-operations.md`. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/modules/concelier/operations/connectors/msrc.md`. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. |
| Sprint 2 | Connector & Data Implementation Wave | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. |
| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh<br>Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. |
@@ -97,8 +97,8 @@ Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures |
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures |
| Sprint 3 | Conflict Resolution Integration & Communications | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures<br>Instructions to work:<br>Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. |
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules<br>Runbook published at `docs/ops/concelier-conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. |
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout<br>Ops review completed, alert thresholds applied, and change log appended in `docs/ops/concelier-conflict-resolution.md`; task closed after connector signals verified. |
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules<br>Runbook published at `docs/modules/concelier/operations/conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. |
| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout<br>Ops review completed, alert thresholds applied, and change log appended in `docs/modules/concelier/operations/conflict-resolution.md`; task closed after connector signals verified. |
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)<br>Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. |
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields<br>Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. |
| Sprint 4 | Schema Parity & Freshness Alignment | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup<br>Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. |
@@ -146,7 +146,7 @@ Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. |
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. |
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `modules/scanner/architecture.md` §3–§4. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. |
| Sprint 9 | Scanner Build-time | src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). |
@@ -174,7 +174,7 @@ Closed sprint tasks archived from SPRINTS.md on 2025-10-19.
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. |
| Sprint 9 | Scanner Core Foundations | src/Scanner/__Libraries/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. |
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors.<br>Progress 2025-10-20: Coordination matrix + rollout dashboard refreshed; upcoming deadlines tracked (Cccs/Cisco 2025-10-21, CertBund 2025-10-22, ICS-CISA 2025-10-23, KISA 2025-10-24) with escalation path documented in FEEDMERGE-COORD-02-900.|
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance<br>Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `src/Tools/FixtureUpdater` updates across connectors.<br>Progress 2025-10-20: Coordination matrix + rollout dashboard refreshed; upcoming deadlines tracked (Cccs/Cisco 2025-10-21, CertBund 2025-10-22, ICS-CISA 2025-10-23, KISA 2025-10-24) with escalation path documented in FEEDMERGE-COORD-02-900.|
| Sprint 1 | Stabilize In-Progress Foundations | src/Concelier/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path<br>Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. |
| Sprint 7 | Contextual Truth Foundations | src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – immutable VEX statements store, consensus signal fields, and migration `20251019-consensus-signals-statements` with tests (`dotnet test src/Excititor/__Tests/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj`, `dotnet test src/Excititor/__Tests/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). |
| Sprint 7 | Contextual Truth Foundations | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. |

4
docs/implplan/SPRINTS_PRIOR_20251027.md
View File

@@ -40,9 +40,9 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Authority Core | DOCS-POLICY-20-003 | Write `/docs/policy/lifecycle.md` covering workflow + roles. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Scheduler Guild | DOCS-POLICY-20-004 | Document policy run modes + cursors in `/docs/policy/runs.md`. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Platform Guild | DOCS-POLICY-20-005 | Produce `/docs/api/policy.md` with endpoint schemas + errors. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-POLICY-20-006 | Author `/docs/cli/policy.md` with commands, exit codes, JSON output. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, CLI Guild | DOCS-POLICY-20-006 | Author `/docs/modules/cli/guides/policy.md` with commands, exit codes, JSON output. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, UI Guild | DOCS-POLICY-20-007 | Create `/docs/ui/policy-editor.md` covering editor, simulation, approvals. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-POLICY-20-008 | Publish `/docs/architecture/policy-engine.md` with sequence diagrams. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Architecture Guild | DOCS-POLICY-20-008 | Publish `/docs/modules/policy/architecture.md` with sequence diagrams. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Observability Guild | DOCS-POLICY-20-009 | Document metrics/traces/logs in `/docs/observability/policy.md`. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Security Guild | DOCS-POLICY-20-010 | Publish `/docs/security/policy-governance.md` for scopes + approvals. |
| Sprint 20 | Policy Engine v2 | docs/TASKS.md | DONE (2025-10-26) | Docs Guild, Policy Guild | DOCS-POLICY-20-011 | Add example policies under `/docs/examples/policies/` with commentary. |