Files

master 651b8e0fa3 feat: Add new projects to solution and implement contract testing documentation

- Added "StellaOps.Policy.Engine", "StellaOps.Cartographer", and "StellaOps.SbomService" projects to the StellaOps solution.
- Created AGENTS.md to outline the Contract Testing Guild Charter, detailing mission, scope, and definition of done.
- Established TASKS.md for the Contract Testing Task Board, outlining tasks for Sprint 62 and Sprint 63 related to mock servers and replay testing.

2025-10-27 07:57:55 +02:00

21 KiB

Raw Blame History

Here’s the full write‑up you can drop into your repo as the canonical reference for Epic 1. It’s written in clean product‑doc style so it’s safe to check in as Markdown. No fluff, just everything you need to build, test, and police it.

Epic 1: Aggregation‑Only Contract (AOC) Enforcement

Short name: AOC enforcement Services touched: Conseiller (advisory ingestion), Excitator (VEX ingestion), Web API, Workers, Policy Engine, CLI, Console, Authority Data stores: MongoDB primary, optional Redis/NATS for jobs

1) What it is

Aggregation‑Only Contract (AOC) is the ingestion covenant for StellaOps. It defines a hard boundary between collection and interpretation:

Ingestion (Conseiller/Excitator) only collects data and preserves it as immutable raw facts with provenance. It does not decide, merge, normalize, prioritize, or assign severity. It may compute links that help future joins (aliases, PURLs, CPEs), but never derived judgments.
Policy evaluation is the only place where merges, deduplication, consensus, severity computation, and status folding are allowed. It’s reproducible and traceable.

The AOC establishes:

Immutable raw stores: advisory_raw and vex_raw documents with full provenance, signatures, checksums, and upstream identifiers.
Linksets: machine‑generated join hints (aliases, PURLs, CPEs, CVE/GHSA IDs) that never change the underlying source content.
Invariants: a strict set of “never do this in ingestion” rules enforced by schema validation, runtime guards, and CI checks.
AOC Verifier: a build‑time and runtime watchdog that blocks non‑compliant code and data writes.

This epic delivers: schemas, guards, error codes, APIs, tests, migration, docs, and ops dashboards to make AOC non‑negotiable across the platform.

2) Why

AOC makes results auditable, deterministic, and organization‑specific. Source vendors disagree; your policies decide. By removing hidden heuristics from ingestion, we avoid unexplainable risk changes, race conditions between collectors, and vendor bias. Policy‑time evaluation yields reproducible deltas with complete “why” traces.

3) How it should work (deep details)

3.1 Core invariants

The following must be true for every write to advisory_raw and vex_raw and for every ingestion pipeline:

No severity in ingestion
- Forbidden fields: severity, cvss, cvss_vector, effective_status, effective_range, merged_from, consensus_provider, reachability, asset_criticality, risk_score.
No merges or de‑dups in ingestion
- No combining two upstream advisories into one. No picking a single truth when multiple VEX statements exist.
Provenance is mandatory
- Every raw doc includes provenance and signature/checksum.
Idempotent upserts
- Same upstream document (by upstream_id + source + content_hash) must not create duplicates.
Append‑only versioning
- Revisions from the source create new immutable documents with supersedes pointers; no in‑place edits.
Linkset only
- Ingestion can compute and store a linkset for join performance. Linkset does not alter or infer severity/status.
Policy‑time only for effective findings
- Only the Policy Engine can write effective_finding_* materializations.
Schema safety
- Strict JSON schema validation at DB level; unknown fields reject writes.
Clock discipline
- Timestamps are UTC, monotonic within a batch; collectors record fetched_at and received_at.

3.2 Data model

3.2.1 `advisory_raw` (Mongo collection)

{
  "_id": "advisory_raw:osv:GHSA-xxxx-....:v3",
  "source": {
    "vendor": "OSV",
    "stream": "github", 
    "api": "https://api.osv.dev/v1/.../GHSA-...",
    "collector_version": "conseiller/1.7.3"
  },
  "upstream": {
    "upstream_id": "GHSA-xxxx-....",
    "document_version": "2024-09-01T12:13:14Z",
    "fetched_at": "2025-01-02T03:04:05Z",
    "received_at": "2025-01-02T03:04:06Z",
    "content_hash": "sha256:...",
    "signature": {
      "present": true,
      "format": "dsse",
      "key_id": "rekor:.../key/abc",
      "sig": "base64..."
    }
  },
  "content": {
    "format": "OSV",
    "spec_version": "1.6",
    "raw": { /* full upstream JSON, unmodified */ }
  },
  "identifiers": {
    "cve": ["CVE-2023-1234"],
    "ghsa": ["GHSA-xxxx-...."],
    "aliases": ["CVE-2023-1234", "GHSA-xxxx-...."]
  },
  "linkset": {
    "purls": ["pkg:npm/lodash@4.17.21", "pkg:maven/..."],
    "cpes": ["cpe:2.3:a:..."],
    "references": [
      {"type":"advisory","url":"https://..."},
      {"type":"fix","url":"https://..."}
    ],
    "reconciled_from": ["content.raw.affected.ranges", "content.raw.pkg"]
  },
  "supersedes": "advisory_raw:osv:GHSA-xxxx-....:v2",
  "tenant": "default"
}

Note: No severity, no cvss, no effective_*. If the upstream payload includes CVSS, it stays inside content.raw and is not promoted or normalized at ingestion.

3.2.2 `vex_raw` (Mongo collection)

{
  "_id": "vex_raw:vendorX:doc-123:v4",
  "source": {
    "vendor": "VendorX",
    "stream": "vex",
    "api": "https://.../vex/doc-123",
    "collector_version": "excitator/0.9.2"
  },
  "upstream": {
    "upstream_id": "doc-123",
    "document_version": "2025-01-15T08:09:10Z",
    "fetched_at": "2025-01-16T01:02:03Z",
    "received_at": "2025-01-16T01:02:03Z",
    "content_hash": "sha256:...",
    "signature": { "present": true, "format": "cms", "key_id": "kid:...", "sig": "..." }
  },
  "content": {
    "format": "CycloneDX-VEX",   // or "CSAF-VEX"
    "spec_version": "1.5",
    "raw": { /* full upstream VEX */ }
  },
  "identifiers": {
    "statements": [
      {
        "advisory_ids": ["CVE-2023-1234","GHSA-..."],
        "component_purls": ["pkg:deb/openssl@1.1.1"],
        "status": "not_affected",
        "justification": "component_not_present"
      }
    ]
  },
  "linkset": {
    "purls": ["pkg:deb/openssl@1.1.1"],
    "cves": ["CVE-2023-1234"],
    "ghsas": ["GHSA-..."]
  },
  "supersedes": "vex_raw:vendorX:doc-123:v3",
  "tenant": "default"
}

VEX statuses remain as raw facts. No cross‑provider consensus is computed here.

3.3 Database validation

MongoDB JSON Schema validators on both collections:
- Reject forbidden fields at the top level.
- Enforce presence of source, upstream, content, linkset, tenant.
- Enforce string formats for timestamps and hashes.

3.4 Write paths

Collector fetches upstream
- Normalize transport (gzip/json), compute content_hash, verify signature if available.
Build raw doc
- Populate source, upstream, content.raw, identifiers, linkset.
Idempotent upsert
- Lookup by (source.vendor, upstream.upstream_id, upstream.content_hash). If exists, skip; if new content hash, insert new revision with supersedes.
AOC guard
- Runtime interceptor inspects write payload; if any forbidden field detected, reject with ERR_AOC_001.
Metrics
- Emit ingestion_write_ok or ingestion_write_reject with reason code.

3.5 Read paths (ingestion scope)

Allow only listing, getting raw docs, and searching by linkset. No endpoints return “effective findings” from ingestion services.

3.6 Error codes

Code	Meaning	HTTP
`ERR_AOC_001`	Forbidden field present (severity/consensus/normalized data)	400
`ERR_AOC_002`	Merge attempt detected (multiple upstreams fused)	400
`ERR_AOC_003`	Idempotency violation (duplicate without supersedes)	409
`ERR_AOC_004`	Missing provenance fields	422
`ERR_AOC_005`	Signature/checksum mismatch	422
`ERR_AOC_006`	Attempt to write effective findings from ingestion context	403
`ERR_AOC_007`	Unknown top‑level fields (schema violation)	400

3.7 AOC Verifier

A build‑time and runtime safeguard:

Static checks (CI)
- Block imports of *.Policy* or *.Merge* from ingestion modules.
- AST lint rule: any write to advisory_raw or vex_raw setting a forbidden key fails the build.
Runtime checks
- Repository layer interceptor inspects documents before insert/update; rejects forbidden fields and multi‑source merges.
Drift detection job
- Nightly job scans newest N docs; if violation found, pages ops and blocks new pipeline runs.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.8 Indexing strategy

advisory_raw:
- { "identifiers.cve": 1 }, { "identifiers.ghsa": 1 }, { "linkset.purls": 1 }, { "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 } (unique), { "tenant": 1 }.
vex_raw:
- { "identifiers.statements.advisory_ids": 1 }, { "linkset.purls": 1 }, { "source.vendor": 1, "upstream.upstream_id": 1, "upstream.content_hash": 1 } (unique), { "tenant": 1 }.

3.9 Interaction with Policy Engine

Policy Engine pulls raw docs by identifiers/linksets and computes:
- De‑dup/merge per policy
- Consensus for VEX statements
- Severity normalization and risk scoring
Writes only to effective_finding_{policyId} collections.

A dedicated write guard refuses effective_finding_* writes from any caller that isn’t the Policy Engine service identity.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.10 Migration plan

Freeze ingestion writes except raw pass‑through.
Backfill: copy existing ingestion collections to _backup_*.
Strip forbidden fields from raw copies, move them into a temporary advisory_view_legacy used only by Policy Engine for parity.
Enable DB schema validators.
Run collectors in dry‑run; ensure only allowed keys land.
Switch Policy Engine to pull exclusively from *_raw and to compute everything else.
Delete legacy normalized fields in ingestion codepaths.
Enable runtime guards and CI lint.

3.11 Observability

Metrics:
- aoc_violation_total{code=...}, ingestion_write_total{result=ok|reject}, ingestion_signature_verified_total{result=ok|fail}, ingestion_latency_seconds, advisory_revision_count.
Tracing: span ingest.fetch, ingest.transform, ingest.write, aoc.guard.
Logs: include tenant, source.vendor, upstream.upstream_id, content_hash, correlation_id.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.12 Security and tenancy

Every raw doc carries a tenant field.
Authority enforces advisory:write and vex:write scopes for ingestion endpoints.
Cross‑tenant reads/writes are blocked by default.
Secrets never logged; signatures verified with pinned trust stores.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.13 CLI and Console behavior

CLI
- stella sources ingest --dry-run prints would‑write payload and explicitly shows that no severity/status fields are present.
- stella aoc verify scans last K documents and reports violations with exit codes.
Console
- Sources dashboard shows AOC pass/fail per job, most recent violation codes, and a drill‑down to the offending document.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

4) API surface (ingestion scope)

4.1 Conseiller (Advisories)

POST /ingest/advisory
- Body: raw upstream advisory with metadata; server constructs document, not the client.
- Rejections: ERR_AOC_00x per table above.
GET /advisories/raw/{id}
GET /advisories/raw?cve=CVE-...&purl=pkg:...&tenant=...
GET /advisories/raw/{id}/provenance
POST /aoc/verify?since=ISO8601 returns summary stats and first N violations.

4.2 Excitator (VEX)

POST /ingest/vex
GET /vex/raw/{id}
GET /vex/raw?advisory_id=CVE-...&purl=pkg:...
POST /aoc/verify?since=ISO8601

All endpoints require tenant scope and appropriate :write or :read.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

5) Example: end‑to‑end flow

Collector fetches GHSA-1234 from OSV.
Build advisory_raw with linkset PURLs.
Insert; AOC guard approves.
Policy Engine later evaluates SBOM S-42 under policy P-7, reads raw advisory and any VEX raw docs, computes effective findings, and writes to effective_finding_P-7.
CLI stella aoc verify --since 24h returns 0 violations.

6) Implementation tasks

Breakdown by component with exact work items. Each section ends with the imposed sentence you requested.