git.stella-ops.org/docs/modules/scanner/design/competitor-ingest-normalization.md

Competitor Ingest Normalization (CM1)

Purpose

Define how external SBOM/scan outputs (Syft, Trivy, Clair) are normalized into StellaOps schemas with deterministic ordering, provenance checks, and offline-ready adapters. Covers CM1–CM10 in the 31-Nov-2025 findings advisory.

Scope

• Import pipeline for external SBOM + vulnerability scan payloads.
• Adapter mappings, validation, provenance/signature verification, and fallback rules.
• Offline ingest kits (adapters + fixtures) and regression tests.

Deliverables (CM tasks)

• CM1: Mapping tables per tool → StellaOps SBOM/scan schema; required/optional fields; deterministic sort rules.
• CM2: Signature/provenance verification policy (acceptable algorithms, trust roots, failure modes).
• CM3: Snapshot governance: versioning, freshness SLA, rollback plan for imported feeds.
• CM4: Anomaly regression suite (schema drift, nullables, encoding, ordering). Golden fixtures + hashes.
• CM5: Offline ingest kit: DSSE-signed adapters/mappings/fixtures with tool versions and hashes.
• CM6: Fallback hierarchy when data incomplete (signed SBOM → unsigned SBOM → scan → defaults) with explicit decision trace.
• CM7: Source transparency fields (tool name/version/hash, build metadata) persisted and surfaced.
• CM8: Benchmark parity plan with upstream tools (pinned versions, hash-logged runs).
• CM9: Coverage matrix by ecosystem; gap tracker.
• CM10: Retry/backoff/error taxonomy and deterministic diagnostics.

Determinism & Validation

• Adapters must sort components and vulnerabilities deterministically (locale-invariant, stable keys).
• All mapping rules and fixtures carry BLAKE3/SHA256 hashes; adapters are pure functions (no network).
• Signature verification rejects unverifiable payloads; logs reason codes; can run offline using bundled trust roots.

Adapter mapping skeleton (CM1)

• Tool coverage v0.1: Syft 1.0.x, Trivy 0.50.x, Clair 6.x (pin exact versions in fixtures).
• Mapping tables (CSV, checked in under docs/modules/scanner/fixtures/competitor-adapters/):
  • component: external fields → name, version, purl, type, hashes, licenses, evidenceRef.
  • vulnerability: id, source, severity (normalised), cvss (score/vector), fixVersions, evidenceRef.
  • metadata: tool name/version/hash, scan timestamp (UTC), data source.
• Sorting: components by purl → name → version; vulns by id → source → severityScore desc → cvss.vector.

Verification policy (CM2)

• Acceptable signatures: DSSE/COSE/JWS with SHA256/Ed25519/ECDSA; trust roots bundled in offline kit.
• Provenance check: require signer identity + hash match; if missing, mark provenance = unknown and apply fallback (CM6).

Snapshot governance (CM3)

• Freshness budget: max age 7 days from scanTimestamp; reject older unless override flag set (logged).
• Versioning: stored as snapshot_version (semver) and source_tool_hash; rollback plan requires prior snapshot hash.

Regression + fixtures (CM4/CM5)

• Fixtures under docs/modules/scanner/fixtures/competitor-adapters/fixtures/ with golden hashes (BLAKE3/SHA256) and expected normalized output.
  • normalized-syft.json BLAKE3=aa42c167d19535709a10df73dc39e6a50b8efbbb0ae596d17183ce62676fa85a SHA256=3f8684ff341808dcb92e97dd2c10acca727baaff05182e81a4364bb3dad0eaa7
  • normalized-trivy.json BLAKE3=0da216b49ebcf823d8d4aa3c9c1d2a1dcc579d836ba66bb2ae94dd781e214130 SHA256=c29aa6251d378c2aca1c3c6165e61bd2e16b6fa1227c976417b8a525ad7c1fc1
  • normalized-clair.json BLAKE3=92985f4cbdeecc8a0e585a70e07f17b07abdd866eecacaca9ba1b331f4b3af68 SHA256=bc232cc19885c53e4d801f5c830e3683a4031e42f6421739c4cc221f33f15e01
• CI step runs adapter → normalized → hash compare; offline, no network. Hashes act as guardrails for deterministic ordering and mapping stability.

Fallback hierarchy (CM6)

1. Signed SBOM w/ valid provenance → accepted.
2. Unsigned SBOM → accepted with provenance=unknown, warnings emitted.
3. Scan-only results → accepted with degraded confidence; policy lattice may penalize.
4. If all absent: reject with reason code no_evidence.

Transparency & coverage (CM7–CM9)

• Persist: source.tool, source.version, source.hash, adapter.version, normalized_hash.
• Coverage matrix maintained in docs/modules/scanner/fixtures/competitor-adapters/coverage.csv (ecosystem yes/no, notes). Current snapshot (2025-12-03): container/java/python/go/os rows populated; dotnet pending Syft/Clair support.
• Bench parity (CM8): pin upstream versions; store run hashes/logs in fixtures folder.

Error taxonomy (CM10)

• Retryable: network/unavailable (should not occur in offline mode), rate-limit, transient IO.
• Non-retryable: signature_invalid, schema_invalid, unsupported_version, no_evidence.
• All errors must carry deterministic reason codes and be logged in normalized output metadata.

Offline kit (CM5)

• Kit contents: adapter CSVs (one per tool), fixtures + hashes above, coverage matrix, trust roots, signature policy, retry taxonomy, and DSSE envelope referencing every file hash. Bundle path: out/offline/competitor-ingest-kit-v1/.

Decisions (2025-12-03)

• Minimal evidence for unsigned SBOM acceptance: must include tool metadata (name/version/hash), component list with purl + SHA256, and scan timestamp; otherwise fallback to scan-only path (CM6 step 3).
• Canonical adapter output hash: BLAKE3 primary, SHA256 secondary; both recorded in fixture hash lists and surfaced in normalized metadata (normalized_hash).
• Signature verification policy is strict fail-closed unless --allow-unsigned flag explicitly set; fallback hierarchy still applied but logged.

Links

• Sprint: docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md (CM1–CM10)
• Advisory: docs/product-advisories/31-Nov-2025 FINDINGS.md