product advisories, stella router improval, tests streghthening
This commit is contained in:
StellaOps Bot
@@ -24,25 +24,25 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **L0 Policy Engine** | | | | | |
|
||||
| 1 | POLICY-5100-001 | TODO | TestKit | Policy Guild | Add property tests for policy evaluation monotonicity: tightening risk budget cannot decrease severity. |
|
||||
| 2 | POLICY-5100-002 | TODO | TestKit | Policy Guild | Add property tests for unknown handling: if unknowns > N then fail verdict (where configured). |
|
||||
| 3 | POLICY-5100-003 | TODO | TestKit | Policy Guild | Add property tests for merge semantics: verify join/meet properties for lattice merge rules. |
|
||||
| 4 | POLICY-5100-004 | TODO | TestKit | Policy Guild | Add snapshot tests for verdict artifact canonical JSON (auditor-facing output). |
|
||||
| 5 | POLICY-5100-005 | TODO | TestKit | Policy Guild | Add snapshot tests for policy evaluation trace summary (stable structure). |
|
||||
| 1 | POLICY-5100-001 | DONE | TestKit | Policy Guild | Add property tests for policy evaluation monotonicity: tightening risk budget cannot decrease severity. |
|
||||
| 2 | POLICY-5100-002 | DONE | TestKit | Policy Guild | Add property tests for unknown handling: if unknowns > N then fail verdict (where configured). |
|
||||
| 3 | POLICY-5100-003 | DONE | TestKit | Policy Guild | Add property tests for merge semantics: verify join/meet properties for lattice merge rules. |
|
||||
| 4 | POLICY-5100-004 | DONE | TestKit | Policy Guild | Add snapshot tests for verdict artifact canonical JSON (auditor-facing output). |
|
||||
| 5 | POLICY-5100-005 | DONE | TestKit | Policy Guild | Add snapshot tests for policy evaluation trace summary (stable structure). |
|
||||
| **L0 Policy DSL** | | | | | |
|
||||
| 6 | POLICY-5100-006 | TODO | TestKit | Policy Guild | Add property tests for DSL parser: roundtrips (parse → print → parse). |
|
||||
| 7 | POLICY-5100-007 | TODO | TestKit | Policy Guild | Add golden tests for PolicyDslValidator: common invalid policy patterns. |
|
||||
| 6 | POLICY-5100-006 | DONE | TestKit | Policy Guild | Add property tests for DSL parser: roundtrips (parse → print → parse). |
|
||||
| 7 | POLICY-5100-007 | DONE | TestKit | Policy Guild | Add golden tests for PolicyDslValidator: common invalid policy patterns. |
|
||||
| **S1 Storage** | | | | | |
|
||||
| 8 | POLICY-5100-008 | DONE | Storage harness | Policy Guild | Add policy versioning immutability tests (published policies cannot be mutated). |
|
||||
| 9 | POLICY-5100-009 | DONE | Storage harness | Policy Guild | Add retrieval ordering determinism tests (explicit ORDER BY checks). |
|
||||
| 10 | POLICY-5100-010 | DONE | Storage harness | Policy Guild | Add migration tests for Policy.Storage (apply from scratch, apply from N-1). |
|
||||
| **W1 Gateway/API** | | | | | |
|
||||
| 11 | POLICY-5100-011 | TODO | WebService fixture | Policy Guild | Add contract tests for Policy Gateway endpoints (policy retrieval, verdict submission) — OpenAPI snapshot. |
|
||||
| 12 | POLICY-5100-012 | TODO | WebService fixture | Policy Guild | Add auth tests (deny-by-default, token expiry, scope enforcement). |
|
||||
| 13 | POLICY-5100-013 | TODO | WebService fixture | Policy Guild | Add OTel trace assertions (verify policy_id, tenant_id, verdict_id tags). |
|
||||
| 11 | POLICY-5100-011 | DONE | WebService fixture | Policy Guild | Add contract tests for Policy Gateway endpoints (policy retrieval, verdict submission) — OpenAPI snapshot. |
|
||||
| 12 | POLICY-5100-012 | DONE | WebService fixture | Policy Guild | Add auth tests (deny-by-default, token expiry, scope enforcement). |
|
||||
| 13 | POLICY-5100-013 | DONE | WebService fixture | Policy Guild | Add OTel trace assertions (verify policy_id, tenant_id, verdict_id tags). |
|
||||
| **Determinism & Quality Gates** | | | | | |
|
||||
| 14 | POLICY-5100-014 | TODO | Determinism gate | Policy Guild | Add determinism test: same policy + same inputs → same verdict artifact hash. |
|
||||
| 15 | POLICY-5100-015 | TODO | Determinism gate | Policy Guild | Add unknown budget enforcement test: validate "fail if unknowns > N" behavior. |
|
||||
| 14 | POLICY-5100-014 | DONE | Determinism gate | Policy Guild | Add determinism test: same policy + same inputs → same verdict artifact hash. |
|
||||
| 15 | POLICY-5100-015 | DONE | Determinism gate | Policy Guild | Add unknown budget enforcement test: validate "fail if unknowns > N" behavior. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (L0 Engine + DSL):** Tasks 1-7.
|
||||
@@ -91,3 +91,6 @@
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created for Policy module test implementation based on advisory Section 3.4 and TEST_CATALOG.yml. | Project Mgmt |
|
||||
| 2025-12-24 | Tasks 8-10 DONE: Added S1 Storage tests. Task 8: `PolicyVersioningImmutabilityTests.cs` (11 tests: published versions immutable, hash/timestamp preserved, version history append-only, activation doesn't modify content). Task 9: `PolicyQueryDeterminismTests.cs` (12 tests: GetAllPacks, GetPackVersions, GetRiskProfiles, GetRules, GetAuditEntries ordering, concurrent queries, tenant isolation). Task 10: `PolicyMigrationTests.cs` (8 tests: from scratch, idempotency, schema integrity, FK constraints, policy tables). | Implementer |
|
||||
| 2025-12-24 | Tasks 1-5 DONE: Added L0 Policy Engine tests. Task 1: `RiskBudgetMonotonicityPropertyTests.cs` (6 property tests: tightening budget increases violations, idempotency, commutativity). Task 2: `UnknownsBudgetPropertyTests.cs` (6 property tests: fail if unknowns > N, severity tracking). Task 3: `VexLatticeMergePropertyTests.cs` (8 property tests: K4 lattice join/meet/absorption). Task 4: `VerdictArtifactSnapshotTests.cs` (6 snapshot tests: passing/failing/unknowns/VEX merge verdicts). Task 5: `PolicyEvaluationTraceSnapshotTests.cs` (5 snapshot tests: trace structure). | Implementer |
|
||||
| 2025-12-24 | Tasks 6-7 DONE: Added L0 Policy DSL tests. Task 6: `PolicyDslRoundtripPropertyTests.cs` (6 property tests: parse→print→parse roundtrip, name/rule/metadata preservation, checksum stability). Task 7: `PolicyDslValidationGoldenTests.cs` (26 golden tests: syntax errors, rule errors, expression errors, metadata/profile errors, edge cases). | Implementer |
|
||||
| 2025-12-24 | Tasks 11-15 DONE: Added W1 Gateway tests and Determinism tests. Task 11-13: `PolicyGatewayIntegrationTests.cs` (15 tests: contract validation for exceptions/deltas endpoints, auth deny-by-default, token expiry, scope enforcement, OTel trace assertions). Task 14-15: `PolicyEngineDeterminismTests.cs` (12 tests: same inputs→same hash, order independence, concurrent evaluation, VEX merge determinism, unknowns budget enforcement). | Implementer |
|
||||
|
||||
@@ -37,9 +37,9 @@
|
||||
| 10 | SCHEDULER-5100-010 | DONE | WebService fixture | Scheduler Guild | Add OTel trace assertions (verify job_id, tenant_id, schedule_id tags). |
|
||||
| **WK1 Worker** | | | | | |
|
||||
| 11 | SCHEDULER-5100-011 | DONE | Storage harness | Scheduler Guild | Add end-to-end test: enqueue job → worker picks up → executes → completion recorded. |
|
||||
| 12 | SCHEDULER-5100-012 | DOING | Storage harness | Scheduler Guild | Add retry tests: transient failure uses exponential backoff; permanent failure routes to poison queue. |
|
||||
| 13 | SCHEDULER-5100-013 | TODO | Storage harness | Scheduler Guild | Add idempotency tests: same job processed twice → single execution result. |
|
||||
| 14 | SCHEDULER-5100-014 | TODO | Storage harness | Scheduler Guild | Add OTel correlation tests: verify trace spans across job lifecycle (enqueue → pick → execute → complete). |
|
||||
| 12 | SCHEDULER-5100-012 | DONE | Storage harness | Scheduler Guild | Add retry tests: transient failure uses exponential backoff; permanent failure routes to poison queue. |
|
||||
| 13 | SCHEDULER-5100-013 | DONE | Storage harness | Scheduler Guild | Add idempotency tests: same job processed twice → single execution result. |
|
||||
| 14 | SCHEDULER-5100-014 | DONE | Storage harness | Scheduler Guild | Add OTel correlation tests: verify trace spans across job lifecycle (enqueue → pick → execute → complete). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (L0 Scheduling Logic):** Tasks 1-4.
|
||||
|
||||
@@ -24,28 +24,28 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **C1 Notification Connectors** | | | | | |
|
||||
| 1 | NOTIFY-5100-001 | TODO | Connector fixtures | Notify Guild | Set up fixture folders for email connector: `Fixtures/email/<case>.json` (event), `Expected/<case>.email.txt` (formatted email). |
|
||||
| 2 | NOTIFY-5100-002 | TODO | Task 1 | Notify Guild | Add payload formatting snapshot tests for email connector: event → formatted email → assert snapshot. |
|
||||
| 3 | NOTIFY-5100-003 | TODO | Task 1 | Notify Guild | Add error handling tests for email connector: SMTP unavailable → retry; invalid recipient → fail gracefully. |
|
||||
| 4 | NOTIFY-5100-004 | TODO | Connector fixtures | Notify Guild | Repeat fixture setup for Slack connector (Tasks 1-3 pattern). |
|
||||
| 5 | NOTIFY-5100-005 | TODO | Connector fixtures | Notify Guild | Repeat fixture setup for Teams connector (Tasks 1-3 pattern). |
|
||||
| 6 | NOTIFY-5100-006 | TODO | Connector fixtures | Notify Guild | Repeat fixture setup for webhook connector (Tasks 1-3 pattern). |
|
||||
| 1 | NOTIFY-5100-001 | DONE | Connector fixtures | Notify Guild | Set up fixture folders for email connector: `Fixtures/email/<case>.json` (event), `Expected/<case>.email.txt` (formatted email). |
|
||||
| 2 | NOTIFY-5100-002 | DONE | Task 1 | Notify Guild | Add payload formatting snapshot tests for email connector: event → formatted email → assert snapshot. |
|
||||
| 3 | NOTIFY-5100-003 | DONE | Task 1 | Notify Guild | Add error handling tests for email connector: SMTP unavailable → retry; invalid recipient → fail gracefully. |
|
||||
| 4 | NOTIFY-5100-004 | DONE | Connector fixtures | Notify Guild | Repeat fixture setup for Slack connector (Tasks 1-3 pattern). |
|
||||
| 5 | NOTIFY-5100-005 | DONE | Connector fixtures | Notify Guild | Repeat fixture setup for Teams connector (Tasks 1-3 pattern). |
|
||||
| 6 | NOTIFY-5100-006 | DONE | Connector fixtures | Notify Guild | Repeat fixture setup for webhook connector (Tasks 1-3 pattern). |
|
||||
| **L0 Core Logic** | | | | | |
|
||||
| 7 | NOTIFY-5100-007 | TODO | TestKit | Notify Guild | Add unit tests for notification templating: event data + template → rendered notification. |
|
||||
| 8 | NOTIFY-5100-008 | TODO | TestKit | Notify Guild | Add unit tests for rate limiting: too many notifications → throttled. |
|
||||
| 7 | NOTIFY-5100-007 | DONE | TestKit | Notify Guild | Add unit tests for notification templating: event data + template → rendered notification. |
|
||||
| 8 | NOTIFY-5100-008 | DONE | TestKit | Notify Guild | Add unit tests for rate limiting: too many notifications → throttled. |
|
||||
| **S1 Storage** | | | | | |
|
||||
| 9 | NOTIFY-5100-009 | DONE | Storage harness | Notify Guild | Add migration tests for Notify.Storage (apply from scratch, apply from N-1). |
|
||||
| 10 | NOTIFY-5100-010 | DONE | Storage harness | Notify Guild | Add idempotency tests: same notification ID enqueued twice → single delivery. |
|
||||
| 11 | NOTIFY-5100-011 | DONE | Storage harness | Notify Guild | Add retry state persistence tests: failed notification → retry state saved → retry on next poll. |
|
||||
| **W1 WebService** | | | | | |
|
||||
| 12 | NOTIFY-5100-012 | TODO | WebService fixture | Notify Guild | Add contract tests for Notify.WebService endpoints (send notification, query status) — OpenAPI snapshot. |
|
||||
| 13 | NOTIFY-5100-013 | TODO | WebService fixture | Notify Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
|
||||
| 14 | NOTIFY-5100-014 | TODO | WebService fixture | Notify Guild | Add OTel trace assertions (verify notification_id, channel, recipient tags). |
|
||||
| 12 | NOTIFY-5100-012 | DONE | WebService fixture | Notify Guild | Add contract tests for Notify.WebService endpoints (send notification, query status) — OpenAPI snapshot. |
|
||||
| 13 | NOTIFY-5100-013 | DONE | WebService fixture | Notify Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
|
||||
| 14 | NOTIFY-5100-014 | DONE | WebService fixture | Notify Guild | Add OTel trace assertions (verify notification_id, channel, recipient tags). |
|
||||
| **WK1 Worker** | | | | | |
|
||||
| 15 | NOTIFY-5100-015 | TODO | Storage harness | Notify Guild | Add end-to-end test: event emitted → notification queued → worker delivers via stub handler → delivery confirmed. |
|
||||
| 16 | NOTIFY-5100-016 | TODO | Storage harness | Notify Guild | Add retry tests: transient failure (e.g., SMTP timeout) → exponential backoff; permanent failure → poison queue. |
|
||||
| 17 | NOTIFY-5100-017 | TODO | Storage harness | Notify Guild | Add rate limit tests: verify rate limiting behavior (e.g., max 10 emails/min). |
|
||||
| 18 | NOTIFY-5100-018 | TODO | Storage harness | Notify Guild | Add OTel correlation tests: verify trace spans across notification lifecycle (enqueue → deliver → confirm). |
|
||||
| 15 | NOTIFY-5100-015 | DONE | Storage harness | Notify Guild | Add end-to-end test: event emitted → notification queued → worker delivers via stub handler → delivery confirmed. |
|
||||
| 16 | NOTIFY-5100-016 | DONE | Storage harness | Notify Guild | Add retry tests: transient failure (e.g., SMTP timeout) → exponential backoff; permanent failure → poison queue. |
|
||||
| 17 | NOTIFY-5100-017 | DONE | Storage harness | Notify Guild | Add rate limit tests: verify rate limiting behavior (e.g., max 10 emails/min). |
|
||||
| 18 | NOTIFY-5100-018 | DONE | Storage harness | Notify Guild | Add OTel correlation tests: verify trace spans across notification lifecycle (enqueue → deliver → confirm). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (C1 Connectors):** Tasks 1-6.
|
||||
@@ -93,3 +93,5 @@
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created for Notify module test implementation based on advisory Section 3.10 and TEST_CATALOG.yml. | Project Mgmt |
|
||||
| 2025-12-24 | Tasks 9-11 DONE: Added S1 Storage tests. Task 9: `NotifyMigrationTests.cs` (8 tests: from scratch, idempotency, schema integrity, FK constraints, deliveries/channels tables, notify schema). Task 10: `DeliveryIdempotencyTests.cs` (10 tests: duplicate ID rejection, correlation ID lookup, tenant isolation, delivered/failed notifications). Task 11: `RetryStatePersistenceTests.cs` (10 tests: retry state persistence, attempt count, error message preservation, independent retry states). | Implementer |
|
||||
| 2025-12-24 | Task 6 DONE: Added Webhook connector tests. Created `StellaOps.Notify.Connectors.Webhook.Tests` project with Fixtures/webhook/*.json (3 event fixtures), Expected/*.webhook.json (3 expected outputs), Snapshot/WebhookConnectorSnapshotTests.cs (10 tests: payload serialization, HMAC-SHA256 signatures, Content-Type headers, determinism, metadata propagation), ErrorHandling/WebhookConnectorErrorHandlingTests.cs (12 tests: endpoint unavailable, timeouts, HTTP errors, signature mismatches, malformed payloads). | Implementer |
|
||||
| 2025-12-24 | Tasks 15-18 DONE: Verified all WK1 Worker test files exist in `src/Notify/__Tests/StellaOps.Notify.Worker.Tests/WK1/`: NotifyWorkerEndToEndTests.cs (Task 15), NotifyWorkerRetryTests.cs (Task 16), NotifyWorkerRateLimitTests.cs (Task 17), NotifyWorkerOTelCorrelationTests.cs (Task 18). Sprint complete. | Implementer |
|
||||
|
||||
@@ -28,17 +28,17 @@
|
||||
| 3 | CLI-5100-003 | DONE | TestKit | CLI Guild | Add exit code tests: system error (API unavailable) → exit 2. |
|
||||
| 4 | CLI-5100-004 | DONE | TestKit | CLI Guild | Add exit code tests: permission denied → exit 3. |
|
||||
| **CLI1 Golden Output** | | | | | |
|
||||
| 5 | CLI-5100-005 | TODO | TestKit | CLI Guild | Add golden output tests for `stellaops scan` command: stdout snapshot (SBOM summary). |
|
||||
| 6 | CLI-5100-006 | TODO | TestKit | CLI Guild | Add golden output tests for `stellaops verify` command: stdout snapshot (verdict summary). |
|
||||
| 7 | CLI-5100-007 | TODO | TestKit | CLI Guild | Add golden output tests for `stellaops policy list` command: stdout snapshot (policy list). |
|
||||
| 8 | CLI-5100-008 | TODO | TestKit | CLI Guild | Add golden output tests for error scenarios: stderr snapshot (error messages). |
|
||||
| 5 | CLI-5100-005 | DONE | TestKit | CLI Guild | Add golden output tests for `stellaops scan` command: stdout snapshot (SBOM summary). |
|
||||
| 6 | CLI-5100-006 | DONE | TestKit | CLI Guild | Add golden output tests for `stellaops verify` command: stdout snapshot (verdict summary). |
|
||||
| 7 | CLI-5100-007 | DONE | TestKit | CLI Guild | Add golden output tests for `stellaops policy list` command: stdout snapshot (policy list). |
|
||||
| 8 | CLI-5100-008 | DONE | TestKit | CLI Guild | Add golden output tests for error scenarios: stderr snapshot (error messages). |
|
||||
| **CLI1 Determinism** | | | | | |
|
||||
| 9 | CLI-5100-009 | TODO | Determinism gate | CLI Guild | Add determinism test: same scan inputs → same SBOM output (byte-for-byte, excluding timestamps). |
|
||||
| 10 | CLI-5100-010 | TODO | Determinism gate | CLI Guild | Add determinism test: same policy + same inputs → same verdict output. |
|
||||
| 9 | CLI-5100-009 | DONE | Determinism gate | CLI Guild | Add determinism test: same scan inputs → same SBOM output (byte-for-byte, excluding timestamps). |
|
||||
| 10 | CLI-5100-010 | DONE | Determinism gate | CLI Guild | Add determinism test: same policy + same inputs → same verdict output. |
|
||||
| **Integration Tests** | | | | | |
|
||||
| 11 | CLI-5100-011 | TODO | TestKit | CLI Guild | Add integration test: CLI `stellaops scan` → calls Scanner.WebService → returns SBOM. |
|
||||
| 12 | CLI-5100-012 | TODO | TestKit | CLI Guild | Add integration test: CLI `stellaops verify` → calls Policy.Gateway → returns verdict. |
|
||||
| 13 | CLI-5100-013 | TODO | TestKit | CLI Guild | Add offline mode test: CLI with `--offline` flag → does not call WebService → uses local cache. |
|
||||
| 11 | CLI-5100-011 | DONE | TestKit | CLI Guild | Add integration test: CLI `stellaops scan` → calls Scanner.WebService → returns SBOM. |
|
||||
| 12 | CLI-5100-012 | DONE | TestKit | CLI Guild | Add integration test: CLI `stellaops verify` → calls Policy.Gateway → returns verdict. |
|
||||
| 13 | CLI-5100-013 | DONE | TestKit | CLI Guild | Add offline mode test: CLI with `--offline` flag → does not call WebService → uses local cache. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (CLI1 Exit Codes + Golden Output):** Tasks 1-8.
|
||||
@@ -85,3 +85,4 @@
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created for CLI module test implementation based on advisory Model CLI1 and TEST_CATALOG.yml. | Project Mgmt |
|
||||
| 2025-12-24 | Tasks 1-4 DONE: Created `CliExitCodeTests.cs` with 28 tests covering: (1) CLI-5100-001 - ProofExitCodes/OfflineExitCodes/DriftExitCodes Success is 0, IsSuccess range tests; (2) CLI-5100-002 - InputError/PolicyViolation/FileNotFound user errors; (3) CLI-5100-003 - SystemError/NetworkError/StorageError system errors; (4) CLI-5100-004 - VerificationFailed/SignatureFailure/PolicyDenied permission errors. Also added POSIX convention tests, exit code uniqueness tests, and DriftCommandResult tests. Updated csproj with FluentAssertions and test SDK packages. | Implementer |
|
||||
| 2025-12-24 | Tasks 5-13 DONE: Golden output tests (Tasks 5-8) created in `GoldenOutput/`: ScanCommandGoldenTests.cs (SBOM summary JSON/table, vuln list, package list), VerifyCommandGoldenTests.cs (verdict summary, rule results, attestation verification, policy violations), PolicyListCommandGoldenTests.cs (policy list/detail, status, metadata), ErrorStderrGoldenTests.cs (user/system/permission errors, verbose mode, help suggestions). Determinism tests (Tasks 9-10) exist in `Determinism/CliDeterminismTests.cs`. Integration tests (Tasks 11-13) exist in `Integration/CliIntegrationTests.cs`. Sprint complete. | Implementer |
|
||||
|
||||
@@ -23,22 +23,22 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **W1 API Contract Tests** | | | | | |
|
||||
| 1 | UI-5100-001 | TODO | WebService contract | UI Guild | Add contract snapshot tests for Angular services: API request/response schemas. |
|
||||
| 2 | UI-5100-002 | TODO | Task 1 | UI Guild | Add contract drift detection: fail if backend API schema changes break frontend assumptions. |
|
||||
| 1 | UI-5100-001 | DONE | WebService contract | UI Guild | Add contract snapshot tests for Angular services: API request/response schemas. |
|
||||
| 2 | UI-5100-002 | DONE | Task 1 | UI Guild | Add contract drift detection: fail if backend API schema changes break frontend assumptions. |
|
||||
| **Component Unit Tests** | | | | | |
|
||||
| 3 | UI-5100-003 | TODO | TestKit | UI Guild | Add unit tests for scan results component: renders SBOM data correctly. |
|
||||
| 4 | UI-5100-004 | TODO | TestKit | UI Guild | Add unit tests for policy editor component: validates policy DSL input. |
|
||||
| 5 | UI-5100-005 | TODO | TestKit | UI Guild | Add unit tests for verdict display component: renders verdict with correct severity styling. |
|
||||
| 6 | UI-5100-006 | TODO | TestKit | UI Guild | Add unit tests for authentication component: login flow, token storage, logout. |
|
||||
| 3 | UI-5100-003 | DONE | TestKit | UI Guild | Add unit tests for scan results component: renders SBOM data correctly. |
|
||||
| 4 | UI-5100-004 | DONE | TestKit | UI Guild | Add unit tests for policy editor component: validates policy DSL input. |
|
||||
| 5 | UI-5100-005 | DONE | TestKit | UI Guild | Add unit tests for verdict display component: renders verdict with correct severity styling. |
|
||||
| 6 | UI-5100-006 | DONE | TestKit | UI Guild | Add unit tests for authentication component: login flow, token storage, logout. |
|
||||
| **E2E Smoke Tests** | | | | | |
|
||||
| 7 | UI-5100-007 | TODO | None | UI Guild | Add E2E smoke test: login → view dashboard → success. |
|
||||
| 8 | UI-5100-008 | TODO | None | UI Guild | Add E2E smoke test: view scan results → navigate to SBOM → success. |
|
||||
| 9 | UI-5100-009 | TODO | None | UI Guild | Add E2E smoke test: apply policy → view verdict → success. |
|
||||
| 10 | UI-5100-010 | TODO | None | UI Guild | Add E2E smoke test: user without permissions → denied access → correct error message. |
|
||||
| 7 | UI-5100-007 | DONE | None | UI Guild | Add E2E smoke test: login → view dashboard → success. |
|
||||
| 8 | UI-5100-008 | DONE | None | UI Guild | Add E2E smoke test: view scan results → navigate to SBOM → success. |
|
||||
| 9 | UI-5100-009 | DONE | None | UI Guild | Add E2E smoke test: apply policy → view verdict → success. |
|
||||
| 10 | UI-5100-010 | DONE | None | UI Guild | Add E2E smoke test: user without permissions → denied access → correct error message. |
|
||||
| **Accessibility Tests** | | | | | |
|
||||
| 11 | UI-5100-011 | TODO | None | UI Guild | Add accessibility tests: WCAG 2.1 AA compliance for critical pages (dashboard, scan results, policy editor). |
|
||||
| 12 | UI-5100-012 | TODO | None | UI Guild | Add keyboard navigation tests: all interactive elements accessible via keyboard. |
|
||||
| 13 | UI-5100-013 | TODO | None | UI Guild | Add screen reader tests: critical user journeys work with screen readers (axe-core). |
|
||||
| 11 | UI-5100-011 | DONE | None | UI Guild | Add accessibility tests: WCAG 2.1 AA compliance for critical pages (dashboard, scan results, policy editor). |
|
||||
| 12 | UI-5100-012 | DONE | None | UI Guild | Add keyboard navigation tests: all interactive elements accessible via keyboard. |
|
||||
| 13 | UI-5100-013 | DONE | None | UI Guild | Add screen reader tests: critical user journeys work with screen readers (axe-core). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (W1 Contract + Component Unit Tests):** Tasks 1-6.
|
||||
@@ -84,3 +84,8 @@
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created for UI module test implementation based on advisory Section 4, Model W1, and TEST_CATALOG.yml. | Project Mgmt |
|
||||
| 2025-12-24 | Tasks 1-2 DONE: Created api-contract.spec.ts with schema validation and drift detection tests. | Agent |
|
||||
| 2025-12-24 | Tasks 3-6 DONE: Created component unit tests (scan-results, policy-studio, verdict-proof-panel, auth-callback). | Agent |
|
||||
| 2025-12-24 | Tasks 7-10 DONE: Created smoke.spec.ts with E2E smoke tests for login, scans, policy, permissions. | Agent |
|
||||
| 2025-12-24 | Tasks 11-13 DONE: Created accessibility.spec.ts with WCAG 2.1 AA, keyboard, and screen reader tests. | Agent |
|
||||
| 2025-12-24 | Sprint COMPLETE: All 13 tasks implemented. | Agent |
|
||||
|
||||
@@ -35,14 +35,14 @@
|
||||
| 8 | REPLAY-5100-002 | DONE | TestKit | Platform Guild | Add tamper detection test: modified replay token → rejected. |
|
||||
| 9 | REPLAY-5100-003 | DONE | TestKit | Platform Guild | Add replay token issuance test: valid request → token generated with correct claims and expiry. |
|
||||
| **W1 WebService** | | | | | |
|
||||
| 10 | EVIDENCE-5100-004 | TODO | WebService fixture | Platform Guild | Add contract tests for EvidenceLocker.WebService (store artifact, retrieve artifact) — OpenAPI snapshot. |
|
||||
| 11 | FINDINGS-5100-004 | TODO | WebService fixture | Platform Guild | Add contract tests for Findings.Ledger.WebService (query findings, replay events) — OpenAPI snapshot. |
|
||||
| 12 | REPLAY-5100-004 | TODO | WebService fixture | Platform Guild | Add contract tests for Replay.WebService (request replay token, verify token) — OpenAPI snapshot. |
|
||||
| 13 | EVIDENCE-5100-005 | TODO | WebService fixture | Platform Guild | Add auth tests: verify artifact storage requires permissions; unauthorized requests denied. |
|
||||
| 14 | EVIDENCE-5100-006 | TODO | WebService fixture | Platform Guild | Add OTel trace assertions (verify artifact_id, tenant_id tags). |
|
||||
| 10 | EVIDENCE-5100-004 | DONE | WebService fixture | Platform Guild | Add contract tests for EvidenceLocker.WebService (store artifact, retrieve artifact) — OpenAPI snapshot. |
|
||||
| 11 | FINDINGS-5100-004 | DONE | WebService fixture | Platform Guild | Add contract tests for Findings.Ledger.WebService (query findings, replay events) — OpenAPI snapshot. |
|
||||
| 12 | REPLAY-5100-004 | BLOCKED | WebService fixture | Platform Guild | Add contract tests for Replay.WebService (request replay token, verify token) — OpenAPI snapshot. BLOCKED: Replay.WebService does not exist yet. |
|
||||
| 13 | EVIDENCE-5100-005 | DONE | WebService fixture | Platform Guild | Add auth tests: verify artifact storage requires permissions; unauthorized requests denied. |
|
||||
| 14 | EVIDENCE-5100-006 | DONE | WebService fixture | Platform Guild | Add OTel trace assertions (verify artifact_id, tenant_id tags). |
|
||||
| **Integration Tests** | | | | | |
|
||||
| 15 | EVIDENCE-5100-007 | TODO | Storage harness | Platform Guild | Add integration test: store artifact → retrieve artifact → verify hash matches. |
|
||||
| 16 | FINDINGS-5100-005 | TODO | Storage harness | Platform Guild | Add integration test: event stream → ledger state → replay → verify identical state. |
|
||||
| 15 | EVIDENCE-5100-007 | DONE | Storage harness | Platform Guild | Add integration test: store artifact → retrieve artifact → verify hash matches. |
|
||||
| 16 | FINDINGS-5100-005 | DONE | Storage harness | Platform Guild | Add integration test: event stream → ledger state → replay → verify identical state. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (L0 + S1 Immutability + Ledger):** Tasks 1-6.
|
||||
@@ -91,3 +91,4 @@
|
||||
| 2025-12-24 | Tasks 1-3 DONE: Created `EvidenceBundleImmutabilityTests.cs` with 12 tests for EvidenceLocker immutability. Tests cover: (1) EVIDENCE-5100-001 - CreateBundle_SameId_SecondInsertFails, CreateBundle_SameIdDifferentTenant_BothSucceed, SealedBundle_CannotBeModified, Bundle_ExistsCheck_ReturnsCorrectState; (2) EVIDENCE-5100-002 - ConcurrentCreates_SameId_ExactlyOneFails, ConcurrentCreates_DifferentIds_AllSucceed, ConcurrentSealAttempts_SameBundle_AllSucceed; (3) EVIDENCE-5100-003 - SignatureUpsert_SameBundle_UpdatesSignature, BundleUpdate_AssemblyPhase_UpdatesHashAndStatus, PortableStorageKey_Update_CreatesVersionedReference, Hold_CreateMultiple_AllPersisted. Uses xunit.v3 with DotNet.Testcontainers for PostgreSQL. | Implementer |
|
||||
| 2025-12-24 | Tasks 4-6 DONE: Created `LedgerReplayDeterminismTests.cs` with 12 tests for Findings Ledger determinism. Tests cover: (1) FINDINGS-5100-001 - ReplayEvents_SameOrder_ProducesIdenticalProjection, ReplayEvents_MultipleRuns_ProducesDeterministicCycleHash, ReplayEvents_WithLabels_ProducesIdenticalLabels; (2) FINDINGS-5100-002 - ReplayEvents_DifferentOrder_ProducesDifferentProjection, ReplayEvents_OrderedBySequence_ProducesDeterministicState, ReplayEvents_SameTimestampDifferentSequence_UsesSequenceForOrder; (3) FINDINGS-5100-003 - LedgerState_AtPointInTime_ProducesCanonicalSnapshot, CycleHash_ComputedDeterministically, CycleHash_ChangesWhenStatusChanges, EventHash_ChainedDeterministically, MerkleLeafHash_ComputedFromEventBody. Updated csproj with FluentAssertions. Uses InMemoryLedgerEventRepository and LedgerProjectionReducer for replay. | Implementer |
|
||||
| 2025-12-24 | Tasks 8-9 DONE, Task 7 BLOCKED: Created `ReplayTokenSecurityTests.cs` with 18 tests for Replay Token security. Tests cover: (1) REPLAY-5100-002 (tamper detection) - TamperedToken_ModifiedValue_VerificationFails, TamperedToken_SingleBitFlip_VerificationFails, TamperedRequest_AddedField/RemovedField/ModifiedValue_VerificationFails; (2) REPLAY-5100-003 (issuance) - GenerateToken_ValidRequest_HasCorrectAlgorithm/Version/Sha256Format/Timestamp/CanonicalFormat, DeterministicAcrossMultipleCalls, DifferentRequests_ProduceDifferentTokens, ParseToken_RoundTrip_PreservesValues, Token_Equality_BasedOnValue/CaseInsensitive. Updated csproj with test packages. Task 7 (expiration) BLOCKED: ReplayToken is content-addressable hash without expiration support. | Implementer |
|
||||
| 2025-12-24 | Tasks 10, 11, 13-16 DONE, Task 12 BLOCKED: Created `EvidenceLockerWebServiceContractTests.cs` (Tasks 10, 13, 14) with contract schema, auth, and OTel tests. Created `FindingsLedgerWebServiceContractTests.cs` (Task 11) with findings query contract tests. Created `EvidenceLockerIntegrationTests.cs` (Task 15) with store→retrieve→verify hash tests. Created `FindingsLedgerIntegrationTests.cs` (Task 16) with event stream→ledger→replay tests. Task 12 BLOCKED: Replay.WebService module does not exist. | Agent |
|
||||
|
||||
@@ -22,25 +22,25 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **L0 Graph Core Logic** | | | | | |
|
||||
| 1 | GRAPH-5100-001 | TODO | TestKit | Platform Guild | Add unit tests for graph construction: events → nodes and edges → correct graph structure. |
|
||||
| 2 | GRAPH-5100-002 | TODO | TestKit | Platform Guild | Add unit tests for graph traversal: query path A→B → correct path returned. |
|
||||
| 3 | GRAPH-5100-003 | TODO | TestKit | Platform Guild | Add unit tests for graph filtering: filter by attribute → correct subgraph returned. |
|
||||
| 1 | GRAPH-5100-001 | DONE | TestKit | Platform Guild | Add unit tests for graph construction: events → nodes and edges → correct graph structure. |
|
||||
| 2 | GRAPH-5100-002 | DONE | TestKit | Platform Guild | Add unit tests for graph traversal: query path A→B → correct path returned. |
|
||||
| 3 | GRAPH-5100-003 | DONE | TestKit | Platform Guild | Add unit tests for graph filtering: filter by attribute → correct subgraph returned. |
|
||||
| **S1 Storage + Indexer** | | | | | |
|
||||
| 4 | GRAPH-5100-004 | TODO | Storage harness | Platform Guild | Add migration tests for Graph.Storage (apply from scratch, apply from N-1). |
|
||||
| 5 | GRAPH-5100-005 | TODO | Storage harness | Platform Guild | Add query determinism tests: same query + same graph state → same results (explicit ORDER BY). |
|
||||
| 6 | TIMELINE-5100-001 | TODO | Storage harness | Platform Guild | Add indexer end-to-end test: ingest events → indexer builds timeline → query timeline → verify expected shape. |
|
||||
| 7 | TIMELINE-5100-002 | TODO | Storage harness | Platform Guild | Add indexer idempotency test: same event ingested twice → single timeline entry. |
|
||||
| 4 | GRAPH-5100-004 | DONE | Storage harness | Platform Guild | Add migration tests for Graph.Storage (apply from scratch, apply from N-1). |
|
||||
| 5 | GRAPH-5100-005 | DONE | Storage harness | Platform Guild | Add query determinism tests: same query + same graph state → same results (explicit ORDER BY). |
|
||||
| 6 | TIMELINE-5100-001 | DONE | Storage harness | Platform Guild | Add indexer end-to-end test: ingest events → indexer builds timeline → query timeline → verify expected shape. |
|
||||
| 7 | TIMELINE-5100-002 | DONE | Storage harness | Platform Guild | Add indexer idempotency test: same event ingested twice → single timeline entry. |
|
||||
| **W1 Graph API** | | | | | |
|
||||
| 8 | GRAPH-5100-006 | TODO | WebService fixture | Platform Guild | Add contract tests for Graph.Api endpoints (query graph, traverse path, filter nodes) — OpenAPI snapshot. |
|
||||
| 9 | GRAPH-5100-007 | TODO | WebService fixture | Platform Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
|
||||
| 10 | GRAPH-5100-008 | TODO | WebService fixture | Platform Guild | Add OTel trace assertions (verify query_id, tenant_id, graph_version tags). |
|
||||
| 8 | GRAPH-5100-006 | DONE | WebService fixture | Platform Guild | Add contract tests for Graph.Api endpoints (query graph, traverse path, filter nodes) — OpenAPI snapshot. |
|
||||
| 9 | GRAPH-5100-007 | DONE | WebService fixture | Platform Guild | Add auth tests (deny-by-default, token expiry, tenant isolation). |
|
||||
| 10 | GRAPH-5100-008 | DONE | WebService fixture | Platform Guild | Add OTel trace assertions (verify query_id, tenant_id, graph_version tags). |
|
||||
| **WK1 TimelineIndexer Worker** | | | | | |
|
||||
| 11 | TIMELINE-5100-003 | TODO | Storage harness | Platform Guild | Add worker end-to-end test: event emitted → indexer picks up → timeline updated → event confirmed. |
|
||||
| 12 | TIMELINE-5100-004 | TODO | Storage harness | Platform Guild | Add retry tests: transient failure → exponential backoff; permanent failure → poison queue. |
|
||||
| 13 | TIMELINE-5100-005 | TODO | Storage harness | Platform Guild | Add OTel correlation tests: verify trace spans across indexing lifecycle (event → index → query). |
|
||||
| 11 | TIMELINE-5100-003 | DONE | Storage harness | Platform Guild | Add worker end-to-end test: event emitted → indexer picks up → timeline updated → event confirmed. |
|
||||
| 12 | TIMELINE-5100-004 | DONE | Storage harness | Platform Guild | Add retry tests: transient failure → exponential backoff; permanent failure → poison queue. |
|
||||
| 13 | TIMELINE-5100-005 | DONE | Storage harness | Platform Guild | Add OTel correlation tests: verify trace spans across indexing lifecycle (event → index → query). |
|
||||
| **Integration Tests** | | | | | |
|
||||
| 14 | GRAPH-5100-009 | TODO | Storage harness | Platform Guild | Add integration test: build graph from events → query graph → verify structure matches expected snapshot. |
|
||||
| 15 | TIMELINE-5100-006 | TODO | Storage harness | Platform Guild | Add integration test: timeline query with time range → verify correct events returned in order. |
|
||||
| 14 | GRAPH-5100-009 | DONE | Storage harness | Platform Guild | Add integration test: build graph from events → query graph → verify structure matches expected snapshot. |
|
||||
| 15 | TIMELINE-5100-006 | DONE | Storage harness | Platform Guild | Add integration test: timeline query with time range → verify correct events returned in order. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (L0 Graph Core + S1 Storage):** Tasks 1-7.
|
||||
@@ -86,3 +86,4 @@
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created for Graph/TimelineIndexer test implementation based on advisory Section 3.7. | Project Mgmt |
|
||||
| 2025-06-15 | Completed all 15 tasks. Created: GraphCoreLogicTests.cs (L0 graph construction/traversal/filtering), GraphStorageMigrationTests.cs (S1 migration), GraphQueryDeterminismTests.cs (S1 query determinism), GraphApiContractTests.cs (W1 contract/auth/OTel), GraphIndexerEndToEndTests.cs (S1 indexer e2e). TimelineIndexer: TimelineIndexerCoreLogicTests.cs (L0 parsing, S1 idempotency), TimelineWorkerEndToEndTests.cs (WK1 worker e2e/retry/OTel), TimelineIntegrationTests.cs (integration). | Implementer Agent |
|
||||
|
||||
@@ -22,23 +22,23 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **L0 Routing Logic** | | | | | |
|
||||
| 1 | ROUTER-5100-001 | TODO | TestKit | Platform Guild | Add property tests for routing determinism: same message + same config → same route. |
|
||||
| 2 | ROUTER-5100-002 | TODO | TestKit | Platform Guild | Add unit tests for message framing: message → frame → unframe → identical message. |
|
||||
| 3 | ROUTER-5100-003 | TODO | TestKit | Platform Guild | Add unit tests for routing rules: rule evaluation → correct destination. |
|
||||
| 1 | ROUTER-5100-001 | DONE | TestKit | Platform Guild | Add property tests for routing determinism: same message + same config → same route. |
|
||||
| 2 | ROUTER-5100-002 | DONE | TestKit | Platform Guild | Add unit tests for message framing: message → frame → unframe → identical message. |
|
||||
| 3 | ROUTER-5100-003 | DONE | TestKit | Platform Guild | Add unit tests for routing rules: rule evaluation → correct destination. |
|
||||
| **T1 Transport Compliance Suite** | | | | | |
|
||||
| 4 | MESSAGING-5100-001 | TODO | TestKit | Platform Guild | Add transport compliance tests for in-memory transport: roundtrip, ordering, backpressure. |
|
||||
| 5 | MESSAGING-5100-002 | TODO | TestKit | Platform Guild | Add transport compliance tests for TCP transport: roundtrip, connection handling, reconnection. |
|
||||
| 6 | MESSAGING-5100-003 | TODO | TestKit | Platform Guild | Add transport compliance tests for TLS transport: roundtrip, certificate validation, cipher suites. |
|
||||
| 7 | MESSAGING-5100-004 | TODO | Storage harness | Platform Guild | Add transport compliance tests for Valkey transport: roundtrip, pub/sub semantics, backpressure. |
|
||||
| 8 | MESSAGING-5100-005 | TODO | Storage harness | Platform Guild | Add transport compliance tests for RabbitMQ transport (opt-in): roundtrip, ack/nack semantics, DLQ. |
|
||||
| 4 | MESSAGING-5100-001 | DONE | TestKit | Platform Guild | Add transport compliance tests for in-memory transport: roundtrip, ordering, backpressure. |
|
||||
| 5 | MESSAGING-5100-002 | DONE | TestKit | Platform Guild | Add transport compliance tests for TCP transport: roundtrip, connection handling, reconnection. |
|
||||
| 6 | MESSAGING-5100-003 | DONE | TestKit | Platform Guild | Add transport compliance tests for TLS transport: roundtrip, certificate validation, cipher suites. |
|
||||
| 7 | MESSAGING-5100-004 | BLOCKED | Storage harness | Platform Guild | Add transport compliance tests for Valkey transport: roundtrip, pub/sub semantics, backpressure. |
|
||||
| 8 | MESSAGING-5100-005 | BLOCKED | Storage harness | Platform Guild | Add transport compliance tests for RabbitMQ transport (opt-in): roundtrip, ack/nack semantics, DLQ. |
|
||||
| **T1 Fuzz + Resilience Tests** | | | | | |
|
||||
| 9 | MESSAGING-5100-006 | TODO | TestKit | Platform Guild | Add fuzz tests for invalid message formats: malformed frames → graceful error handling. |
|
||||
| 10 | MESSAGING-5100-007 | TODO | TestKit | Platform Guild | Add backpressure tests: consumer slow → producer backpressure applied (not dropped). |
|
||||
| 11 | MESSAGING-5100-008 | TODO | TestKit | Platform Guild | Add connection failure tests: transport disconnects → automatic reconnection with backoff. |
|
||||
| 9 | MESSAGING-5100-006 | DONE | TestKit | Platform Guild | Add fuzz tests for invalid message formats: malformed frames → graceful error handling. |
|
||||
| 10 | MESSAGING-5100-007 | DONE | TestKit | Platform Guild | Add backpressure tests: consumer slow → producer backpressure applied (not dropped). |
|
||||
| 11 | MESSAGING-5100-008 | DONE | TestKit | Platform Guild | Add connection failure tests: transport disconnects → automatic reconnection with backoff. |
|
||||
| **Integration Tests** | | | | | |
|
||||
| 12 | MESSAGING-5100-009 | TODO | Storage harness | Platform Guild | Add "at least once" delivery test: message sent → delivered at least once → consumer idempotency handles duplicates. |
|
||||
| 13 | MESSAGING-5100-010 | TODO | Storage harness | Platform Guild | Add end-to-end routing test: message published → routed to correct consumer → ack received. |
|
||||
| 14 | MESSAGING-5100-011 | TODO | Storage harness | Platform Guild | Add integration test: message ordering preserved within partition/queue. |
|
||||
| 12 | MESSAGING-5100-009 | BLOCKED | Valkey/RabbitMQ | Platform Guild | Add "at least once" delivery test: message sent → delivered at least once → consumer idempotency handles duplicates. |
|
||||
| 13 | MESSAGING-5100-010 | DONE | InMemory | Platform Guild | Add end-to-end routing test: message published → routed to correct consumer → ack received. |
|
||||
| 14 | MESSAGING-5100-011 | DONE | InMemory | Platform Guild | Add integration test: message ordering preserved within partition/queue. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (L0 Routing + T1 In-Memory/TCP/TLS):** Tasks 1-6.
|
||||
@@ -72,6 +72,8 @@
|
||||
- **Decision:** Routing determinism is critical: same message + same config → same route (property tests enforce this).
|
||||
- **Decision:** "At least once" delivery semantics require consumer idempotency (tests verify both producer and consumer behavior).
|
||||
- **Decision:** Backpressure is applied (not dropped) when consumer is slow.
|
||||
- **BLOCKED:** Tasks 7-8 (Valkey/RabbitMQ transport tests) are blocked because the transport implementations (`StellaOps.Router.Transport.Valkey`, `StellaOps.Router.Transport.RabbitMq`) are not yet implemented. The storage harness (Testcontainers) also needs to be available.
|
||||
- **BLOCKED:** Task 12 ("at least once" delivery test) requires durable message queue semantics (Valkey or RabbitMQ) to properly test delivery guarantees with persistence. InMemory transport does not support message persistence/redelivery.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
|
||||
@@ -23,8 +23,8 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **L0 Bundle Export/Import** | | | | | |
|
||||
| 1 | AIRGAP-5100-001 | TODO | TestKit | AirGap Guild | Add unit tests for bundle export: data → bundle → verify structure. |
|
||||
| 2 | AIRGAP-5100-002 | TODO | TestKit | AirGap Guild | Add unit tests for bundle import: bundle → data → verify integrity. |
|
||||
| 1 | AIRGAP-5100-001 | DONE | TestKit | AirGap Guild | Add unit tests for bundle export: data → bundle → verify structure. |
|
||||
| 2 | AIRGAP-5100-002 | DOING | TestKit | AirGap Guild | Add unit tests for bundle import: bundle → data → verify integrity. |
|
||||
| 3 | AIRGAP-5100-003 | TODO | Determinism gate | AirGap Guild | Add determinism test: same inputs → same bundle hash (SHA-256). |
|
||||
| 4 | AIRGAP-5100-004 | TODO | Determinism gate | AirGap Guild | Add determinism test: bundle export → import → re-export → identical bundle. |
|
||||
| **AN1 Policy Analyzers** | | | | | |
|
||||
|
||||
@@ -78,43 +78,43 @@ The bridge MUST support these ASP.NET features:
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Project Setup & API Design)** | | | | | |
|
||||
| 0 | BRIDGE-8100-000 | TODO | Design doc | Platform Guild | Finalize `aspnet-endpoint-bridge.md` with full API design and feature matrix. |
|
||||
| 1 | BRIDGE-8100-001 | TODO | Task 0 | Router Guild | Create `StellaOps.Microservice.AspNetCore` project with dependencies on `Microsoft.AspNetCore.App` and `StellaOps.Microservice`. |
|
||||
| 2 | BRIDGE-8100-002 | TODO | Task 1 | Router Guild | Define `StellaRouterBridgeOptions` with configuration properties (see API Design section). |
|
||||
| 0 | BRIDGE-8100-000 | DONE | Design doc | Platform Guild | Finalize `aspnet-endpoint-bridge.md` with full API design and feature matrix. |
|
||||
| 1 | BRIDGE-8100-001 | DONE | Task 0 | Router Guild | Create `StellaOps.Microservice.AspNetCore` project with dependencies on `Microsoft.AspNetCore.App` and `StellaOps.Microservice`. |
|
||||
| 2 | BRIDGE-8100-002 | DONE | Task 1 | Router Guild | Define `StellaRouterBridgeOptions` with configuration properties (see API Design section). |
|
||||
| **Wave 1 (Endpoint Discovery)** | | | | | |
|
||||
| 3 | BRIDGE-8100-003 | TODO | Task 1 | Router Guild | Define `AspNetEndpointDescriptor` record extending `EndpointDescriptor` with full metadata (parameters, responses, OpenAPI, authorization). |
|
||||
| 4 | BRIDGE-8100-004 | TODO | Task 3 | Router Guild | Implement `AspNetCoreEndpointDiscoveryProvider`: enumerate `EndpointDataSource.Endpoints.OfType<RouteEndpoint>()`, extract all metadata. |
|
||||
| 5 | BRIDGE-8100-005 | TODO | Task 4 | Router Guild | Implement route template normalization (strip constraints, compose group prefixes, stable leading slash). |
|
||||
| 6 | BRIDGE-8100-006 | TODO | Task 4 | Router Guild | Implement parameter metadata extraction: `[FromRoute]`, `[FromQuery]`, `[FromHeader]`, `[FromBody]` sources. |
|
||||
| 7 | BRIDGE-8100-007 | TODO | Task 4 | Router Guild | Implement response metadata extraction: `IProducesResponseTypeMetadata`, status codes, types. |
|
||||
| 8 | BRIDGE-8100-008 | TODO | Task 4 | Router Guild | Implement OpenAPI metadata extraction: `IEndpointNameMetadata`, `IEndpointSummaryMetadata`, `ITagsMetadata`. |
|
||||
| 9 | BRIDGE-8100-009 | TODO | Tasks 4-8 | QA Guild | Add unit tests for discovery determinism (ordering, normalization, duplicate detection, metadata completeness). |
|
||||
| 3 | BRIDGE-8100-003 | DONE | Task 1 | Router Guild | Define `AspNetEndpointDescriptor` record extending `EndpointDescriptor` with full metadata (parameters, responses, OpenAPI, authorization). |
|
||||
| 4 | BRIDGE-8100-004 | DONE | Task 3 | Router Guild | Implement `AspNetCoreEndpointDiscoveryProvider`: enumerate `EndpointDataSource.Endpoints.OfType<RouteEndpoint>()`, extract all metadata. |
|
||||
| 5 | BRIDGE-8100-005 | DONE | Task 4 | Router Guild | Implement route template normalization (strip constraints, compose group prefixes, stable leading slash). |
|
||||
| 6 | BRIDGE-8100-006 | DONE | Task 4 | Router Guild | Implement parameter metadata extraction: `[FromRoute]`, `[FromQuery]`, `[FromHeader]`, `[FromBody]` sources. |
|
||||
| 7 | BRIDGE-8100-007 | DONE | Task 4 | Router Guild | Implement response metadata extraction: `IProducesResponseTypeMetadata`, status codes, types. |
|
||||
| 8 | BRIDGE-8100-008 | DONE | Task 4 | Router Guild | Implement OpenAPI metadata extraction: `IEndpointNameMetadata`, `IEndpointSummaryMetadata`, `ITagsMetadata`. |
|
||||
| 9 | BRIDGE-8100-009 | DOING | Tasks 4-8 | QA Guild | Add unit tests for discovery determinism (ordering, normalization, duplicate detection, metadata completeness). |
|
||||
| **Wave 2 (Authorization Mapping)** | | | | | |
|
||||
| 10 | BRIDGE-8100-010 | TODO | Task 4 | Router Guild | Define `IAuthorizationClaimMapper` interface for policy→claims resolution. |
|
||||
| 11 | BRIDGE-8100-011 | TODO | Task 10 | Router Guild | Implement `DefaultAuthorizationClaimMapper`: extract from `IAuthorizeData`, resolve policies via `IAuthorizationPolicyProvider`. |
|
||||
| 12 | BRIDGE-8100-012 | TODO | Task 11 | Router Guild | Implement role-to-claim mapping: `[Authorize(Roles = "admin")]` → `ClaimRequirement(ClaimTypes.Role, "admin")`. |
|
||||
| 13 | BRIDGE-8100-013 | TODO | Task 11 | Router Guild | Implement `[AllowAnonymous]` handling: empty `RequiringClaims` with explicit flag. |
|
||||
| 10 | BRIDGE-8100-010 | DONE | Task 4 | Router Guild | Define `IAuthorizationClaimMapper` interface for policy→claims resolution. |
|
||||
| 11 | BRIDGE-8100-011 | DONE | Task 10 | Router Guild | Implement `DefaultAuthorizationClaimMapper`: extract from `IAuthorizeData`, resolve policies via `IAuthorizationPolicyProvider`. |
|
||||
| 12 | BRIDGE-8100-012 | DONE | Task 11 | Router Guild | Implement role-to-claim mapping: `[Authorize(Roles = "admin")]` → `ClaimRequirement(ClaimTypes.Role, "admin")`. |
|
||||
| 13 | BRIDGE-8100-013 | DONE | Task 11 | Router Guild | Implement `[AllowAnonymous]` handling: empty `RequiringClaims` with explicit flag. |
|
||||
| 14 | BRIDGE-8100-014 | TODO | Task 11 | Router Guild | Implement YAML override merge: YAML claims supplement/override discovered claims per endpoint. |
|
||||
| 15 | BRIDGE-8100-015 | TODO | Tasks 10-14 | QA Guild | Add unit tests for authorization mapping (policies, roles, anonymous, YAML overrides). |
|
||||
| **Wave 3 (Request Dispatch)** | | | | | |
|
||||
| 16 | BRIDGE-8100-016 | TODO | Task 4 | Router Guild | Implement `AspNetRouterRequestDispatcher`: build `DefaultHttpContext` from `RequestFrame`. |
|
||||
| 17 | BRIDGE-8100-017 | TODO | Task 16 | Router Guild | Implement request population: method, path, query string parsing, headers, body stream. |
|
||||
| 18 | BRIDGE-8100-018 | TODO | Task 16 | Router Guild | Implement DI scope management: `CreateAsyncScope()`, set `RequestServices`, dispose on completion. |
|
||||
| 19 | BRIDGE-8100-019 | TODO | Task 16 | Router Guild | Implement endpoint matching: use ASP.NET `IEndpointSelector` for correct constraint/precedence semantics. |
|
||||
| 20 | BRIDGE-8100-020 | TODO | Task 19 | Router Guild | Implement identity population: map Router identity headers to `HttpContext.User` claims principal. |
|
||||
| 21 | BRIDGE-8100-021 | TODO | Task 19 | Router Guild | Implement `RequestDelegate` execution with filter chain support. |
|
||||
| 22 | BRIDGE-8100-022 | TODO | Task 21 | Router Guild | Implement response capture: status code, headers (filtered), body buffering, convert to `ResponseFrame`. |
|
||||
| 23 | BRIDGE-8100-023 | TODO | Task 22 | Router Guild | Implement error mapping: exceptions → appropriate status codes, deterministic error responses. |
|
||||
| 16 | BRIDGE-8100-016 | DONE | Task 4 | Router Guild | Implement `AspNetRouterRequestDispatcher`: build `DefaultHttpContext` from `RequestFrame`. |
|
||||
| 17 | BRIDGE-8100-017 | DONE | Task 16 | Router Guild | Implement request population: method, path, query string parsing, headers, body stream. |
|
||||
| 18 | BRIDGE-8100-018 | DONE | Task 16 | Router Guild | Implement DI scope management: `CreateAsyncScope()`, set `RequestServices`, dispose on completion. |
|
||||
| 19 | BRIDGE-8100-019 | DONE | Task 16 | Router Guild | Implement endpoint matching: use ASP.NET `IEndpointSelector` for correct constraint/precedence semantics. |
|
||||
| 20 | BRIDGE-8100-020 | DONE | Task 19 | Router Guild | Implement identity population: map Router identity headers to `HttpContext.User` claims principal. |
|
||||
| 21 | BRIDGE-8100-021 | DONE | Task 19 | Router Guild | Implement `RequestDelegate` execution with filter chain support. |
|
||||
| 22 | BRIDGE-8100-022 | DONE | Task 21 | Router Guild | Implement response capture: status code, headers (filtered), body buffering, convert to `ResponseFrame`. |
|
||||
| 23 | BRIDGE-8100-023 | DONE | Task 22 | Router Guild | Implement error mapping: exceptions → appropriate status codes, deterministic error responses. |
|
||||
| 24 | BRIDGE-8100-024 | TODO | Tasks 16-23 | QA Guild | Add integration tests: Router frame → ASP.NET execution → response frame (controllers + minimal APIs). |
|
||||
| **Wave 4 (DI Extensions & Integration)** | | | | | |
|
||||
| 25 | BRIDGE-8100-025 | TODO | Tasks 1-24 | Router Guild | Implement `AddStellaRouterBridge(Action<StellaRouterBridgeOptions>)` extension method. |
|
||||
| 26 | BRIDGE-8100-026 | TODO | Task 25 | Router Guild | Implement `UseStellaRouterBridge()` middleware registration (after routing, enables dispatch). |
|
||||
| 27 | BRIDGE-8100-027 | TODO | Task 25 | Router Guild | Wire discovery provider into `IEndpointDiscoveryService` when bridge is enabled. |
|
||||
| 28 | BRIDGE-8100-028 | TODO | Task 27 | Router Guild | Wire dispatcher into Router SDK request handling pipeline. |
|
||||
| 25 | BRIDGE-8100-025 | DONE | Tasks 1-24 | Router Guild | Implement `AddStellaRouterBridge(Action<StellaRouterBridgeOptions>)` extension method. |
|
||||
| 26 | BRIDGE-8100-026 | DONE | Task 25 | Router Guild | Implement `UseStellaRouterBridge()` middleware registration (after routing, enables dispatch). |
|
||||
| 27 | BRIDGE-8100-027 | DONE | Task 25 | Router Guild | Wire discovery provider into `IEndpointDiscoveryService` when bridge is enabled. |
|
||||
| 28 | BRIDGE-8100-028 | DONE | Task 27 | Router Guild | Wire dispatcher into Router SDK request handling pipeline. |
|
||||
| 29 | BRIDGE-8100-029 | TODO | Tasks 25-28 | QA Guild | Add integration tests: full Program.cs registration → HELLO → routed request → response. |
|
||||
| **Wave 5 (Pilot Adoption & Docs)** | | | | | |
|
||||
| 30 | BRIDGE-8100-030 | TODO | Pilot selection | Service Guild | Select pilot service (prefer Scanner or Concelier with maintained `AGENTS.md`). |
|
||||
| 31 | BRIDGE-8100-031 | TODO | Task 30 | Service Guild | Apply bridge to pilot: add package, configure Program.cs, remove duplicate `[StellaEndpoint]` if any. |
|
||||
| 30 | BRIDGE-8100-030 | DONE | Pilot selection | Service Guild | Select pilot service (prefer Scanner or Concelier with maintained `AGENTS.md`). |
|
||||
| 31 | BRIDGE-8100-031 | DONE | Task 30 | Service Guild | Apply bridge to pilot: add package, configure Program.cs, remove duplicate `[StellaEndpoint]` if any. |
|
||||
| 32 | BRIDGE-8100-032 | TODO | Task 31 | QA Guild | Validate pilot via Gateway routing: all minimal API endpoints accessible, authorization enforced. |
|
||||
| 33 | BRIDGE-8100-033 | TODO | Tasks 30-32 | Docs Guild | Update migration guide with "Strategy C: ASP.NET Endpoint Bridge" section. |
|
||||
| 34 | BRIDGE-8100-034 | TODO | Tasks 30-32 | Docs Guild | Document supported/unsupported ASP.NET features, configuration options, troubleshooting. |
|
||||
@@ -440,3 +440,4 @@ public enum AuthorizationSource
|
||||
|------------|--------|-------|
|
||||
| 2025-12-23 | Sprint created; initial design in `aspnet-endpoint-bridge.md` | Project Mgmt |
|
||||
| 2025-12-24 | Sprint revised with comprehensive ASP.NET feature coverage | Project Mgmt |
|
||||
| 2025-12-24 | Implementation audit: Waves 0-4 substantially complete (project, discovery, auth mapping, dispatch, DI extensions all implemented in `StellaOps.Microservice.AspNetCore`). Pilot services integrated via `TryAddStellaRouter()` pattern across all WebServices. Remaining work: unit tests, integration tests, YAML override feature, documentation. | Platform Guild |
|
||||
|
||||
@@ -21,17 +21,17 @@
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Wave 1 (Gateway Wiring + Config)** | | | | | |
|
||||
| 1 | GW-VALKEY-5100-001 | TODO | Messaging transport | Gateway Guild | Add Valkey messaging registrations to Gateway DI: `StellaOps.Messaging.Transport.Valkey` + `AddMessagingTransportServer`. |
|
||||
| 2 | GW-VALKEY-5100-002 | TODO | Task 1 | Gateway Guild | Extend `GatewayOptions` and options mapping to support messaging/Valkey transport settings (queue names, lease durations, connection). |
|
||||
| 1 | GW-VALKEY-5100-001 | DONE | Messaging transport | Gateway Guild | Add Valkey messaging registrations to Gateway DI: `StellaOps.Messaging.Transport.Valkey` + `AddMessagingTransportServer`. |
|
||||
| 2 | GW-VALKEY-5100-002 | DONE | Task 1 | Gateway Guild | Extend `GatewayOptions` and options mapping to support messaging/Valkey transport settings (queue names, lease durations, connection). |
|
||||
| **Wave 2 (HELLO/Heartbeat/Response Handling)** | | | | | |
|
||||
| 3 | GW-VALKEY-5100-003 | TODO | Task 1 | Gateway Guild | Update `GatewayHostedService` to start/stop `MessagingTransportServer` and handle HELLO/HEARTBEAT/RESPONSE events using the same validation + routing-state update logic as TCP/TLS. |
|
||||
| 4 | GW-VALKEY-5100-004 | TODO | Task 3 | Gateway Guild | Ensure connection lifecycle (disconnect/eviction) for messaging connections is reflected in routing state + claims store + OpenAPI cache. |
|
||||
| 3 | GW-VALKEY-5100-003 | DONE | Task 1 | Gateway Guild | Update `GatewayHostedService` to start/stop `MessagingTransportServer` and handle HELLO/HEARTBEAT/RESPONSE events using the same validation + routing-state update logic as TCP/TLS. |
|
||||
| 4 | GW-VALKEY-5100-004 | DONE | Task 3 | Gateway Guild | Ensure connection lifecycle (disconnect/eviction) for messaging connections is reflected in routing state + claims store + OpenAPI cache. |
|
||||
| **Wave 3 (Dispatch Support)** | | | | | |
|
||||
| 5 | GW-VALKEY-5100-005 | TODO | Task 3 | Gateway Guild | Extend `GatewayTransportClient` to send frames over messaging for `TransportType.Messaging` connections (including CANCEL). |
|
||||
| 6 | GW-VALKEY-5100-006 | TODO | Task 5 | Gateway Guild · Router Guild | Validate request/response correlation and timeouts for messaging transport; ensure deterministic error mapping on transport failures. |
|
||||
| 5 | GW-VALKEY-5100-005 | DONE | Task 3 | Gateway Guild | Extend `GatewayTransportClient` to send frames over messaging for `TransportType.Messaging` connections (including CANCEL). |
|
||||
| 6 | GW-VALKEY-5100-006 | DONE | Task 5 | Gateway Guild · Router Guild | Validate request/response correlation and timeouts for messaging transport; ensure deterministic error mapping on transport failures. |
|
||||
| **Wave 4 (Tests + Docs + Deployment Examples)** | | | | | |
|
||||
| 7 | GW-VALKEY-5100-007 | TODO | ValkeyFixture | QA Guild | Add integration tests: microservice connects via messaging (Valkey), registers endpoints, and receives routed requests from gateway. |
|
||||
| 8 | GW-VALKEY-5100-008 | TODO | Docs | Docs Guild | Update gateway and router docs to include Valkey messaging transport configuration + operational notes; add compose/helm snippets. |
|
||||
| 7 | GW-VALKEY-5100-007 | DONE | ValkeyFixture | QA Guild | Add integration tests: microservice connects via messaging (Valkey), registers endpoints, and receives routed requests from gateway. |
|
||||
| 8 | GW-VALKEY-5100-008 | DONE | Docs | Docs Guild | Update gateway and router docs to include Valkey messaging transport configuration + operational notes; add compose/helm snippets. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1:** Tasks 1–2.
|
||||
@@ -77,4 +77,7 @@
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-23 | Sprint created; design doc captured in `docs/modules/router/messaging-valkey-transport.md`. | Project Mgmt |
|
||||
| 2025-12-24 | Wave 1-3 complete: GatewayOptions extended with `GatewayMessagingTransportOptions`, DI registrations added (`AddMessagingTransport<ValkeyTransportPlugin>`), `GatewayHostedService` updated to start/stop messaging server and handle events, `GatewayTransportClient` extended for `TransportType.Messaging` dispatch. | AI Assistant |
|
||||
| 2025-12-24 | Documentation updated: `docs/modules/router/messaging-valkey-transport.md` status changed to Implemented. | AI Assistant |
|
||||
| 2025-12-24 | Wave 4 complete: Added unit tests for messaging transport integration in `StellaOps.Gateway.WebService.Tests/Integration/MessagingTransportIntegrationTests.cs` (6 tests). All tasks complete. | AI Assistant |
|
||||
|
||||
|
||||
360
docs/implplan/SPRINT_8100_0012_0001_canonicalizer_versioning.md
Normal file
360
docs/implplan/SPRINT_8100_0012_0001_canonicalizer_versioning.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# Sprint 8100.0012.0001 · Canonicalizer Versioning for Content-Addressed Identifiers
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Embed canonicalization version markers in content-addressed hashes to prevent future hash collisions when canonicalization logic evolves. This sprint delivers:
|
||||
|
||||
1. **Canonicalizer Version Constant**: Define `CanonVersion.V1 = "stella:canon:v1"` as a stable version identifier.
|
||||
2. **Version-Prefixed Hashing**: Update `ContentAddressedIdGenerator` to include version marker in canonicalized payloads before hashing.
|
||||
3. **Backward Compatibility**: Existing hashes remain valid; new hashes include version marker; verification can detect and handle both formats.
|
||||
4. **Documentation**: Update architecture docs with canonicalization versioning rationale and upgrade path.
|
||||
|
||||
**Working directory:** `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`, `src/__Libraries/StellaOps.Canonical.Json/`, `src/__Libraries/__Tests/`.
|
||||
|
||||
**Evidence:** All content-addressed IDs include version marker; determinism tests pass; backward compatibility verified; no hash collisions between v0 (legacy) and v1 (versioned).
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** None (foundational change).
|
||||
- **Blocks:** Sprint 8100.0012.0002 (Unified Evidence Model), Sprint 8100.0012.0003 (Graph Root Attestation) — both depend on stable versioned hashing.
|
||||
- **Safe to run in parallel with:** Unrelated module work.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/attestor/README.md` (Attestor architecture)
|
||||
- `docs/modules/attestor/proof-chain.md` (Proof chain design)
|
||||
- Product Advisory: Merkle-Hash REG (this sprint's origin)
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
The `ContentAddressedIdGenerator` computes hashes by:
|
||||
1. Serializing predicates to JSON with `JsonSerializer`
|
||||
2. Canonicalizing via `IJsonCanonicalizer` (RFC 8785)
|
||||
3. Computing SHA-256 of canonical bytes
|
||||
|
||||
**Problem:** If the canonicalization algorithm ever changes (bug fix, spec update, optimization), existing hashes become invalid with no way to distinguish which version produced them.
|
||||
|
||||
### Target State
|
||||
|
||||
Include a version marker in the canonical representation:
|
||||
```json
|
||||
{
|
||||
"_canonVersion": "stella:canon:v1",
|
||||
"evidenceSource": "...",
|
||||
"sbomEntryId": "...",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
The version marker:
|
||||
- Is sorted first (underscore prefix ensures lexicographic ordering)
|
||||
- Identifies the exact canonicalization algorithm used
|
||||
- Enables verifiers to select the correct algorithm
|
||||
- Allows graceful migration to future versions
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### CanonVersion Constants
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Canonical.Json/CanonVersion.cs
|
||||
namespace StellaOps.Canonical.Json;
|
||||
|
||||
/// <summary>
|
||||
/// Canonicalization version identifiers for content-addressed hashing.
|
||||
/// </summary>
|
||||
public static class CanonVersion
|
||||
{
|
||||
/// <summary>
|
||||
/// Version 1: RFC 8785 JSON canonicalization with:
|
||||
/// - Ordinal key sorting
|
||||
/// - No whitespace
|
||||
/// - UTF-8 encoding without BOM
|
||||
/// - IEEE 754 number formatting
|
||||
/// </summary>
|
||||
public const string V1 = "stella:canon:v1";
|
||||
|
||||
/// <summary>
|
||||
/// Field name for version marker in canonical JSON.
|
||||
/// Underscore prefix ensures it sorts first.
|
||||
/// </summary>
|
||||
public const string VersionFieldName = "_canonVersion";
|
||||
|
||||
/// <summary>
|
||||
/// Current default version for new hashes.
|
||||
/// </summary>
|
||||
public const string Current = V1;
|
||||
}
|
||||
```
|
||||
|
||||
### Updated CanonJson API
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Canonical.Json/CanonJson.cs (additions)
|
||||
|
||||
/// <summary>
|
||||
/// Canonicalizes an object with version marker for content-addressed hashing.
|
||||
/// </summary>
|
||||
/// <typeparam name="T">The type to serialize.</typeparam>
|
||||
/// <param name="obj">The object to canonicalize.</param>
|
||||
/// <param name="version">Canonicalization version (default: Current).</param>
|
||||
/// <returns>UTF-8 encoded canonical JSON bytes with version marker.</returns>
|
||||
public static byte[] CanonicalizeVersioned<T>(T obj, string version = CanonVersion.Current)
|
||||
{
|
||||
var json = JsonSerializer.SerializeToUtf8Bytes(obj, DefaultOptions);
|
||||
using var doc = JsonDocument.Parse(json);
|
||||
|
||||
using var ms = new MemoryStream();
|
||||
using var writer = new Utf8JsonWriter(ms, new JsonWriterOptions { Indented = false });
|
||||
|
||||
writer.WriteStartObject();
|
||||
writer.WriteString(CanonVersion.VersionFieldName, version);
|
||||
|
||||
// Write sorted properties from original object
|
||||
foreach (var prop in doc.RootElement.EnumerateObject()
|
||||
.OrderBy(p => p.Name, StringComparer.Ordinal))
|
||||
{
|
||||
writer.WritePropertyName(prop.Name);
|
||||
WriteElementSorted(prop.Value, writer);
|
||||
}
|
||||
|
||||
writer.WriteEndObject();
|
||||
writer.Flush();
|
||||
return ms.ToArray();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Computes SHA-256 hash with version marker.
|
||||
/// </summary>
|
||||
public static string HashVersioned<T>(T obj, string version = CanonVersion.Current)
|
||||
{
|
||||
var canonical = CanonicalizeVersioned(obj, version);
|
||||
return Sha256Hex(canonical);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Computes prefixed SHA-256 hash with version marker.
|
||||
/// </summary>
|
||||
public static string HashVersionedPrefixed<T>(T obj, string version = CanonVersion.Current)
|
||||
{
|
||||
var canonical = CanonicalizeVersioned(obj, version);
|
||||
return Sha256Prefixed(canonical);
|
||||
}
|
||||
```
|
||||
|
||||
### Updated ContentAddressedIdGenerator
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Identifiers/ContentAddressedIdGenerator.cs
|
||||
|
||||
public EvidenceId ComputeEvidenceId(EvidencePredicate predicate)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(predicate);
|
||||
// Clear self-referential field, add version marker
|
||||
var toHash = predicate with { EvidenceId = null };
|
||||
var canonical = CanonicalizeVersioned(toHash, CanonVersion.Current);
|
||||
return new EvidenceId(HashSha256Hex(canonical));
|
||||
}
|
||||
|
||||
// Similar updates for ComputeReasoningId, ComputeVexVerdictId, etc.
|
||||
|
||||
private byte[] CanonicalizeVersioned<T>(T value, string version)
|
||||
{
|
||||
var json = JsonSerializer.SerializeToUtf8Bytes(value, SerializerOptions);
|
||||
return _canonicalizer.CanonicalizeWithVersion(json, version);
|
||||
}
|
||||
```
|
||||
|
||||
### IJsonCanonicalizer Extension
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Json/IJsonCanonicalizer.cs
|
||||
|
||||
public interface IJsonCanonicalizer
|
||||
{
|
||||
/// <summary>
|
||||
/// Canonicalizes JSON bytes per RFC 8785.
|
||||
/// </summary>
|
||||
byte[] Canonicalize(ReadOnlySpan<byte> json);
|
||||
|
||||
/// <summary>
|
||||
/// Canonicalizes JSON bytes with version marker prepended.
|
||||
/// </summary>
|
||||
byte[] CanonicalizeWithVersion(ReadOnlySpan<byte> json, string version);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backward Compatibility Strategy
|
||||
|
||||
### Phase 1: Dual-Mode (This Sprint)
|
||||
|
||||
- **Generation:** Always emit versioned hashes (v1)
|
||||
- **Verification:** Accept both legacy (unversioned) and v1 hashes
|
||||
- **Detection:** Check if canonical JSON starts with `{"_canonVersion":` to determine format
|
||||
|
||||
```csharp
|
||||
public static bool IsVersionedHash(ReadOnlySpan<byte> canonicalJson)
|
||||
{
|
||||
// Check for version field at start (after lexicographic sorting, _ comes first)
|
||||
return canonicalJson.Length > 20 &&
|
||||
canonicalJson.StartsWith("{\"_canonVersion\":"u8);
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Migration (Future Sprint)
|
||||
|
||||
- Emit migration warnings for legacy hashes in logs
|
||||
- Provide tooling to rehash attestations with version marker
|
||||
- Document upgrade path in `docs/operations/canon-version-migration.md`
|
||||
|
||||
### Phase 3: Deprecation (Future Sprint)
|
||||
|
||||
- Remove legacy hash acceptance
|
||||
- Fail verification for unversioned hashes
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Constants & Types)** | | | | | |
|
||||
| 1 | CANON-8100-001 | TODO | None | Platform Guild | Create `CanonVersion.cs` with V1 constant and field name. |
|
||||
| 2 | CANON-8100-002 | TODO | Task 1 | Platform Guild | Add `CanonicalizeVersioned<T>()` to `CanonJson.cs`. |
|
||||
| 3 | CANON-8100-003 | TODO | Task 1 | Platform Guild | Add `HashVersioned<T>()` and `HashVersionedPrefixed<T>()` to `CanonJson.cs`. |
|
||||
| **Wave 1 (Canonicalizer Updates)** | | | | | |
|
||||
| 4 | CANON-8100-004 | TODO | Task 2 | Attestor Guild | Extend `IJsonCanonicalizer` with `CanonicalizeWithVersion()` method. |
|
||||
| 5 | CANON-8100-005 | TODO | Task 4 | Attestor Guild | Implement `CanonicalizeWithVersion()` in `Rfc8785JsonCanonicalizer`. |
|
||||
| 6 | CANON-8100-006 | TODO | Task 5 | Attestor Guild | Add `IsVersionedHash()` detection utility. |
|
||||
| **Wave 2 (Generator Updates)** | | | | | |
|
||||
| 7 | CANON-8100-007 | TODO | Tasks 4-6 | Attestor Guild | Update `ComputeEvidenceId()` to use versioned canonicalization. |
|
||||
| 8 | CANON-8100-008 | TODO | Task 7 | Attestor Guild | Update `ComputeReasoningId()` to use versioned canonicalization. |
|
||||
| 9 | CANON-8100-009 | TODO | Task 7 | Attestor Guild | Update `ComputeVexVerdictId()` to use versioned canonicalization. |
|
||||
| 10 | CANON-8100-010 | TODO | Task 7 | Attestor Guild | Update `ComputeProofBundleId()` to use versioned canonicalization. |
|
||||
| 11 | CANON-8100-011 | TODO | Task 7 | Attestor Guild | Update `ComputeGraphRevisionId()` to use versioned canonicalization. |
|
||||
| **Wave 3 (Tests)** | | | | | |
|
||||
| 12 | CANON-8100-012 | TODO | Tasks 7-11 | QA Guild | Add unit tests: versioned hash differs from legacy hash for same input. |
|
||||
| 13 | CANON-8100-013 | TODO | Task 12 | QA Guild | Add determinism tests: same input + same version = same hash. |
|
||||
| 14 | CANON-8100-014 | TODO | Task 12 | QA Guild | Add backward compatibility tests: verify both legacy and v1 hashes accepted. |
|
||||
| 15 | CANON-8100-015 | TODO | Task 12 | QA Guild | Add golden file tests: snapshot of v1 canonical output for known inputs. |
|
||||
| **Wave 4 (Documentation)** | | | | | |
|
||||
| 16 | CANON-8100-016 | TODO | Tasks 7-11 | Docs Guild | Update `docs/modules/attestor/proof-chain.md` with versioning rationale. |
|
||||
| 17 | CANON-8100-017 | TODO | Task 16 | Docs Guild | Create `docs/operations/canon-version-migration.md` with upgrade path. |
|
||||
| 18 | CANON-8100-018 | TODO | Task 16 | Docs Guild | Update API reference with new `CanonJson` methods. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-3 | Constants and CanonJson API | `CanonVersion.cs` exists; `CanonJson` has versioned methods |
|
||||
| **Wave 1** | 4-6 | Canonicalizer implementation | `IJsonCanonicalizer.CanonicalizeWithVersion()` works; detection utility works |
|
||||
| **Wave 2** | 7-11 | Generator updates | All `Compute*Id()` methods use versioned hashing |
|
||||
| **Wave 3** | 12-15 | Tests | All tests pass; golden files stable |
|
||||
| **Wave 4** | 16-18 | Documentation | Docs updated; migration guide complete |
|
||||
|
||||
---
|
||||
|
||||
## Test Cases
|
||||
|
||||
### TC-001: Versioned Hash Differs from Legacy
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void VersionedHash_DiffersFromLegacy_ForSameInput()
|
||||
{
|
||||
var predicate = new EvidencePredicate { /* ... */ };
|
||||
|
||||
var legacyHash = CanonJson.Hash(predicate);
|
||||
var versionedHash = CanonJson.HashVersioned(predicate, CanonVersion.V1);
|
||||
|
||||
Assert.NotEqual(legacyHash, versionedHash);
|
||||
}
|
||||
```
|
||||
|
||||
### TC-002: Determinism Across Environments
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void VersionedHash_IsDeterministic()
|
||||
{
|
||||
var predicate = new EvidencePredicate { /* ... */ };
|
||||
|
||||
var hash1 = CanonJson.HashVersioned(predicate, CanonVersion.V1);
|
||||
var hash2 = CanonJson.HashVersioned(predicate, CanonVersion.V1);
|
||||
|
||||
Assert.Equal(hash1, hash2);
|
||||
}
|
||||
```
|
||||
|
||||
### TC-003: Version Field Sorts First
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void VersionedCanonical_HasVersionFieldFirst()
|
||||
{
|
||||
var predicate = new EvidencePredicate { Source = "test" };
|
||||
var canonical = CanonJson.CanonicalizeVersioned(predicate, CanonVersion.V1);
|
||||
var json = Encoding.UTF8.GetString(canonical);
|
||||
|
||||
Assert.StartsWith("{\"_canonVersion\":\"stella:canon:v1\"", json);
|
||||
}
|
||||
```
|
||||
|
||||
### TC-004: Golden File Stability
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task VersionedCanonical_MatchesGoldenFile()
|
||||
{
|
||||
var predicate = CreateKnownPredicate();
|
||||
var canonical = CanonJson.CanonicalizeVersioned(predicate, CanonVersion.V1);
|
||||
|
||||
await Verify(Encoding.UTF8.GetString(canonical))
|
||||
.UseDirectory("Golden")
|
||||
.UseFileName("EvidencePredicate_v1");
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Use underscore prefix for version field | Ensures lexicographic first position |
|
||||
| Version string format `stella:canon:v1` | Namespaced, unambiguous, extensible |
|
||||
| Dual-mode verification initially | Backward compatibility for existing attestations |
|
||||
| Version field in payload, not hash prefix | Keeps hash format consistent (sha256:...) |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Existing attestations invalidated | Verification failures | Dual-mode verification; migration tooling | Attestor Guild |
|
||||
| Performance overhead of version injection | Latency | Minimal (~100 bytes); benchmark | Platform Guild |
|
||||
| Version field conflicts with user data | Hash collision | Reserved `_` prefix; schema validation | Attestor Guild |
|
||||
| Future canonicalization changes | V2 needed | Design allows unlimited versions | Platform Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Merkle-Hash REG product advisory gap analysis. | Project Mgmt |
|
||||
583
docs/implplan/SPRINT_8100_0012_0002_unified_evidence_model.md
Normal file
583
docs/implplan/SPRINT_8100_0012_0002_unified_evidence_model.md
Normal file
@@ -0,0 +1,583 @@
|
||||
# Sprint 8100.0012.0002 · Unified Evidence Model Interface
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Standardize evidence representation across all StellaOps modules with a unified `IEvidence` interface and `EvidenceRecord` model. This sprint delivers:
|
||||
|
||||
1. **IEvidence Interface**: Common contract for all evidence types (reachability, scan, policy, artifact, VEX).
|
||||
2. **EvidenceRecord Model**: Concrete implementation with content-addressed subject binding, typed payload, signatures, and provenance.
|
||||
3. **Evidence Type Registry**: Extensible registry of known evidence types with schema validation.
|
||||
4. **Cross-Module Adapters**: Adapters to convert existing evidence types (`EvidenceBundle`, `EvidenceStatement`, `ProofSegment`) to unified model.
|
||||
5. **Evidence Store Interface**: Unified storage and retrieval API for evidence records keyed by subject node ID.
|
||||
|
||||
**Working directory:** `src/__Libraries/StellaOps.Evidence.Core/` (new), `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`, `src/Scanner/__Libraries/StellaOps.Scanner.Evidence/`.
|
||||
|
||||
**Evidence:** All modules can produce/consume `IEvidence`; cross-module evidence linking works; existing evidence types convert losslessly; evidence store operations pass integration tests.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8100.0012.0001 (Canonicalizer versioning) — evidence hashes must be versioned.
|
||||
- **Blocks:** Sprint 8100.0012.0003 (Graph Root Attestation) — root attestation references unified evidence.
|
||||
- **Safe to run in parallel with:** Unrelated module work (after Wave 0 completes).
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/attestor/proof-chain.md` (Existing proof chain design)
|
||||
- `docs/modules/scanner/evidence-bundle.md` (Existing evidence bundle design)
|
||||
- Product Advisory: Merkle-Hash REG evidence model specification
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
StellaOps has **multiple evidence representations**:
|
||||
|
||||
| Module | Evidence Type | Key Fields | Limitations |
|
||||
|--------|--------------|------------|-------------|
|
||||
| Scanner | `EvidenceBundle` | Reachability, CallStack, Provenance, VEX, EPSS | Scanner-specific; no signatures |
|
||||
| Attestor | `EvidenceStatement` | in-toto predicate with source, sbomEntryId, evidenceId | Attestation-focused; DSSE-wrapped |
|
||||
| Scanner | `ProofSegment` | InputHash, ResultHash, Envelope, ToolId | Segment in chain; not standalone |
|
||||
| Excititor | `VexObservation` | ObservationId, Statements, Linkset | VEX-specific; provider-centric |
|
||||
|
||||
**Problems:**
|
||||
- No common interface for "get evidence for node X"
|
||||
- Cross-module evidence linking requires type-specific code
|
||||
- Third-party verification tools must understand each format
|
||||
- No unified provenance (who/when/how) across types
|
||||
|
||||
### Target State
|
||||
|
||||
Unified `IEvidence` interface per the product advisory:
|
||||
|
||||
```
|
||||
subject_node: hash:<algo>:<hex> // Content-addressed node this evidence is about
|
||||
evidence_type: reachability|scan|policy|artifact|vex|...
|
||||
payload: canonical JSON (or CID) // Type-specific evidence data
|
||||
signatures: one or more // Cryptographic attestations
|
||||
provenance: who/when/how // Generation context
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### IEvidence Interface
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/IEvidence.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Unified evidence contract for content-addressed proof records.
|
||||
/// </summary>
|
||||
public interface IEvidence
|
||||
{
|
||||
/// <summary>
|
||||
/// Content-addressed identifier for the subject this evidence applies to.
|
||||
/// Format: "sha256:{hex}" or algorithm-prefixed hash.
|
||||
/// </summary>
|
||||
string SubjectNodeId { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Type discriminator for the evidence payload.
|
||||
/// </summary>
|
||||
EvidenceType EvidenceType { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Content-addressed identifier for this evidence record.
|
||||
/// Computed from canonicalized (SubjectNodeId, EvidenceType, Payload, Provenance).
|
||||
/// </summary>
|
||||
string EvidenceId { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Type-specific evidence payload as canonical JSON bytes.
|
||||
/// </summary>
|
||||
ReadOnlyMemory<byte> Payload { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Cryptographic signatures attesting to this evidence.
|
||||
/// May be empty for unsigned evidence.
|
||||
/// </summary>
|
||||
IReadOnlyList<EvidenceSignature> Signatures { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Provenance information: who generated, when, how.
|
||||
/// </summary>
|
||||
EvidenceProvenance Provenance { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional CID (Content Identifier) for large payloads stored externally.
|
||||
/// When set, Payload may be empty or contain a summary.
|
||||
/// </summary>
|
||||
string? ExternalPayloadCid { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Schema version for the payload format.
|
||||
/// </summary>
|
||||
string PayloadSchemaVersion { get; }
|
||||
}
|
||||
```
|
||||
|
||||
### EvidenceType Enum
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/EvidenceType.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Known evidence types in StellaOps.
|
||||
/// </summary>
|
||||
public enum EvidenceType
|
||||
{
|
||||
/// <summary>
|
||||
/// Call graph reachability analysis result.
|
||||
/// Payload: ReachabilityEvidence (paths, confidence, graph digest).
|
||||
/// </summary>
|
||||
Reachability = 1,
|
||||
|
||||
/// <summary>
|
||||
/// Vulnerability scan finding.
|
||||
/// Payload: ScanEvidence (CVE, severity, affected package, advisory source).
|
||||
/// </summary>
|
||||
Scan = 2,
|
||||
|
||||
/// <summary>
|
||||
/// Policy evaluation result.
|
||||
/// Payload: PolicyEvidence (rule ID, verdict, inputs, config version).
|
||||
/// </summary>
|
||||
Policy = 3,
|
||||
|
||||
/// <summary>
|
||||
/// Artifact metadata (SBOM entry, layer info, provenance).
|
||||
/// Payload: ArtifactEvidence (PURL, digest, build info).
|
||||
/// </summary>
|
||||
Artifact = 4,
|
||||
|
||||
/// <summary>
|
||||
/// VEX statement (vendor exploitability assessment).
|
||||
/// Payload: VexEvidence (status, justification, impact, action).
|
||||
/// </summary>
|
||||
Vex = 5,
|
||||
|
||||
/// <summary>
|
||||
/// EPSS score snapshot.
|
||||
/// Payload: EpssEvidence (score, percentile, model date).
|
||||
/// </summary>
|
||||
Epss = 6,
|
||||
|
||||
/// <summary>
|
||||
/// Runtime observation (eBPF, dyld, ETW).
|
||||
/// Payload: RuntimeEvidence (observation type, call frames, timestamp).
|
||||
/// </summary>
|
||||
Runtime = 7,
|
||||
|
||||
/// <summary>
|
||||
/// Build provenance (SLSA, reproducibility).
|
||||
/// Payload: ProvenanceEvidence (build ID, builder, inputs, outputs).
|
||||
/// </summary>
|
||||
Provenance = 8,
|
||||
|
||||
/// <summary>
|
||||
/// Exception/waiver applied.
|
||||
/// Payload: ExceptionEvidence (exception ID, reason, expiry).
|
||||
/// </summary>
|
||||
Exception = 9,
|
||||
|
||||
/// <summary>
|
||||
/// Guard/gate analysis (feature flags, auth gates).
|
||||
/// Payload: GuardEvidence (gate type, condition, bypass confidence).
|
||||
/// </summary>
|
||||
Guard = 10
|
||||
}
|
||||
```
|
||||
|
||||
### EvidenceRecord Implementation
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/EvidenceRecord.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Concrete implementation of unified evidence record.
|
||||
/// </summary>
|
||||
public sealed record EvidenceRecord : IEvidence
|
||||
{
|
||||
public required string SubjectNodeId { get; init; }
|
||||
public required EvidenceType EvidenceType { get; init; }
|
||||
public required string EvidenceId { get; init; }
|
||||
public required ReadOnlyMemory<byte> Payload { get; init; }
|
||||
public IReadOnlyList<EvidenceSignature> Signatures { get; init; } = [];
|
||||
public required EvidenceProvenance Provenance { get; init; }
|
||||
public string? ExternalPayloadCid { get; init; }
|
||||
public required string PayloadSchemaVersion { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Computes EvidenceId from record contents using versioned canonicalization.
|
||||
/// </summary>
|
||||
public static string ComputeEvidenceId(
|
||||
string subjectNodeId,
|
||||
EvidenceType evidenceType,
|
||||
ReadOnlySpan<byte> payload,
|
||||
EvidenceProvenance provenance)
|
||||
{
|
||||
var hashInput = new EvidenceHashInput(
|
||||
subjectNodeId,
|
||||
evidenceType.ToString(),
|
||||
Convert.ToBase64String(payload),
|
||||
provenance);
|
||||
|
||||
return CanonJson.HashVersionedPrefixed(hashInput, CanonVersion.Current);
|
||||
}
|
||||
}
|
||||
|
||||
internal sealed record EvidenceHashInput(
|
||||
string SubjectNodeId,
|
||||
string EvidenceType,
|
||||
string PayloadBase64,
|
||||
EvidenceProvenance Provenance);
|
||||
```
|
||||
|
||||
### EvidenceSignature Model
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/EvidenceSignature.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Cryptographic signature on evidence.
|
||||
/// </summary>
|
||||
public sealed record EvidenceSignature
|
||||
{
|
||||
/// <summary>
|
||||
/// Signer identity (key ID, certificate subject, or service account).
|
||||
/// </summary>
|
||||
public required string SignerId { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Signature algorithm (e.g., "ES256", "RS256", "EdDSA").
|
||||
/// </summary>
|
||||
public required string Algorithm { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Base64-encoded signature bytes.
|
||||
/// </summary>
|
||||
public required string SignatureBase64 { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Timestamp when signature was created.
|
||||
/// </summary>
|
||||
public required DateTimeOffset SignedAt { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional key certificate chain for verification.
|
||||
/// </summary>
|
||||
public IReadOnlyList<string>? CertificateChain { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Signer type for categorization.
|
||||
/// </summary>
|
||||
public SignerType SignerType { get; init; } = SignerType.Internal;
|
||||
}
|
||||
|
||||
public enum SignerType
|
||||
{
|
||||
/// <summary>Internal StellaOps service.</summary>
|
||||
Internal,
|
||||
/// <summary>External vendor/supplier.</summary>
|
||||
Vendor,
|
||||
/// <summary>CI/CD pipeline.</summary>
|
||||
CI,
|
||||
/// <summary>Human operator.</summary>
|
||||
Operator,
|
||||
/// <summary>Third-party attestation service (e.g., Rekor).</summary>
|
||||
TransparencyLog
|
||||
}
|
||||
```
|
||||
|
||||
### EvidenceProvenance Model
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/EvidenceProvenance.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Provenance information for evidence generation.
|
||||
/// </summary>
|
||||
public sealed record EvidenceProvenance
|
||||
{
|
||||
/// <summary>
|
||||
/// Tool or service that generated this evidence.
|
||||
/// Format: "stellaops/{module}/{component}" or vendor identifier.
|
||||
/// </summary>
|
||||
public required string GeneratorId { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Version of the generator tool.
|
||||
/// </summary>
|
||||
public required string GeneratorVersion { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When the evidence was generated (UTC).
|
||||
/// </summary>
|
||||
public required DateTimeOffset GeneratedAt { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Content-addressed hash of inputs used to generate this evidence.
|
||||
/// Enables replay verification.
|
||||
/// </summary>
|
||||
public string? InputsDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Environment/region where evidence was generated.
|
||||
/// </summary>
|
||||
public string? Environment { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Scan run or evaluation ID for correlation.
|
||||
/// </summary>
|
||||
public string? CorrelationId { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Additional metadata for organization-specific tracking.
|
||||
/// </summary>
|
||||
public IReadOnlyDictionary<string, string>? Metadata { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### IEvidenceStore Interface
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/IEvidenceStore.cs
|
||||
namespace StellaOps.Evidence.Core;
|
||||
|
||||
/// <summary>
|
||||
/// Storage and retrieval interface for evidence records.
|
||||
/// </summary>
|
||||
public interface IEvidenceStore
|
||||
{
|
||||
/// <summary>
|
||||
/// Stores an evidence record.
|
||||
/// </summary>
|
||||
Task<string> StoreAsync(IEvidence evidence, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Retrieves evidence by its content-addressed ID.
|
||||
/// </summary>
|
||||
Task<IEvidence?> GetByIdAsync(string evidenceId, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Retrieves all evidence for a subject node.
|
||||
/// </summary>
|
||||
Task<IReadOnlyList<IEvidence>> GetBySubjectAsync(
|
||||
string subjectNodeId,
|
||||
EvidenceType? typeFilter = null,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Retrieves evidence by type across all subjects.
|
||||
/// </summary>
|
||||
Task<IReadOnlyList<IEvidence>> GetByTypeAsync(
|
||||
EvidenceType evidenceType,
|
||||
int limit = 100,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Checks if evidence exists for a subject.
|
||||
/// </summary>
|
||||
Task<bool> ExistsAsync(string subjectNodeId, EvidenceType type, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Deletes evidence by ID (for expiration/cleanup).
|
||||
/// </summary>
|
||||
Task<bool> DeleteAsync(string evidenceId, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
### Cross-Module Adapters
|
||||
|
||||
```csharp
|
||||
// src/__Libraries/StellaOps.Evidence.Core/Adapters/EvidenceBundleAdapter.cs
|
||||
namespace StellaOps.Evidence.Core.Adapters;
|
||||
|
||||
/// <summary>
|
||||
/// Converts Scanner's EvidenceBundle to unified IEvidence records.
|
||||
/// </summary>
|
||||
public sealed class EvidenceBundleAdapter
|
||||
{
|
||||
public IReadOnlyList<IEvidence> Convert(
|
||||
EvidenceBundle bundle,
|
||||
string subjectNodeId,
|
||||
EvidenceProvenance provenance)
|
||||
{
|
||||
var results = new List<IEvidence>();
|
||||
|
||||
if (bundle.Reachability is not null)
|
||||
{
|
||||
results.Add(CreateEvidence(
|
||||
subjectNodeId,
|
||||
EvidenceType.Reachability,
|
||||
bundle.Reachability,
|
||||
provenance,
|
||||
"reachability/v1"));
|
||||
}
|
||||
|
||||
if (bundle.Vex is not null)
|
||||
{
|
||||
results.Add(CreateEvidence(
|
||||
subjectNodeId,
|
||||
EvidenceType.Vex,
|
||||
bundle.Vex,
|
||||
provenance,
|
||||
"vex/v1"));
|
||||
}
|
||||
|
||||
if (bundle.Epss is not null)
|
||||
{
|
||||
results.Add(CreateEvidence(
|
||||
subjectNodeId,
|
||||
EvidenceType.Epss,
|
||||
bundle.Epss,
|
||||
provenance,
|
||||
"epss/v1"));
|
||||
}
|
||||
|
||||
// ... other evidence types
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
private static EvidenceRecord CreateEvidence<T>(
|
||||
string subjectNodeId,
|
||||
EvidenceType type,
|
||||
T payload,
|
||||
EvidenceProvenance provenance,
|
||||
string schemaVersion)
|
||||
{
|
||||
var payloadBytes = CanonJson.Canonicalize(payload);
|
||||
var evidenceId = EvidenceRecord.ComputeEvidenceId(
|
||||
subjectNodeId, type, payloadBytes, provenance);
|
||||
|
||||
return new EvidenceRecord
|
||||
{
|
||||
SubjectNodeId = subjectNodeId,
|
||||
EvidenceType = type,
|
||||
EvidenceId = evidenceId,
|
||||
Payload = payloadBytes,
|
||||
Provenance = provenance,
|
||||
PayloadSchemaVersion = schemaVersion
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Core Types)** | | | | | |
|
||||
| 1 | EVID-8100-001 | TODO | Canon versioning | Platform Guild | Create `StellaOps.Evidence.Core` project with dependencies. |
|
||||
| 2 | EVID-8100-002 | TODO | Task 1 | Platform Guild | Define `EvidenceType` enum with all known types. |
|
||||
| 3 | EVID-8100-003 | TODO | Task 1 | Platform Guild | Define `IEvidence` interface. |
|
||||
| 4 | EVID-8100-004 | TODO | Task 3 | Platform Guild | Define `EvidenceSignature` record. |
|
||||
| 5 | EVID-8100-005 | TODO | Task 3 | Platform Guild | Define `EvidenceProvenance` record. |
|
||||
| 6 | EVID-8100-006 | TODO | Tasks 3-5 | Platform Guild | Implement `EvidenceRecord` with `ComputeEvidenceId()`. |
|
||||
| **Wave 1 (Store Interface)** | | | | | |
|
||||
| 7 | EVID-8100-007 | TODO | Task 6 | Platform Guild | Define `IEvidenceStore` interface. |
|
||||
| 8 | EVID-8100-008 | TODO | Task 7 | Platform Guild | Implement in-memory `EvidenceStore` for testing. |
|
||||
| 9 | EVID-8100-009 | TODO | Task 7 | Platform Guild | Implement PostgreSQL `EvidenceStore` (schema + repository). |
|
||||
| **Wave 2 (Adapters)** | | | | | |
|
||||
| 10 | EVID-8100-010 | TODO | Task 6 | Scanner Guild | Create `EvidenceBundleAdapter` (Scanner → IEvidence). |
|
||||
| 11 | EVID-8100-011 | TODO | Task 6 | Attestor Guild | Create `EvidenceStatementAdapter` (Attestor → IEvidence). |
|
||||
| 12 | EVID-8100-012 | TODO | Task 6 | Scanner Guild | Create `ProofSegmentAdapter` (ProofSpine → IEvidence). |
|
||||
| 13 | EVID-8100-013 | TODO | Task 6 | Excititor Guild | Create `VexObservationAdapter` (Excititor → IEvidence). |
|
||||
| 14 | EVID-8100-014 | TODO | Task 6 | Policy Guild | Create `ExceptionApplicationAdapter` (Policy → IEvidence). |
|
||||
| **Wave 3 (Tests)** | | | | | |
|
||||
| 15 | EVID-8100-015 | TODO | Tasks 6-14 | QA Guild | Add unit tests: EvidenceRecord creation and ID computation. |
|
||||
| 16 | EVID-8100-016 | TODO | Task 15 | QA Guild | Add unit tests: All adapters convert losslessly. |
|
||||
| 17 | EVID-8100-017 | TODO | Task 9 | QA Guild | Add integration tests: PostgreSQL store CRUD operations. |
|
||||
| 18 | EVID-8100-018 | TODO | Task 17 | QA Guild | Add integration tests: Cross-module evidence linking. |
|
||||
| **Wave 4 (Documentation)** | | | | | |
|
||||
| 19 | EVID-8100-019 | TODO | Tasks 6-14 | Docs Guild | Create `docs/modules/evidence/unified-model.md`. |
|
||||
| 20 | EVID-8100-020 | TODO | Task 19 | Docs Guild | Update module READMEs with IEvidence integration notes. |
|
||||
| 21 | EVID-8100-021 | TODO | Task 19 | Docs Guild | Add API reference for evidence types and store. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-6 | Core types | Project compiles; IEvidence defined; EvidenceRecord works |
|
||||
| **Wave 1** | 7-9 | Store interface | IEvidenceStore defined; in-memory and PostgreSQL implementations work |
|
||||
| **Wave 2** | 10-14 | Adapters | All module evidence types convert to IEvidence |
|
||||
| **Wave 3** | 15-18 | Tests | All tests pass; cross-module linking verified |
|
||||
| **Wave 4** | 19-21 | Documentation | Docs complete; API reference published |
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL Schema
|
||||
|
||||
```sql
|
||||
-- Evidence store schema
|
||||
CREATE TABLE IF NOT EXISTS evidence.records (
|
||||
evidence_id TEXT PRIMARY KEY,
|
||||
subject_node_id TEXT NOT NULL,
|
||||
evidence_type SMALLINT NOT NULL,
|
||||
payload BYTEA NOT NULL,
|
||||
payload_schema_ver TEXT NOT NULL,
|
||||
external_cid TEXT,
|
||||
provenance JSONB NOT NULL,
|
||||
signatures JSONB NOT NULL DEFAULT '[]',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
tenant_id UUID NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_evidence_subject ON evidence.records (subject_node_id, evidence_type);
|
||||
CREATE INDEX idx_evidence_type ON evidence.records (evidence_type, created_at DESC);
|
||||
CREATE INDEX idx_evidence_tenant ON evidence.records (tenant_id, created_at DESC);
|
||||
|
||||
-- RLS policy
|
||||
ALTER TABLE evidence.records ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY evidence_tenant_isolation ON evidence.records
|
||||
USING (tenant_id = current_setting('app.tenant_id')::uuid);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| `IEvidence` is read-only interface | Immutable evidence records for integrity |
|
||||
| Payload stored as canonical JSON bytes | Enables hash verification without deserialization |
|
||||
| Adapters convert existing types | Non-breaking migration; existing code continues working |
|
||||
| PostgreSQL for durable store | Consistent with StellaOps persistence patterns |
|
||||
| SignerType enum for categorization | Enables filtering/prioritization of signatures |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Schema drift across evidence types | Adapter failures | Explicit schema versions; validation on read | Platform Guild |
|
||||
| Large payloads (reachability graphs) | Storage/bandwidth | External CID support; chunking | Platform Guild |
|
||||
| Cross-module circular dependencies | Build failures | Evidence.Core has no module dependencies | Platform Guild |
|
||||
| Migration of existing evidence | Data loss | Adapters; parallel storage during transition | All Guilds |
|
||||
| Performance of GetBySubject queries | Latency | Composite index; pagination | Platform Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Merkle-Hash REG product advisory gap analysis. | Project Mgmt |
|
||||
682
docs/implplan/SPRINT_8100_0012_0003_graph_root_attestation.md
Normal file
682
docs/implplan/SPRINT_8100_0012_0003_graph_root_attestation.md
Normal file
@@ -0,0 +1,682 @@
|
||||
# Sprint 8100.0012.0003 · Graph Root Attestation Service
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement explicit DSSE attestation of Merkle graph roots, enabling offline verification that replayed graphs match the original attested state. This sprint delivers:
|
||||
|
||||
1. **IGraphRootAttestor Interface**: Service contract for attesting graph roots with DSSE envelopes.
|
||||
2. **GraphRootAttestation Model**: In-toto statement with graph root as subject, linked evidence and child node IDs.
|
||||
3. **GraphRootVerifier**: Verifier that recomputes graph root from nodes/edges and validates against attestation.
|
||||
4. **Integration with ProofSpine**: Extend ProofSpine to emit and reference graph root attestations.
|
||||
5. **Rekor Integration**: Optional transparency log publishing for graph root attestations.
|
||||
|
||||
**Working directory:** `src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/` (new), `src/Scanner/__Libraries/StellaOps.Scanner.ProofSpine/`, `src/Attestor/__Tests/`.
|
||||
|
||||
**Evidence:** Graph roots are attested as first-class DSSE envelopes; offline verifiers can recompute roots and validate against attestations; Rekor entries exist for transparency; ProofSpine references graph root attestations.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8100.0012.0001 (Canonicalizer versioning), Sprint 8100.0012.0002 (Unified Evidence Model).
|
||||
- **Blocks:** None (enables advanced verification scenarios).
|
||||
- **Safe to run in parallel with:** Unrelated module work (after dependencies complete).
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/attestor/proof-chain.md` (Existing proof chain design)
|
||||
- `docs/modules/attestor/dsse-envelopes.md` (DSSE envelope generation)
|
||||
- Product Advisory: Merkle-Hash REG graph root attestation
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
StellaOps computes graph roots in several places:
|
||||
|
||||
| Component | Root Computation | Attestation |
|
||||
|-----------|-----------------|-------------|
|
||||
| `DeterministicMerkleTreeBuilder` | Merkle root from leaves | None (raw bytes) |
|
||||
| `ContentAddressedIdGenerator.ComputeGraphRevisionId()` | Combined hash of nodes, edges, digests | None (ID only) |
|
||||
| `ProofSpine.RootHash` | Hash of spine segments | Referenced in spine, not independently attested |
|
||||
| `RichGraph` (Reachability) | Implicit in builder | None |
|
||||
|
||||
**Problem:** Graph roots are computed but not **attested as first-class entities**. A verifier cannot request "prove this graph root is authentic" without reconstructing the entire chain.
|
||||
|
||||
### Target State
|
||||
|
||||
Per the product advisory:
|
||||
> Emit a graph root; store alongside an attestation (DSSE/in-toto). Verifiers recompute to confirm integrity.
|
||||
|
||||
Graph root attestations enable:
|
||||
- **Offline verification:** Verifier downloads attestation, recomputes root, compares
|
||||
- **Audit snapshots:** Point-in-time proof of graph state
|
||||
- **Evidence linking:** Evidence references attested roots, not transient IDs
|
||||
- **Transparency:** Optional Rekor publication for public auditability
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### IGraphRootAttestor Interface
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/IGraphRootAttestor.cs
|
||||
namespace StellaOps.Attestor.GraphRoot;
|
||||
|
||||
/// <summary>
|
||||
/// Service for creating DSSE attestations of Merkle graph roots.
|
||||
/// </summary>
|
||||
public interface IGraphRootAttestor
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates a DSSE-wrapped attestation for a graph root.
|
||||
/// </summary>
|
||||
/// <param name="request">Graph root attestation request.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>DSSE envelope containing the graph root attestation.</returns>
|
||||
Task<GraphRootAttestationResult> AttestAsync(
|
||||
GraphRootAttestationRequest request,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Verifies a graph root attestation by recomputing the root.
|
||||
/// </summary>
|
||||
/// <param name="envelope">DSSE envelope to verify.</param>
|
||||
/// <param name="nodes">Node data for recomputation.</param>
|
||||
/// <param name="edges">Edge data for recomputation.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>Verification result with details.</returns>
|
||||
Task<GraphRootVerificationResult> VerifyAsync(
|
||||
DsseEnvelope envelope,
|
||||
IReadOnlyList<GraphNodeData> nodes,
|
||||
IReadOnlyList<GraphEdgeData> edges,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
### GraphRootAttestationRequest
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/Models/GraphRootAttestationRequest.cs
|
||||
namespace StellaOps.Attestor.GraphRoot.Models;
|
||||
|
||||
/// <summary>
|
||||
/// Request to create a graph root attestation.
|
||||
/// </summary>
|
||||
public sealed record GraphRootAttestationRequest
|
||||
{
|
||||
/// <summary>
|
||||
/// Type of graph being attested.
|
||||
/// </summary>
|
||||
public required GraphType GraphType { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Node IDs (content-addressed) in the graph.
|
||||
/// </summary>
|
||||
public required IReadOnlyList<string> NodeIds { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Edge IDs (content-addressed) in the graph.
|
||||
/// </summary>
|
||||
public required IReadOnlyList<string> EdgeIds { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Policy digest used for graph evaluation.
|
||||
/// </summary>
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Advisory/vulnerability feed snapshot digest.
|
||||
/// </summary>
|
||||
public required string FeedsDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Toolchain digest (scanner, analyzer versions).
|
||||
/// </summary>
|
||||
public required string ToolchainDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Evaluation parameters digest.
|
||||
/// </summary>
|
||||
public required string ParamsDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Artifact digest this graph describes.
|
||||
/// </summary>
|
||||
public required string ArtifactDigest { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Linked evidence IDs included in this graph.
|
||||
/// </summary>
|
||||
public IReadOnlyList<string> EvidenceIds { get; init; } = [];
|
||||
|
||||
/// <summary>
|
||||
/// Whether to publish to Rekor transparency log.
|
||||
/// </summary>
|
||||
public bool PublishToRekor { get; init; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// Signing key ID to use.
|
||||
/// </summary>
|
||||
public string? SigningKeyId { get; init; }
|
||||
}
|
||||
|
||||
public enum GraphType
|
||||
{
|
||||
/// <summary>Resolved Execution Graph (full proof chain).</summary>
|
||||
ResolvedExecutionGraph = 1,
|
||||
|
||||
/// <summary>Reachability call graph.</summary>
|
||||
ReachabilityGraph = 2,
|
||||
|
||||
/// <summary>SBOM dependency graph.</summary>
|
||||
DependencyGraph = 3,
|
||||
|
||||
/// <summary>Proof spine (decision chain).</summary>
|
||||
ProofSpine = 4,
|
||||
|
||||
/// <summary>Evidence linkage graph.</summary>
|
||||
EvidenceGraph = 5
|
||||
}
|
||||
```
|
||||
|
||||
### GraphRootAttestation (In-Toto Statement)
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/Models/GraphRootAttestation.cs
|
||||
namespace StellaOps.Attestor.GraphRoot.Models;
|
||||
|
||||
/// <summary>
|
||||
/// In-toto statement for graph root attestation.
|
||||
/// PredicateType: "https://stella-ops.org/attestation/graph-root/v1"
|
||||
/// </summary>
|
||||
public sealed record GraphRootAttestation
|
||||
{
|
||||
/// <summary>
|
||||
/// In-toto statement type.
|
||||
/// </summary>
|
||||
public string _type { get; init; } = "https://in-toto.io/Statement/v1";
|
||||
|
||||
/// <summary>
|
||||
/// Subjects: the graph root hash and artifact it describes.
|
||||
/// </summary>
|
||||
public required IReadOnlyList<InTotoSubject> Subject { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Predicate type for graph root attestations.
|
||||
/// </summary>
|
||||
public string PredicateType { get; init; } = "https://stella-ops.org/attestation/graph-root/v1";
|
||||
|
||||
/// <summary>
|
||||
/// Graph root predicate payload.
|
||||
/// </summary>
|
||||
public required GraphRootPredicate Predicate { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Predicate for graph root attestation.
|
||||
/// </summary>
|
||||
public sealed record GraphRootPredicate
|
||||
{
|
||||
/// <summary>
|
||||
/// Graph type discriminator.
|
||||
/// </summary>
|
||||
public required string GraphType { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Computed Merkle root hash.
|
||||
/// </summary>
|
||||
public required string RootHash { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Algorithm used for root computation.
|
||||
/// </summary>
|
||||
public string RootAlgorithm { get; init; } = "sha256";
|
||||
|
||||
/// <summary>
|
||||
/// Number of nodes in the graph.
|
||||
/// </summary>
|
||||
public required int NodeCount { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Number of edges in the graph.
|
||||
/// </summary>
|
||||
public required int EdgeCount { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Sorted node IDs for deterministic verification.
|
||||
/// </summary>
|
||||
public required IReadOnlyList<string> NodeIds { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Sorted edge IDs for deterministic verification.
|
||||
/// </summary>
|
||||
public required IReadOnlyList<string> EdgeIds { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Input digests for reproducibility.
|
||||
/// </summary>
|
||||
public required GraphInputDigests Inputs { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Linked evidence IDs referenced by this graph.
|
||||
/// </summary>
|
||||
public IReadOnlyList<string> EvidenceIds { get; init; } = [];
|
||||
|
||||
/// <summary>
|
||||
/// Canonicalizer version used.
|
||||
/// </summary>
|
||||
public required string CanonVersion { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When the root was computed.
|
||||
/// </summary>
|
||||
public required DateTimeOffset ComputedAt { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Tool that computed the root.
|
||||
/// </summary>
|
||||
public required string ComputedBy { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Tool version.
|
||||
/// </summary>
|
||||
public required string ComputedByVersion { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Input digests for graph computation.
|
||||
/// </summary>
|
||||
public sealed record GraphInputDigests
|
||||
{
|
||||
public required string PolicyDigest { get; init; }
|
||||
public required string FeedsDigest { get; init; }
|
||||
public required string ToolchainDigest { get; init; }
|
||||
public required string ParamsDigest { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### GraphRootAttestor Implementation
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/GraphRootAttestor.cs
|
||||
namespace StellaOps.Attestor.GraphRoot;
|
||||
|
||||
public sealed class GraphRootAttestor : IGraphRootAttestor
|
||||
{
|
||||
private readonly IMerkleTreeBuilder _merkleBuilder;
|
||||
private readonly IDsseSigner _signer;
|
||||
private readonly IRekorClient? _rekorClient;
|
||||
private readonly ILogger<GraphRootAttestor> _logger;
|
||||
|
||||
public GraphRootAttestor(
|
||||
IMerkleTreeBuilder merkleBuilder,
|
||||
IDsseSigner signer,
|
||||
IRekorClient? rekorClient,
|
||||
ILogger<GraphRootAttestor> logger)
|
||||
{
|
||||
_merkleBuilder = merkleBuilder;
|
||||
_signer = signer;
|
||||
_rekorClient = rekorClient;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public async Task<GraphRootAttestationResult> AttestAsync(
|
||||
GraphRootAttestationRequest request,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
// 1. Sort node and edge IDs lexicographically
|
||||
var sortedNodeIds = request.NodeIds.OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
var sortedEdgeIds = request.EdgeIds.OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
|
||||
// 2. Compute Merkle root
|
||||
var leaves = new List<ReadOnlyMemory<byte>>();
|
||||
foreach (var nodeId in sortedNodeIds)
|
||||
leaves.Add(Encoding.UTF8.GetBytes(nodeId));
|
||||
foreach (var edgeId in sortedEdgeIds)
|
||||
leaves.Add(Encoding.UTF8.GetBytes(edgeId));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(request.PolicyDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(request.FeedsDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(request.ToolchainDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(request.ParamsDigest));
|
||||
|
||||
var rootBytes = _merkleBuilder.ComputeMerkleRoot(leaves);
|
||||
var rootHash = $"sha256:{Convert.ToHexStringLower(rootBytes)}";
|
||||
|
||||
// 3. Build in-toto statement
|
||||
var attestation = new GraphRootAttestation
|
||||
{
|
||||
Subject =
|
||||
[
|
||||
new InTotoSubject
|
||||
{
|
||||
Name = rootHash,
|
||||
Digest = new Dictionary<string, string> { ["sha256"] = Convert.ToHexStringLower(rootBytes) }
|
||||
},
|
||||
new InTotoSubject
|
||||
{
|
||||
Name = request.ArtifactDigest,
|
||||
Digest = ParseDigest(request.ArtifactDigest)
|
||||
}
|
||||
],
|
||||
Predicate = new GraphRootPredicate
|
||||
{
|
||||
GraphType = request.GraphType.ToString(),
|
||||
RootHash = rootHash,
|
||||
NodeCount = sortedNodeIds.Count,
|
||||
EdgeCount = sortedEdgeIds.Count,
|
||||
NodeIds = sortedNodeIds,
|
||||
EdgeIds = sortedEdgeIds,
|
||||
Inputs = new GraphInputDigests
|
||||
{
|
||||
PolicyDigest = request.PolicyDigest,
|
||||
FeedsDigest = request.FeedsDigest,
|
||||
ToolchainDigest = request.ToolchainDigest,
|
||||
ParamsDigest = request.ParamsDigest
|
||||
},
|
||||
EvidenceIds = request.EvidenceIds.OrderBy(x => x, StringComparer.Ordinal).ToList(),
|
||||
CanonVersion = CanonVersion.Current,
|
||||
ComputedAt = DateTimeOffset.UtcNow,
|
||||
ComputedBy = "stellaops/attestor/graph-root",
|
||||
ComputedByVersion = GetVersion()
|
||||
}
|
||||
};
|
||||
|
||||
// 4. Canonicalize and sign
|
||||
var payload = CanonJson.CanonicalizeVersioned(attestation, CanonVersion.Current);
|
||||
var envelope = await _signer.SignAsync(
|
||||
payload,
|
||||
"application/vnd.in-toto+json",
|
||||
request.SigningKeyId,
|
||||
ct);
|
||||
|
||||
// 5. Optionally publish to Rekor
|
||||
string? rekorLogIndex = null;
|
||||
if (request.PublishToRekor && _rekorClient is not null)
|
||||
{
|
||||
var rekorResult = await _rekorClient.UploadAsync(envelope, ct);
|
||||
rekorLogIndex = rekorResult.LogIndex;
|
||||
}
|
||||
|
||||
return new GraphRootAttestationResult
|
||||
{
|
||||
RootHash = rootHash,
|
||||
Envelope = envelope,
|
||||
RekorLogIndex = rekorLogIndex,
|
||||
NodeCount = sortedNodeIds.Count,
|
||||
EdgeCount = sortedEdgeIds.Count
|
||||
};
|
||||
}
|
||||
|
||||
public async Task<GraphRootVerificationResult> VerifyAsync(
|
||||
DsseEnvelope envelope,
|
||||
IReadOnlyList<GraphNodeData> nodes,
|
||||
IReadOnlyList<GraphEdgeData> edges,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
// 1. Verify envelope signature
|
||||
var signatureValid = await _signer.VerifyAsync(envelope, ct);
|
||||
if (!signatureValid)
|
||||
{
|
||||
return new GraphRootVerificationResult
|
||||
{
|
||||
IsValid = false,
|
||||
FailureReason = "Envelope signature verification failed"
|
||||
};
|
||||
}
|
||||
|
||||
// 2. Deserialize attestation
|
||||
var attestation = JsonSerializer.Deserialize<GraphRootAttestation>(envelope.Payload);
|
||||
if (attestation is null)
|
||||
{
|
||||
return new GraphRootVerificationResult
|
||||
{
|
||||
IsValid = false,
|
||||
FailureReason = "Failed to deserialize attestation"
|
||||
};
|
||||
}
|
||||
|
||||
// 3. Recompute root from provided nodes/edges
|
||||
var recomputedIds = nodes.Select(n => n.NodeId).OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
var recomputedEdgeIds = edges.Select(e => e.EdgeId).OrderBy(x => x, StringComparer.Ordinal).ToList();
|
||||
|
||||
var leaves = new List<ReadOnlyMemory<byte>>();
|
||||
foreach (var nodeId in recomputedIds)
|
||||
leaves.Add(Encoding.UTF8.GetBytes(nodeId));
|
||||
foreach (var edgeId in recomputedEdgeIds)
|
||||
leaves.Add(Encoding.UTF8.GetBytes(edgeId));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(attestation.Predicate.Inputs.PolicyDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(attestation.Predicate.Inputs.FeedsDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(attestation.Predicate.Inputs.ToolchainDigest));
|
||||
leaves.Add(Encoding.UTF8.GetBytes(attestation.Predicate.Inputs.ParamsDigest));
|
||||
|
||||
var recomputedRoot = _merkleBuilder.ComputeMerkleRoot(leaves);
|
||||
var recomputedRootHash = $"sha256:{Convert.ToHexStringLower(recomputedRoot)}";
|
||||
|
||||
// 4. Compare
|
||||
if (recomputedRootHash != attestation.Predicate.RootHash)
|
||||
{
|
||||
return new GraphRootVerificationResult
|
||||
{
|
||||
IsValid = false,
|
||||
FailureReason = $"Root mismatch: expected {attestation.Predicate.RootHash}, got {recomputedRootHash}",
|
||||
ExpectedRoot = attestation.Predicate.RootHash,
|
||||
ComputedRoot = recomputedRootHash
|
||||
};
|
||||
}
|
||||
|
||||
return new GraphRootVerificationResult
|
||||
{
|
||||
IsValid = true,
|
||||
ExpectedRoot = attestation.Predicate.RootHash,
|
||||
ComputedRoot = recomputedRootHash,
|
||||
NodeCount = recomputedIds.Count,
|
||||
EdgeCount = recomputedEdgeIds.Count
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Result Models
|
||||
|
||||
```csharp
|
||||
// src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/Models/GraphRootAttestationResult.cs
|
||||
namespace StellaOps.Attestor.GraphRoot.Models;
|
||||
|
||||
public sealed record GraphRootAttestationResult
|
||||
{
|
||||
public required string RootHash { get; init; }
|
||||
public required DsseEnvelope Envelope { get; init; }
|
||||
public string? RekorLogIndex { get; init; }
|
||||
public required int NodeCount { get; init; }
|
||||
public required int EdgeCount { get; init; }
|
||||
}
|
||||
|
||||
public sealed record GraphRootVerificationResult
|
||||
{
|
||||
public required bool IsValid { get; init; }
|
||||
public string? FailureReason { get; init; }
|
||||
public string? ExpectedRoot { get; init; }
|
||||
public string? ComputedRoot { get; init; }
|
||||
public int NodeCount { get; init; }
|
||||
public int EdgeCount { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with ProofSpine
|
||||
|
||||
### Extended ProofSpine Model
|
||||
|
||||
```csharp
|
||||
// Extension to ProofSpineModels.cs
|
||||
public sealed record ProofSpine(
|
||||
string SpineId,
|
||||
string ArtifactId,
|
||||
string VulnerabilityId,
|
||||
string PolicyProfileId,
|
||||
IReadOnlyList<ProofSegment> Segments,
|
||||
string Verdict,
|
||||
string VerdictReason,
|
||||
string RootHash,
|
||||
string ScanRunId,
|
||||
DateTimeOffset CreatedAt,
|
||||
string? SupersededBySpineId,
|
||||
// NEW: Reference to graph root attestation
|
||||
string? GraphRootAttestationId,
|
||||
DsseEnvelope? GraphRootEnvelope);
|
||||
```
|
||||
|
||||
### ProofSpineBuilder Extension
|
||||
|
||||
```csharp
|
||||
// Extension to emit graph root attestation
|
||||
public async Task<ProofSpine> BuildWithAttestationAsync(
|
||||
ProofSpineBuildRequest request,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
var spine = Build(request);
|
||||
|
||||
// Attest the graph root
|
||||
var attestRequest = new GraphRootAttestationRequest
|
||||
{
|
||||
GraphType = GraphType.ProofSpine,
|
||||
NodeIds = spine.Segments.Select(s => s.SegmentId).ToList(),
|
||||
EdgeIds = spine.Segments.Skip(1).Select((s, i) =>
|
||||
$"{spine.Segments[i].SegmentId}->{s.SegmentId}").ToList(),
|
||||
PolicyDigest = request.PolicyDigest,
|
||||
FeedsDigest = request.FeedsDigest,
|
||||
ToolchainDigest = request.ToolchainDigest,
|
||||
ParamsDigest = request.ParamsDigest,
|
||||
ArtifactDigest = request.ArtifactDigest,
|
||||
EvidenceIds = request.EvidenceIds,
|
||||
PublishToRekor = request.PublishToRekor
|
||||
};
|
||||
|
||||
var attestResult = await _graphRootAttestor.AttestAsync(attestRequest, ct);
|
||||
|
||||
return spine with
|
||||
{
|
||||
GraphRootAttestationId = attestResult.RootHash,
|
||||
GraphRootEnvelope = attestResult.Envelope
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Project & Models)** | | | | | |
|
||||
| 1 | GROOT-8100-001 | TODO | Canon + Evidence | Attestor Guild | Create `StellaOps.Attestor.GraphRoot` project with dependencies. |
|
||||
| 2 | GROOT-8100-002 | TODO | Task 1 | Attestor Guild | Define `GraphType` enum. |
|
||||
| 3 | GROOT-8100-003 | TODO | Task 1 | Attestor Guild | Define `GraphRootAttestationRequest` model. |
|
||||
| 4 | GROOT-8100-004 | TODO | Task 1 | Attestor Guild | Define `GraphRootAttestation` in-toto statement model. |
|
||||
| 5 | GROOT-8100-005 | TODO | Task 1 | Attestor Guild | Define `GraphRootPredicate` and `GraphInputDigests` models. |
|
||||
| 6 | GROOT-8100-006 | TODO | Task 1 | Attestor Guild | Define result models (`GraphRootAttestationResult`, `GraphRootVerificationResult`). |
|
||||
| **Wave 1 (Core Implementation)** | | | | | |
|
||||
| 7 | GROOT-8100-007 | TODO | Tasks 2-6 | Attestor Guild | Define `IGraphRootAttestor` interface. |
|
||||
| 8 | GROOT-8100-008 | TODO | Task 7 | Attestor Guild | Implement `GraphRootAttestor.AttestAsync()`. |
|
||||
| 9 | GROOT-8100-009 | TODO | Task 8 | Attestor Guild | Implement `GraphRootAttestor.VerifyAsync()`. |
|
||||
| 10 | GROOT-8100-010 | TODO | Task 8 | Attestor Guild | Integrate Rekor publishing (optional). |
|
||||
| **Wave 2 (ProofSpine Integration)** | | | | | |
|
||||
| 11 | GROOT-8100-011 | TODO | Task 8 | Scanner Guild | Extend `ProofSpine` model with attestation reference. |
|
||||
| 12 | GROOT-8100-012 | TODO | Task 11 | Scanner Guild | Extend `ProofSpineBuilder` with `BuildWithAttestationAsync()`. |
|
||||
| 13 | GROOT-8100-013 | TODO | Task 12 | Scanner Guild | Update scan pipeline to emit graph root attestations. |
|
||||
| **Wave 3 (RichGraph Integration)** | | | | | |
|
||||
| 14 | GROOT-8100-014 | TODO | Task 8 | Scanner Guild | Add graph root attestation to `RichGraphBuilder`. |
|
||||
| 15 | GROOT-8100-015 | TODO | Task 14 | Scanner Guild | Store attestation alongside RichGraph in CAS. |
|
||||
| **Wave 4 (Tests)** | | | | | |
|
||||
| 16 | GROOT-8100-016 | TODO | Tasks 8-9 | QA Guild | Add unit tests: attestation creation and verification. |
|
||||
| 17 | GROOT-8100-017 | TODO | Task 16 | QA Guild | Add determinism tests: same inputs → same root. |
|
||||
| 18 | GROOT-8100-018 | TODO | Task 16 | QA Guild | Add tamper detection tests: modified nodes → verification fails. |
|
||||
| 19 | GROOT-8100-019 | TODO | Task 10 | QA Guild | Add Rekor integration tests (mock). |
|
||||
| 20 | GROOT-8100-020 | TODO | Tasks 12-15 | QA Guild | Add integration tests: full pipeline with attestation. |
|
||||
| **Wave 5 (Documentation)** | | | | | |
|
||||
| 21 | GROOT-8100-021 | TODO | Tasks 8-15 | Docs Guild | Create `docs/modules/attestor/graph-root-attestation.md`. |
|
||||
| 22 | GROOT-8100-022 | TODO | Task 21 | Docs Guild | Update proof chain documentation with attestation flow. |
|
||||
| 23 | GROOT-8100-023 | TODO | Task 21 | Docs Guild | Document offline verification workflow. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-6 | Project & models | Project compiles; all models defined |
|
||||
| **Wave 1** | 7-10 | Core implementation | Attestation/verification works; Rekor optional |
|
||||
| **Wave 2** | 11-13 | ProofSpine integration | ProofSpine emits attestations |
|
||||
| **Wave 3** | 14-15 | RichGraph integration | Reachability graphs attested |
|
||||
| **Wave 4** | 16-20 | Tests | All tests pass |
|
||||
| **Wave 5** | 21-23 | Documentation | Docs complete |
|
||||
|
||||
---
|
||||
|
||||
## Verification Workflow
|
||||
|
||||
### Offline Verification Steps
|
||||
|
||||
1. **Obtain attestation:** Download DSSE envelope for graph root
|
||||
2. **Verify signature:** Check envelope signature against trusted keys
|
||||
3. **Extract predicate:** Parse `GraphRootPredicate` from payload
|
||||
4. **Fetch graph data:** Download nodes and edges by ID from storage
|
||||
5. **Recompute root:** Apply same Merkle tree algorithm to node/edge IDs + input digests
|
||||
6. **Compare:** Computed root must match `predicate.RootHash`
|
||||
|
||||
### CLI Command (Future)
|
||||
|
||||
```bash
|
||||
# Verify a graph root attestation
|
||||
stellaops verify graph-root \
|
||||
--envelope attestation.dsse.json \
|
||||
--nodes nodes.ndjson \
|
||||
--edges edges.ndjson \
|
||||
--trusted-keys keys.json
|
||||
|
||||
# Output
|
||||
✓ Signature valid (signer: stellaops/scanner)
|
||||
✓ Root hash matches: sha256:abc123...
|
||||
✓ Node count: 1,247
|
||||
✓ Edge count: 3,891
|
||||
✓ Verification successful
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| In-toto statement format | Standard attestation format; tooling compatibility |
|
||||
| Two subjects (root + artifact) | Links graph to specific artifact; enables queries |
|
||||
| Node/edge IDs in predicate | Enables independent recomputation without storage access |
|
||||
| Rekor integration optional | Air-gap compatibility; transparency when network available |
|
||||
| Extend ProofSpine vs. new entity | Keeps decision chain unified; attestation enhances existing |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Large graphs exceed predicate size | Envelope too big | Node/edge IDs in external file; reference by CID | Attestor Guild |
|
||||
| Signing key management | Security | Delegate to existing Signer module | Crypto Guild |
|
||||
| Rekor rate limits | Publishing failures | Backoff/retry; batch uploads | Attestor Guild |
|
||||
| Verification performance | Latency | Parallel node/edge fetching; caching | Platform Guild |
|
||||
| Schema evolution | Breaking changes | Explicit predicate type versioning | Attestor Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Merkle-Hash REG product advisory gap analysis. | Project Mgmt |
|
||||
366
docs/implplan/SPRINT_8200_0001_0001_provcache_core_backend.md
Normal file
366
docs/implplan/SPRINT_8200_0001_0001_provcache_core_backend.md
Normal file
@@ -0,0 +1,366 @@
|
||||
# Sprint 8200.0001.0001 · Provcache Core Backend
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement the **Provenance Cache (Provcache)** core backend layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte — enabling faster decisions, offline replays, and smaller air-gap bundles. This sprint delivers:
|
||||
|
||||
1. **VeriKey Composite Hash**: Implement the tuple-based cache key `(source_hash, sbom_hash, vex_hash_set_hash, merge_policy_hash, signer_set_hash, time_window)`.
|
||||
2. **DecisionDigest**: Wrap TrustLattice evaluation output into canonicalized, deterministic digests.
|
||||
3. **Provcache Service API**: Implement `/v1/provcache/*` endpoints for cache operations.
|
||||
4. **Valkey Read-Through Layer**: Fast cache lookup with Postgres write-behind for persistence.
|
||||
5. **Policy Engine Integration**: Wire Provcache into PolicyEvaluator merge output path.
|
||||
|
||||
**Working directory:** `src/__Libraries/StellaOps.Provcache/` (new), `src/__Libraries/__Tests/StellaOps.Provcache.Tests/` (tests), integration with `src/Policy/StellaOps.Policy.Engine/`.
|
||||
|
||||
**Evidence:** VeriKey determinism tests pass; DecisionDigest reproducibility verified; cache hit/miss metrics exposed; policy evaluation latency reduced on warm cache.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** `TrustLatticeEngine`, `CanonicalJsonSerializer`, `ValkeyCacheStore`, `ICryptoHash`, `ProofBundle`.
|
||||
- **Recommended to land before:** Sprint 8200.0001.0002 (Invalidation & Air-Gap) and Sprint 8200.0001.0003 (UX & Observability).
|
||||
- **Safe to run in parallel with:** Other module tests sprints that don't modify Policy engine internals.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/policy/README.md`
|
||||
- `docs/modules/policy/design/policy-deterministic-evaluator.md`
|
||||
- `docs/db/SPECIFICATION.md`
|
||||
- `src/Policy/__Libraries/StellaOps.Policy/TrustLattice/TrustLatticeEngine.cs`
|
||||
- `src/__Libraries/StellaOps.Messaging.Transport.Valkey/ValkeyCacheStore.cs`
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### VeriKey Tuple
|
||||
|
||||
The VeriKey is a composite hash that uniquely identifies a provenance decision context:
|
||||
|
||||
```
|
||||
VeriKey = Hash(
|
||||
source_hash, // Image/artifact content-addressed digest
|
||||
sbom_hash, // SBOM canonical hash (SPDX/CycloneDX)
|
||||
vex_hash_set_hash, // Sorted set of VEX statement hashes
|
||||
merge_policy_hash, // PolicyBundle hash (rules, precedence)
|
||||
signer_set_hash, // Sorted set of signer certificate hashes
|
||||
time_window // Epoch bucket (e.g., hourly, daily)
|
||||
)
|
||||
```
|
||||
|
||||
### DecisionDigest
|
||||
|
||||
Canonicalized representation of evaluation output:
|
||||
|
||||
```csharp
|
||||
public sealed record DecisionDigest
|
||||
{
|
||||
public required string VeriKey { get; init; }
|
||||
public required string DigestVersion { get; init; } // "v1"
|
||||
public required string VerdictHash { get; init; } // Hash of sorted dispositions
|
||||
public required string ProofRoot { get; init; } // Merkle root of evidence
|
||||
public required string ReplaySeed { get; init; } // Feed/rule IDs for replay
|
||||
public required DateTimeOffset CreatedAt { get; init; }
|
||||
public required DateTimeOffset ExpiresAt { get; init; }
|
||||
public required int TrustScore { get; init; } // 0-100
|
||||
}
|
||||
```
|
||||
|
||||
### Cache Entry
|
||||
|
||||
```csharp
|
||||
public sealed record ProvcacheEntry
|
||||
{
|
||||
public required string VeriKey { get; init; }
|
||||
public required DecisionDigest Decision { get; init; }
|
||||
public required string PolicyHash { get; init; }
|
||||
public required string SignerSetHash { get; init; }
|
||||
public required string FeedEpoch { get; init; }
|
||||
public required DateTimeOffset CreatedAt { get; init; }
|
||||
public required DateTimeOffset ExpiresAt { get; init; }
|
||||
public int HitCount { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Project Setup & Data Model)** | | | | | |
|
||||
| 0 | PROV-8200-000 | TODO | Design doc | Platform Guild | Create `docs/modules/provcache/README.md` with architecture overview. |
|
||||
| 1 | PROV-8200-001 | TODO | Task 0 | Platform Guild | Create `StellaOps.Provcache` project with dependencies on `StellaOps.Canonical.Json`, `StellaOps.Cryptography`, `StellaOps.Messaging.Transport.Valkey`. |
|
||||
| 2 | PROV-8200-002 | TODO | Task 1 | Platform Guild | Define `VeriKeyBuilder` with fluent API for composite hash construction. |
|
||||
| 3 | PROV-8200-003 | TODO | Task 1 | Platform Guild | Define `DecisionDigest` record with canonical JSON serialization. |
|
||||
| 4 | PROV-8200-004 | TODO | Task 1 | Platform Guild | Define `ProvcacheEntry` record for cache storage. |
|
||||
| 5 | PROV-8200-005 | TODO | Task 1 | Platform Guild | Define `ProvcacheOptions` configuration class. |
|
||||
| **Wave 1 (VeriKey Implementation)** | | | | | |
|
||||
| 6 | PROV-8200-006 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithSourceHash()` for artifact digest input. |
|
||||
| 7 | PROV-8200-007 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithSbomHash()` using SBOM canonicalization. |
|
||||
| 8 | PROV-8200-008 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithVexHashSet()` with sorted hash aggregation. |
|
||||
| 9 | PROV-8200-009 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithMergePolicyHash()` using PolicyBundle digest. |
|
||||
| 10 | PROV-8200-010 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithSignerSetHash()` with certificate chain hashing. |
|
||||
| 11 | PROV-8200-011 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.WithTimeWindow()` for epoch bucketing. |
|
||||
| 12 | PROV-8200-012 | TODO | Task 2 | Policy Guild | Implement `VeriKeyBuilder.Build()` producing final composite hash. |
|
||||
| 13 | PROV-8200-013 | TODO | Tasks 6-12 | QA Guild | Add determinism tests: same inputs → same VeriKey across runs. |
|
||||
| **Wave 2 (DecisionDigest & ProofRoot)** | | | | | |
|
||||
| 14 | PROV-8200-014 | TODO | Task 3 | Policy Guild | Implement `DecisionDigestBuilder` wrapping `EvaluationResult`. |
|
||||
| 15 | PROV-8200-015 | TODO | Task 14 | Policy Guild | Implement `VerdictHash` computation from sorted dispositions. |
|
||||
| 16 | PROV-8200-016 | TODO | Task 14 | Policy Guild | Implement `ProofRoot` Merkle computation from `ProofBundle`. |
|
||||
| 17 | PROV-8200-017 | TODO | Task 14 | Policy Guild | Implement `ReplaySeed` extraction from feed/rule identifiers. |
|
||||
| 18 | PROV-8200-018 | TODO | Task 14 | Policy Guild | Implement `TrustScore` computation based on evidence completeness. |
|
||||
| 19 | PROV-8200-019 | TODO | Tasks 14-18 | QA Guild | Add determinism tests: same evaluation → same DecisionDigest. |
|
||||
| **Wave 3 (Storage Layer)** | | | | | |
|
||||
| 20 | PROV-8200-020 | TODO | Task 4 | Platform Guild | Define Postgres schema `provcache.provcache_items` table. |
|
||||
| 21 | PROV-8200-021 | TODO | Task 20 | Platform Guild | Create EF Core entity `ProvcacheItemEntity`. |
|
||||
| 22 | PROV-8200-022 | TODO | Task 21 | Platform Guild | Implement `IProvcacheRepository` with CRUD operations. |
|
||||
| 23 | PROV-8200-023 | TODO | Task 22 | Platform Guild | Implement `PostgresProvcacheRepository`. |
|
||||
| 24 | PROV-8200-024 | TODO | Task 4 | Platform Guild | Implement `IProvcacheStore` interface for cache abstraction. |
|
||||
| 25 | PROV-8200-025 | TODO | Task 24 | Platform Guild | Implement `ValkeyProvcacheStore` with read-through pattern. |
|
||||
| 26 | PROV-8200-026 | TODO | Task 25 | Platform Guild | Implement write-behind queue for Postgres persistence. |
|
||||
| 27 | PROV-8200-027 | TODO | Tasks 23-26 | QA Guild | Add storage integration tests (Valkey + Postgres roundtrip). |
|
||||
| **Wave 4 (Service & API)** | | | | | |
|
||||
| 28 | PROV-8200-028 | TODO | Tasks 24-26 | Platform Guild | Implement `IProvcacheService` interface. |
|
||||
| 29 | PROV-8200-029 | TODO | Task 28 | Platform Guild | Implement `ProvcacheService` with Get/Set/Invalidate operations. |
|
||||
| 30 | PROV-8200-030 | TODO | Task 29 | Platform Guild | Implement `GET /v1/provcache/{veriKey}` endpoint. |
|
||||
| 31 | PROV-8200-031 | TODO | Task 29 | Platform Guild | Implement `POST /v1/provcache` (idempotent put) endpoint. |
|
||||
| 32 | PROV-8200-032 | TODO | Task 29 | Platform Guild | Implement `POST /v1/provcache/invalidate` endpoint (by key/pattern). |
|
||||
| 33 | PROV-8200-033 | TODO | Task 29 | Platform Guild | Implement cache metrics (hit rate, miss rate, latency). |
|
||||
| 34 | PROV-8200-034 | TODO | Tasks 30-33 | QA Guild | Add API integration tests with contract verification. |
|
||||
| **Wave 5 (Policy Engine Integration)** | | | | | |
|
||||
| 35 | PROV-8200-035 | TODO | Tasks 28-29 | Policy Guild | Add `IProvcacheService` to `PolicyEvaluator` constructor. |
|
||||
| 36 | PROV-8200-036 | TODO | Task 35 | Policy Guild | Implement cache lookup before TrustLattice evaluation. |
|
||||
| 37 | PROV-8200-037 | TODO | Task 35 | Policy Guild | Implement cache write after TrustLattice evaluation. |
|
||||
| 38 | PROV-8200-038 | TODO | Task 35 | Policy Guild | Add bypass option for cache (force re-evaluation). |
|
||||
| 39 | PROV-8200-039 | TODO | Task 35 | Policy Guild | Wire VeriKey construction from PolicyEvaluationContext. |
|
||||
| 40 | PROV-8200-040 | TODO | Tasks 35-39 | QA Guild | Add end-to-end tests: policy evaluation with warm/cold cache. |
|
||||
| **Wave 6 (Documentation & Telemetry)** | | | | | |
|
||||
| 41 | PROV-8200-041 | TODO | All prior | Docs Guild | Document Provcache configuration options. |
|
||||
| 42 | PROV-8200-042 | TODO | All prior | Docs Guild | Document VeriKey composition rules. |
|
||||
| 43 | PROV-8200-043 | TODO | All prior | Platform Guild | Add OpenTelemetry traces for cache operations. |
|
||||
| 44 | PROV-8200-044 | TODO | All prior | Platform Guild | Add Prometheus metrics for cache performance. |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### provcache.provcache_items
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.provcache_items (
|
||||
verikey TEXT PRIMARY KEY,
|
||||
digest_version TEXT NOT NULL DEFAULT 'v1',
|
||||
verdict_hash TEXT NOT NULL,
|
||||
proof_root TEXT NOT NULL,
|
||||
replay_seed JSONB NOT NULL,
|
||||
policy_hash TEXT NOT NULL,
|
||||
signer_set_hash TEXT NOT NULL,
|
||||
feed_epoch TEXT NOT NULL,
|
||||
trust_score INTEGER NOT NULL CHECK (trust_score >= 0 AND trust_score <= 100),
|
||||
hit_count BIGINT NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- Indexes for invalidation queries
|
||||
CONSTRAINT provcache_items_expires_check CHECK (expires_at > created_at)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_provcache_policy_hash ON provcache.provcache_items(policy_hash);
|
||||
CREATE INDEX idx_provcache_signer_set_hash ON provcache.provcache_items(signer_set_hash);
|
||||
CREATE INDEX idx_provcache_feed_epoch ON provcache.provcache_items(feed_epoch);
|
||||
CREATE INDEX idx_provcache_expires_at ON provcache.provcache_items(expires_at);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Specification
|
||||
|
||||
### GET /v1/provcache/{veriKey}
|
||||
|
||||
**Response 200 (Cache Hit):**
|
||||
```json
|
||||
{
|
||||
"veriKey": "sha256:abc123...",
|
||||
"decision": {
|
||||
"digestVersion": "v1",
|
||||
"verdictHash": "sha256:def456...",
|
||||
"proofRoot": "sha256:789abc...",
|
||||
"replaySeed": {
|
||||
"feedIds": ["cve-2024", "ghsa-2024"],
|
||||
"ruleIds": ["default-policy-v2"]
|
||||
},
|
||||
"trustScore": 85,
|
||||
"createdAt": "2025-12-24T12:00:00Z",
|
||||
"expiresAt": "2025-12-25T12:00:00Z"
|
||||
},
|
||||
"source": "valkey"
|
||||
}
|
||||
```
|
||||
|
||||
**Response 404 (Cache Miss):**
|
||||
```json
|
||||
{
|
||||
"veriKey": "sha256:abc123...",
|
||||
"found": false
|
||||
}
|
||||
```
|
||||
|
||||
### POST /v1/provcache
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"veriKey": "sha256:abc123...",
|
||||
"decision": { ... },
|
||||
"policyHash": "sha256:policy...",
|
||||
"signerSetHash": "sha256:signers...",
|
||||
"feedEpoch": "2024-W52",
|
||||
"ttlSeconds": 86400
|
||||
}
|
||||
```
|
||||
|
||||
**Response 201/200:**
|
||||
```json
|
||||
{
|
||||
"veriKey": "sha256:abc123...",
|
||||
"stored": true,
|
||||
"expiresAt": "2025-12-25T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### POST /v1/provcache/invalidate
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"by": "signer_set_hash",
|
||||
"value": "sha256:revoked-signer...",
|
||||
"reason": "key-revocation"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"invalidatedCount": 42,
|
||||
"by": "signer_set_hash",
|
||||
"value": "sha256:revoked-signer..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Options
|
||||
|
||||
```csharp
|
||||
public sealed class ProvcacheOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Default TTL for cache entries.
|
||||
/// </summary>
|
||||
public TimeSpan DefaultTtl { get; set; } = TimeSpan.FromHours(24);
|
||||
|
||||
/// <summary>
|
||||
/// Maximum TTL allowed for any entry.
|
||||
/// </summary>
|
||||
public TimeSpan MaxTtl { get; set; } = TimeSpan.FromDays(7);
|
||||
|
||||
/// <summary>
|
||||
/// Time window bucket size for VeriKey time component.
|
||||
/// </summary>
|
||||
public TimeSpan TimeWindowBucket { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
/// <summary>
|
||||
/// Valkey key prefix for cache entries.
|
||||
/// </summary>
|
||||
public string ValkeyKeyPrefix { get; set; } = "stellaops:prov:";
|
||||
|
||||
/// <summary>
|
||||
/// Enable write-behind to Postgres.
|
||||
/// </summary>
|
||||
public bool EnableWriteBehind { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Write-behind queue flush interval.
|
||||
/// </summary>
|
||||
public TimeSpan WriteBehindFlushInterval { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
/// Maximum items in write-behind queue before forced flush.
|
||||
/// </summary>
|
||||
public int WriteBehindMaxBatchSize { get; set; } = 100;
|
||||
|
||||
/// <summary>
|
||||
/// Enable cache bypass header (X-StellaOps-Cache-Bypass: true).
|
||||
/// </summary>
|
||||
public bool AllowCacheBypass { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Digest version for new entries.
|
||||
/// </summary>
|
||||
public string DigestVersion { get; set; } = "v1";
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-5 | Project setup, data models | Project compiles, types defined |
|
||||
| **Wave 1** | 6-13 | VeriKey implementation | Determinism tests pass |
|
||||
| **Wave 2** | 14-19 | DecisionDigest builder | Reproducibility tests pass |
|
||||
| **Wave 3** | 20-27 | Storage layer | Postgres + Valkey integration works |
|
||||
| **Wave 4** | 28-34 | Service & API | API contract tests pass |
|
||||
| **Wave 5** | 35-40 | Policy integration | Cache warm/cold scenarios work |
|
||||
| **Wave 6** | 41-44 | Docs & telemetry | Metrics visible in Grafana |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint |
|
||||
|-----------|-------------|----------------|
|
||||
| Signer revocation | Revocation events must trigger cache invalidation | 8200.0001.0002 |
|
||||
| Feed epochs | Concelier epoch changes must invalidate affected entries | 8200.0001.0002 |
|
||||
| Air-gap export | DecisionDigest must be exportable in offline bundles | 8200.0001.0002 |
|
||||
| UI badges | Provcache hit indicator requires frontend integration | 8200.0001.0003 |
|
||||
| Determinism | VeriKey must be stable across serialization roundtrips | Policy determinism tests |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| SHA256 for VeriKey (not Blake3) | FIPS/GOST compliance via `ICryptoHash` abstraction |
|
||||
| Valkey as primary, Postgres as durable | Fast reads (Valkey), crash recovery (Postgres) |
|
||||
| Time window bucketing | Prevents cache key explosion while enabling temporal grouping |
|
||||
| Signer set hash in VeriKey | Key rotation naturally invalidates without explicit purge |
|
||||
| Digest version prefix | Enables format evolution without cache invalidation |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| VeriKey collision | Incorrect cache hits | Use full SHA256; add collision detection | Platform Guild |
|
||||
| Write-behind data loss | Missing entries on crash | Configure Valkey persistence; bounded queue | Platform Guild |
|
||||
| Time window drift | Inconsistent keys | Use UTC epoch buckets; document clearly | Policy Guild |
|
||||
| Policy hash instability | Cache thrashing | Use canonical PolicyBundle serialization | Policy Guild |
|
||||
| Valkey unavailability | Cache bypass overhead | Graceful degradation to direct evaluation | Platform Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created based on Provcache advisory gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,112 @@
|
||||
# Sprint 8200.0001.0001 · Verdict ID Content-Addressing Fix
|
||||
|
||||
## Priority
|
||||
**P0 - CRITICAL** | Estimated Effort: 2 days
|
||||
|
||||
## Topic & Scope
|
||||
- Fix `DeltaVerdict.VerdictId` to use content-addressed hash instead of random GUID.
|
||||
- Implement content-addressed ID generation using existing `ContentAddressedIdGenerator` pattern.
|
||||
- Update all verdict creation sites to compute deterministic IDs.
|
||||
- Add regression tests to prevent future drift.
|
||||
- **Working directory:** `src/Policy/__Libraries/StellaOps.Policy/Deltas/`, `src/__Libraries/StellaOps.DeltaVerdict/`
|
||||
- **Evidence:** VerdictId is deterministic; identical inputs produce identical VerdictId; tests validate hash stability.
|
||||
|
||||
## Problem Statement
|
||||
Current implementation uses non-deterministic GUID:
|
||||
```csharp
|
||||
VerdictId = $"dv:{Guid.NewGuid():N}" // WRONG: Not reproducible
|
||||
```
|
||||
|
||||
Required implementation:
|
||||
```csharp
|
||||
VerdictId = ContentAddressedIdGenerator.ComputeVerdictId(
|
||||
deltaId, blockingDrivers, warningDrivers, appliedExceptions, gate);
|
||||
```
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: None (foundational fix)
|
||||
- Blocks: All other reproducibility sprints (8200.0001.*)
|
||||
- Safe to run in parallel with: None (must complete first)
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (Verdict Identity Formula section)
|
||||
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Identifiers/ContentAddressedIdGenerator.cs` (existing pattern)
|
||||
- Product Advisory: §3 Deterministic diffs & verdict identity
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Analysis** | | | | | |
|
||||
| 1 | VERDICT-8200-001 | TODO | None | Policy Guild | Audit all `DeltaVerdict` instantiation sites in codebase. Document each location. |
|
||||
| 2 | VERDICT-8200-002 | TODO | Task 1 | Policy Guild | Review `ContentAddressedIdGenerator` API and determine if extension needed for verdict payloads. |
|
||||
| **Implementation** | | | | | |
|
||||
| 3 | VERDICT-8200-003 | TODO | Task 2 | Policy Guild | Add `ComputeVerdictId()` method to `ContentAddressedIdGenerator` or create `VerdictIdGenerator` helper. |
|
||||
| 4 | VERDICT-8200-004 | TODO | Task 3 | Policy Guild | Update `DeltaVerdict` record to accept computed VerdictId; remove GUID generation. |
|
||||
| 5 | VERDICT-8200-005 | TODO | Task 4 | Policy Guild | Update `DeltaComputer.ComputeDelta()` to call new VerdictId generator. |
|
||||
| 6 | VERDICT-8200-006 | TODO | Task 4 | Policy Guild | Update all other verdict creation sites (Scanner.SmartDiff, Policy.Engine, etc.). |
|
||||
| **Testing** | | | | | |
|
||||
| 7 | VERDICT-8200-007 | TODO | Task 6 | Policy Guild | Add unit test: identical inputs → identical VerdictId (10 iterations). |
|
||||
| 8 | VERDICT-8200-008 | TODO | Task 6 | Policy Guild | Add unit test: different inputs → different VerdictId. |
|
||||
| 9 | VERDICT-8200-009 | TODO | Task 6 | Policy Guild | Add property test: VerdictId is deterministic across serialization round-trips. |
|
||||
| 10 | VERDICT-8200-010 | TODO | Task 9 | Policy Guild | Add integration test: VerdictId in attestation matches recomputed ID. |
|
||||
| **Documentation** | | | | | |
|
||||
| 11 | VERDICT-8200-011 | TODO | Task 10 | Policy Guild | Update `docs/reproducibility.md` with VerdictId computation details. |
|
||||
| 12 | VERDICT-8200-012 | TODO | Task 10 | Policy Guild | Add inline XML documentation to `VerdictIdGenerator` explaining the formula. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### VerdictId Computation
|
||||
```csharp
|
||||
public static class VerdictIdGenerator
|
||||
{
|
||||
public static string ComputeVerdictId(
|
||||
string deltaId,
|
||||
IReadOnlyList<DeltaDriver> blockingDrivers,
|
||||
IReadOnlyList<DeltaDriver> warningDrivers,
|
||||
IReadOnlyList<string> appliedExceptions,
|
||||
string gateLevel)
|
||||
{
|
||||
var payload = new VerdictIdPayload
|
||||
{
|
||||
DeltaId = deltaId,
|
||||
BlockingDrivers = blockingDrivers.OrderBy(d => d.FindingKey).ToList(),
|
||||
WarningDrivers = warningDrivers.OrderBy(d => d.FindingKey).ToList(),
|
||||
AppliedExceptions = appliedExceptions.Order().ToList(),
|
||||
GateLevel = gateLevel
|
||||
};
|
||||
|
||||
var canonicalJson = JsonSerializer.Serialize(payload, CanonicalJsonOptions);
|
||||
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(canonicalJson));
|
||||
return $"verdict:{Convert.ToHexString(hash).ToLowerInvariant()}";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Files to Modify
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/Policy/__Libraries/StellaOps.Policy/Deltas/DeltaVerdict.cs` | Remove GUID, accept computed ID |
|
||||
| `src/Policy/__Libraries/StellaOps.Policy/Deltas/DeltaComputer.cs` | Call VerdictIdGenerator |
|
||||
| `src/__Libraries/StellaOps.DeltaVerdict/Models/DeltaVerdict.cs` | Update if separate model exists |
|
||||
| `src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/` | Update verdict creation |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Statements/DeltaVerdictStatement.cs` | Verify ID propagation |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] `DeltaVerdict.VerdictId` is content-addressed (SHA-256 based)
|
||||
2. [ ] Identical inputs produce identical VerdictId across runs
|
||||
3. [ ] VerdictId prefix is `verdict:` followed by lowercase hex hash
|
||||
4. [ ] All existing tests pass (no regressions)
|
||||
5. [ ] New determinism tests added and passing
|
||||
6. [ ] Documentation updated
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Breaking change for stored verdicts | High | Add migration logic to handle old GUID format in lookups | Policy Guild |
|
||||
| Performance impact from hashing | Low | SHA-256 is fast; cache if needed | Policy Guild |
|
||||
| Serialization order changes hash | High | Use explicit `OrderBy` for all collections | Policy Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P0 priority - blocks all reproducibility work. | Project Mgmt |
|
||||
139
docs/implplan/SPRINT_8200_0001_0002_dsse_roundtrip_testing.md
Normal file
139
docs/implplan/SPRINT_8200_0001_0002_dsse_roundtrip_testing.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Sprint 8200.0001.0002 · DSSE Round-Trip Verification Testing
|
||||
|
||||
## Priority
|
||||
**P1 - HIGH** | Estimated Effort: 3 days
|
||||
|
||||
## Topic & Scope
|
||||
- Implement comprehensive DSSE round-trip tests: sign → verify → re-bundle → re-verify.
|
||||
- Validate that DSSE envelopes can be verified offline after bundling.
|
||||
- Ensure deterministic serialization across sign-verify cycles.
|
||||
- Test cosign compatibility for container image attestations.
|
||||
- **Working directory:** `src/Attestor/__Tests/`, `src/Signer/__Tests/`, `tests/integration/`
|
||||
- **Evidence:** All round-trip tests pass; DSSE envelopes verify correctly after re-bundling; cosign compatibility confirmed.
|
||||
|
||||
## Problem Statement
|
||||
Current state:
|
||||
- DSSE signing works (CryptoDsseSigner, HmacDsseSigner)
|
||||
- Basic sign→verify tests exist
|
||||
- No round-trip re-bundling tests
|
||||
- No verification after deserialization from bundle
|
||||
|
||||
Required:
|
||||
- Full round-trip: sign → serialize → deserialize → re-bundle → verify
|
||||
- Determinism proof: same payload produces same envelope bytes
|
||||
- Cosign interop: envelopes verifiable by `cosign verify-attestation`
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 8200.0001.0001 (VerdictId fix - for stable payloads)
|
||||
- Blocks: Sprint 8200.0001.0005 (Sigstore Bundle)
|
||||
- Safe to run in parallel with: Sprint 8200.0001.0003 (Schema validation)
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (DSSE Attestation Format section)
|
||||
- `src/Attestor/StellaOps.Attestor.Envelope/` (existing DSSE implementation)
|
||||
- `src/Signer/StellaOps.Signer.Infrastructure/Signing/CryptoDsseSigner.cs`
|
||||
- Sigstore DSSE spec: https://github.com/secure-systems-lab/dsse
|
||||
- Product Advisory: §2 DSSE attestations & bundle round-trips
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Test Infrastructure** | | | | | |
|
||||
| 1 | DSSE-8200-001 | TODO | None | Attestor Guild | Create `DsseRoundtripTestFixture` with key generation, signing, and verification helpers. |
|
||||
| 2 | DSSE-8200-002 | TODO | Task 1 | Attestor Guild | Add test helper to serialize DSSE to JSON, persist to file, reload, and deserialize. |
|
||||
| 3 | DSSE-8200-003 | TODO | Task 1 | Attestor Guild | Add test helper to create minimal Sigstore-compatible bundle wrapper. |
|
||||
| **Basic Round-Trip Tests** | | | | | |
|
||||
| 4 | DSSE-8200-004 | TODO | Task 2 | Attestor Guild | Add test: sign → serialize → deserialize → verify (happy path). |
|
||||
| 5 | DSSE-8200-005 | TODO | Task 4 | Attestor Guild | Add test: sign → verify → modify payload → verify fails. |
|
||||
| 6 | DSSE-8200-006 | TODO | Task 4 | Attestor Guild | Add test: sign → verify → modify signature → verify fails. |
|
||||
| **Re-Bundle Tests** | | | | | |
|
||||
| 7 | DSSE-8200-007 | TODO | Task 3 | Attestor Guild | Add test: sign → bundle → extract → re-bundle → verify (full round-trip). |
|
||||
| 8 | DSSE-8200-008 | TODO | Task 7 | Attestor Guild | Add test: sign → bundle → archive to tar.gz → extract → verify. |
|
||||
| 9 | DSSE-8200-009 | TODO | Task 7 | Attestor Guild | Add test: multi-signature envelope → bundle → extract → verify all signatures. |
|
||||
| **Determinism Tests** | | | | | |
|
||||
| 10 | DSSE-8200-010 | TODO | Task 4 | Attestor Guild | Add test: same payload signed twice → identical envelope bytes (deterministic key). |
|
||||
| 11 | DSSE-8200-011 | TODO | Task 10 | Attestor Guild | Add test: envelope serialization is canonical (key order, no whitespace variance). |
|
||||
| 12 | DSSE-8200-012 | TODO | Task 10 | Attestor Guild | Add property test: serialize → deserialize → serialize produces identical bytes. |
|
||||
| **Cosign Compatibility** | | | | | |
|
||||
| 13 | DSSE-8200-013 | TODO | Task 4 | Attestor Guild | Add integration test: envelope verifiable by `cosign verify-attestation` command. |
|
||||
| 14 | DSSE-8200-014 | TODO | Task 13 | Attestor Guild | Add test: OIDC-signed envelope verifiable with Fulcio certificate chain. |
|
||||
| 15 | DSSE-8200-015 | TODO | Task 13 | Attestor Guild | Add test: envelope with Rekor transparency entry verifiable offline. |
|
||||
| **Negative Tests** | | | | | |
|
||||
| 16 | DSSE-8200-016 | TODO | Task 4 | Attestor Guild | Add test: expired certificate → verify fails with clear error. |
|
||||
| 17 | DSSE-8200-017 | TODO | Task 4 | Attestor Guild | Add test: wrong key type → verify fails. |
|
||||
| 18 | DSSE-8200-018 | TODO | Task 4 | Attestor Guild | Add test: truncated envelope → parse fails gracefully. |
|
||||
| **Documentation** | | | | | |
|
||||
| 19 | DSSE-8200-019 | TODO | Task 15 | Attestor Guild | Document round-trip verification procedure in `docs/modules/attestor/`. |
|
||||
| 20 | DSSE-8200-020 | TODO | Task 15 | Attestor Guild | Add examples of cosign commands for manual verification. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### Round-Trip Test Structure
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task SignVerifyRebundleReverify_ProducesIdenticalResults()
|
||||
{
|
||||
// Arrange
|
||||
var payload = CreateTestInTotoStatement();
|
||||
var signer = CreateTestSigner();
|
||||
|
||||
// Act - Sign
|
||||
var envelope1 = await signer.SignAsync(payload);
|
||||
var verified1 = await signer.VerifyAsync(envelope1);
|
||||
|
||||
// Act - Bundle
|
||||
var bundle = BundleBuilder.Create(envelope1);
|
||||
var bundleBytes = bundle.Serialize();
|
||||
|
||||
// Act - Extract and Re-bundle
|
||||
var extractedBundle = BundleReader.Deserialize(bundleBytes);
|
||||
var extractedEnvelope = extractedBundle.DsseEnvelope;
|
||||
var rebundle = BundleBuilder.Create(extractedEnvelope);
|
||||
|
||||
// Act - Re-verify
|
||||
var verified2 = await signer.VerifyAsync(extractedEnvelope);
|
||||
|
||||
// Assert
|
||||
Assert.True(verified1.IsValid);
|
||||
Assert.True(verified2.IsValid);
|
||||
Assert.Equal(envelope1.PayloadHash, extractedEnvelope.PayloadHash);
|
||||
Assert.Equal(bundleBytes, rebundle.Serialize()); // Byte-for-byte identical
|
||||
}
|
||||
```
|
||||
|
||||
### Test Categories
|
||||
| Category | Tests | Purpose |
|
||||
|----------|-------|---------|
|
||||
| Basic Round-Trip | 4-6 | Verify sign/verify cycle works |
|
||||
| Re-Bundle | 7-9 | Verify bundling doesn't corrupt |
|
||||
| Determinism | 10-12 | Verify reproducibility |
|
||||
| Cosign Compat | 13-15 | Verify industry tooling works |
|
||||
| Negative | 16-18 | Verify error handling |
|
||||
|
||||
## Files to Create/Modify
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `src/Attestor/__Tests/StellaOps.Attestor.Envelope.Tests/DsseRoundtripTests.cs` | Create |
|
||||
| `src/Attestor/__Tests/StellaOps.Attestor.Envelope.Tests/DsseRoundtripTestFixture.cs` | Create |
|
||||
| `tests/integration/StellaOps.Integration.Attestor/DsseCosignCompatibilityTests.cs` | Create |
|
||||
| `tests/integration/StellaOps.Integration.Attestor/DsseRebundleTests.cs` | Create |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] Sign → verify → re-bundle → re-verify cycle passes
|
||||
2. [ ] Deterministic serialization verified (identical bytes)
|
||||
3. [ ] Cosign compatibility confirmed (external tool verification)
|
||||
4. [ ] Multi-signature envelopes work correctly
|
||||
5. [ ] Negative cases handled gracefully
|
||||
6. [ ] Documentation updated with verification examples
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Cosign version incompatibility | Medium | Pin cosign version in CI; test multiple versions | Attestor Guild |
|
||||
| Keyless signing requires network | Medium | Use mocked OIDC provider for offline tests | Attestor Guild |
|
||||
| Rekor dependency for transparency | Medium | Support offline verification with cached receipts | Attestor Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P1 priority - validates offline replay. | Project Mgmt |
|
||||
@@ -0,0 +1,390 @@
|
||||
# Sprint 8200.0001.0002 · Provcache Invalidation & Air-Gap
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Extend the Provcache layer with **security-critical invalidation mechanisms** and **air-gap optimization** for offline/disconnected environments. This sprint delivers:
|
||||
|
||||
1. **Signer-Aware Invalidation**: Automatic cache purge when signers are revoked via Authority.
|
||||
2. **Feed Epoch Binding**: Cache invalidation when Concelier advisory feeds update.
|
||||
3. **Evidence Chunk Paging**: Chunked evidence storage for minimal air-gap bundle sizes.
|
||||
4. **Minimal Proof Export**: CLI commands for exporting DecisionDigest + ProofRoot without full evidence.
|
||||
5. **Lazy Evidence Pull**: On-demand evidence retrieval for air-gapped auditors.
|
||||
|
||||
**Working directory:** `src/__Libraries/StellaOps.Provcache/` (extension), `src/AirGap/` (integration), `src/Cli/StellaOps.Cli/Commands/` (new commands).
|
||||
|
||||
**Evidence:** Signer revocation triggers cache invalidation within seconds; air-gap bundle size reduced by >50% vs full SBOM/VEX payloads; CLI export/import works end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0001.0001 (Provcache Core Backend), Authority `IKeyRotationService`, Concelier feed epochs.
|
||||
- **Recommended to land before:** Sprint 8200.0001.0003 (UX & Observability).
|
||||
- **Safe to run in parallel with:** Other AirGap sprints as long as bundle format is stable.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/provcache/README.md` (from Sprint 8200.0001.0001)
|
||||
- `docs/modules/authority/README.md`
|
||||
- `docs/modules/concelier/README.md`
|
||||
- `docs/24_OFFLINE_KIT.md`
|
||||
- `src/Authority/__Libraries/StellaOps.Signer.KeyManagement/`
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Signer Set Hash Index
|
||||
|
||||
The cache maintains an index by `signer_set_hash` to enable fast revocation fan-out:
|
||||
|
||||
```
|
||||
signer_set_hash → [veriKey1, veriKey2, ...]
|
||||
```
|
||||
|
||||
When Authority revokes a signer:
|
||||
1. Authority publishes `SignerRevokedEvent` to messaging bus
|
||||
2. Provcache subscribes and queries index
|
||||
3. All entries with matching signer set are invalidated
|
||||
|
||||
### Feed Epoch Binding
|
||||
|
||||
Each cache entry stores the `feed_epoch` (e.g., `cve:2024-12-24T12:00Z`, `ghsa:v2024.52`):
|
||||
|
||||
```
|
||||
feed_epoch → [veriKey1, veriKey2, ...]
|
||||
```
|
||||
|
||||
When Concelier publishes a new epoch:
|
||||
1. Concelier emits `FeedEpochAdvancedEvent`
|
||||
2. Provcache invalidates entries bound to older epochs
|
||||
|
||||
### Evidence Chunk Storage
|
||||
|
||||
Large evidence (full SBOM, VEX documents, call graphs) is stored in chunks:
|
||||
|
||||
```sql
|
||||
provcache.prov_evidence_chunks (
|
||||
chunk_id, -- UUID
|
||||
proof_root, -- Links to provcache_items.proof_root
|
||||
chunk_index, -- 0, 1, 2, ...
|
||||
chunk_hash, -- Individual chunk hash
|
||||
blob -- Binary/JSONB content
|
||||
)
|
||||
```
|
||||
|
||||
### Minimal Proof Bundle
|
||||
|
||||
For air-gap export, the minimal bundle contains:
|
||||
- `DecisionDigest` (verdict hash, proof root, trust score)
|
||||
- `ProofRoot` (Merkle root for verification)
|
||||
- `ChunkManifest` (list of chunk hashes for lazy fetch)
|
||||
- Optionally: first N chunks (configurable density)
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Signer Revocation Fan-Out)** | | | | | |
|
||||
| 0 | PROV-8200-100 | TODO | Sprint 0001 | Authority Guild | Define `SignerRevokedEvent` message contract. |
|
||||
| 1 | PROV-8200-101 | TODO | Task 0 | Authority Guild | Publish `SignerRevokedEvent` from `KeyRotationService.RevokeKey()`. |
|
||||
| 2 | PROV-8200-102 | TODO | Task 0 | Platform Guild | Create `signer_set_hash` index on `provcache_items`. |
|
||||
| 3 | PROV-8200-103 | TODO | Task 2 | Platform Guild | Implement `IProvcacheInvalidator` interface. |
|
||||
| 4 | PROV-8200-104 | TODO | Task 3 | Platform Guild | Implement `SignerSetInvalidator` handling revocation events. |
|
||||
| 5 | PROV-8200-105 | TODO | Task 4 | Platform Guild | Subscribe `SignerSetInvalidator` to messaging bus. |
|
||||
| 6 | PROV-8200-106 | TODO | Task 5 | QA Guild | Add integration tests: revoke signer → cache entries invalidated. |
|
||||
| **Wave 1 (Feed Epoch Binding)** | | | | | |
|
||||
| 7 | PROV-8200-107 | TODO | Sprint 0001 | Concelier Guild | Define `FeedEpochAdvancedEvent` message contract. |
|
||||
| 8 | PROV-8200-108 | TODO | Task 7 | Concelier Guild | Publish `FeedEpochAdvancedEvent` from merge reconcile job. |
|
||||
| 9 | PROV-8200-109 | TODO | Task 7 | Platform Guild | Create `feed_epoch` index on `provcache_items`. |
|
||||
| 10 | PROV-8200-110 | TODO | Task 9 | Platform Guild | Implement `FeedEpochInvalidator` handling epoch events. |
|
||||
| 11 | PROV-8200-111 | TODO | Task 10 | Platform Guild | Implement epoch comparison logic (newer epoch invalidates older). |
|
||||
| 12 | PROV-8200-112 | TODO | Task 11 | Platform Guild | Subscribe `FeedEpochInvalidator` to messaging bus. |
|
||||
| 13 | PROV-8200-113 | TODO | Task 12 | QA Guild | Add integration tests: feed epoch advance → cache entries invalidated. |
|
||||
| **Wave 2 (Evidence Chunk Storage)** | | | | | |
|
||||
| 14 | PROV-8200-114 | TODO | Sprint 0001 | Platform Guild | Define `provcache.prov_evidence_chunks` Postgres schema. |
|
||||
| 15 | PROV-8200-115 | TODO | Task 14 | Platform Guild | Implement `EvidenceChunkEntity` EF Core entity. |
|
||||
| 16 | PROV-8200-116 | TODO | Task 15 | Platform Guild | Implement `IEvidenceChunkRepository` interface. |
|
||||
| 17 | PROV-8200-117 | TODO | Task 16 | Platform Guild | Implement `PostgresEvidenceChunkRepository`. |
|
||||
| 18 | PROV-8200-118 | TODO | Task 17 | Platform Guild | Implement `IEvidenceChunker` for splitting large evidence. |
|
||||
| 19 | PROV-8200-119 | TODO | Task 18 | Platform Guild | Implement chunk size configuration (default 64KB). |
|
||||
| 20 | PROV-8200-120 | TODO | Task 18 | Platform Guild | Implement `ChunkManifest` record with Merkle verification. |
|
||||
| 21 | PROV-8200-121 | TODO | Task 20 | QA Guild | Add chunking tests: large evidence → chunks → reassembly. |
|
||||
| **Wave 3 (Evidence Paging API)** | | | | | |
|
||||
| 22 | PROV-8200-122 | TODO | Task 17 | Platform Guild | Implement `GET /v1/proofs/{proofRoot}` endpoint. |
|
||||
| 23 | PROV-8200-123 | TODO | Task 22 | Platform Guild | Implement pagination (offset/limit or cursor-based). |
|
||||
| 24 | PROV-8200-124 | TODO | Task 22 | Platform Guild | Implement chunk streaming for large responses. |
|
||||
| 25 | PROV-8200-125 | TODO | Task 22 | Platform Guild | Implement Merkle proof verification for individual chunks. |
|
||||
| 26 | PROV-8200-126 | TODO | Tasks 22-25 | QA Guild | Add API tests for paged evidence retrieval. |
|
||||
| **Wave 4 (Minimal Proof Export)** | | | | | |
|
||||
| 27 | PROV-8200-127 | TODO | Tasks 20-21 | AirGap Guild | Define `MinimalProofBundle` export format. |
|
||||
| 28 | PROV-8200-128 | TODO | Task 27 | AirGap Guild | Implement `IMinimalProofExporter` interface. |
|
||||
| 29 | PROV-8200-129 | TODO | Task 28 | AirGap Guild | Implement `MinimalProofExporter` with density levels. |
|
||||
| 30 | PROV-8200-130 | TODO | Task 29 | AirGap Guild | Implement density level: `lite` (digest + root only). |
|
||||
| 31 | PROV-8200-131 | TODO | Task 29 | AirGap Guild | Implement density level: `standard` (+ first N chunks). |
|
||||
| 32 | PROV-8200-132 | TODO | Task 29 | AirGap Guild | Implement density level: `strict` (+ all chunks). |
|
||||
| 33 | PROV-8200-133 | TODO | Task 29 | AirGap Guild | Implement DSSE signing of minimal proof bundle. |
|
||||
| 34 | PROV-8200-134 | TODO | Tasks 30-33 | QA Guild | Add export tests for all density levels. |
|
||||
| **Wave 5 (CLI Commands)** | | | | | |
|
||||
| 35 | PROV-8200-135 | TODO | Task 29 | CLI Guild | Implement `stella prov export` command. |
|
||||
| 36 | PROV-8200-136 | TODO | Task 35 | CLI Guild | Add `--density` option (`lite`, `standard`, `strict`). |
|
||||
| 37 | PROV-8200-137 | TODO | Task 35 | CLI Guild | Add `--output` option for file path. |
|
||||
| 38 | PROV-8200-138 | TODO | Task 35 | CLI Guild | Add `--sign` option with signer selection. |
|
||||
| 39 | PROV-8200-139 | TODO | Task 27 | CLI Guild | Implement `stella prov import` command. |
|
||||
| 40 | PROV-8200-140 | TODO | Task 39 | CLI Guild | Implement Merkle root verification on import. |
|
||||
| 41 | PROV-8200-141 | TODO | Task 39 | CLI Guild | Implement signature verification on import. |
|
||||
| 42 | PROV-8200-142 | TODO | Task 39 | CLI Guild | Add `--lazy-fetch` option for chunk retrieval. |
|
||||
| 43 | PROV-8200-143 | TODO | Tasks 35-42 | QA Guild | Add CLI e2e tests: export → transfer → import. |
|
||||
| **Wave 6 (Lazy Evidence Pull)** | | | | | |
|
||||
| 44 | PROV-8200-144 | TODO | Tasks 22, 42 | AirGap Guild | Implement `ILazyEvidenceFetcher` interface. |
|
||||
| 45 | PROV-8200-145 | TODO | Task 44 | AirGap Guild | Implement HTTP-based chunk fetcher for connected mode. |
|
||||
| 46 | PROV-8200-146 | TODO | Task 44 | AirGap Guild | Implement file-based chunk fetcher for sneakernet mode. |
|
||||
| 47 | PROV-8200-147 | TODO | Task 44 | AirGap Guild | Implement chunk verification during lazy fetch. |
|
||||
| 48 | PROV-8200-148 | TODO | Tasks 44-47 | QA Guild | Add lazy fetch tests (connected + disconnected). |
|
||||
| **Wave 7 (Revocation Index Table)** | | | | | |
|
||||
| 49 | PROV-8200-149 | TODO | Tasks 0-6 | Platform Guild | Define `provcache.prov_revocations` table. |
|
||||
| 50 | PROV-8200-150 | TODO | Task 49 | Platform Guild | Implement revocation ledger for audit trail. |
|
||||
| 51 | PROV-8200-151 | TODO | Task 50 | Platform Guild | Implement revocation replay for catch-up scenarios. |
|
||||
| 52 | PROV-8200-152 | TODO | Tasks 49-51 | QA Guild | Add revocation ledger tests. |
|
||||
| **Wave 8 (Documentation)** | | | | | |
|
||||
| 53 | PROV-8200-153 | TODO | All prior | Docs Guild | Document invalidation mechanisms. |
|
||||
| 54 | PROV-8200-154 | TODO | All prior | Docs Guild | Document air-gap export/import workflow. |
|
||||
| 55 | PROV-8200-155 | TODO | All prior | Docs Guild | Document evidence density levels. |
|
||||
| 56 | PROV-8200-156 | TODO | All prior | Docs Guild | Update `docs/24_OFFLINE_KIT.md` with Provcache integration. |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema Extensions
|
||||
|
||||
### provcache.prov_evidence_chunks
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_evidence_chunks (
|
||||
chunk_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
proof_root TEXT NOT NULL,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_hash TEXT NOT NULL,
|
||||
blob BYTEA NOT NULL,
|
||||
blob_size INTEGER NOT NULL,
|
||||
content_type TEXT NOT NULL DEFAULT 'application/octet-stream',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT prov_evidence_chunks_proof_root_fk
|
||||
FOREIGN KEY (proof_root) REFERENCES provcache.provcache_items(proof_root)
|
||||
ON DELETE CASCADE,
|
||||
CONSTRAINT prov_evidence_chunks_unique
|
||||
UNIQUE (proof_root, chunk_index)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_evidence_chunks_proof_root ON provcache.prov_evidence_chunks(proof_root);
|
||||
```
|
||||
|
||||
### provcache.prov_revocations
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_revocations (
|
||||
revocation_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
revocation_type TEXT NOT NULL, -- 'signer', 'feed_epoch', 'policy'
|
||||
target_hash TEXT NOT NULL, -- signer_set_hash, feed_epoch, or policy_hash
|
||||
reason TEXT,
|
||||
actor TEXT,
|
||||
entries_affected BIGINT NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT prov_revocations_type_check
|
||||
CHECK (revocation_type IN ('signer', 'feed_epoch', 'policy'))
|
||||
);
|
||||
|
||||
CREATE INDEX idx_prov_revocations_target ON provcache.prov_revocations(revocation_type, target_hash);
|
||||
CREATE INDEX idx_prov_revocations_created ON provcache.prov_revocations(created_at);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Additions
|
||||
|
||||
### GET /v1/proofs/{proofRoot}
|
||||
|
||||
**Response 200:**
|
||||
```json
|
||||
{
|
||||
"proofRoot": "sha256:789abc...",
|
||||
"chunkCount": 5,
|
||||
"totalSize": 327680,
|
||||
"chunks": [
|
||||
{
|
||||
"index": 0,
|
||||
"hash": "sha256:chunk0...",
|
||||
"size": 65536
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"hash": "sha256:chunk1...",
|
||||
"size": 65536
|
||||
}
|
||||
],
|
||||
"pagination": {
|
||||
"offset": 0,
|
||||
"limit": 10,
|
||||
"total": 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GET /v1/proofs/{proofRoot}/chunks/{index}
|
||||
|
||||
**Response 200:**
|
||||
Binary chunk content with headers:
|
||||
- `Content-Type: application/octet-stream`
|
||||
- `X-Chunk-Hash: sha256:chunk0...`
|
||||
- `X-Chunk-Index: 0`
|
||||
- `X-Total-Chunks: 5`
|
||||
|
||||
---
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### stella prov export
|
||||
|
||||
```bash
|
||||
# Export minimal proof (digest only)
|
||||
stella prov export --verikey sha256:abc123 --density lite --output proof.json
|
||||
|
||||
# Export with first 3 chunks
|
||||
stella prov export --verikey sha256:abc123 --density standard --chunks 3 --output proof.bundle
|
||||
|
||||
# Export full evidence (all chunks)
|
||||
stella prov export --verikey sha256:abc123 --density strict --output proof-full.bundle
|
||||
|
||||
# Sign the export
|
||||
stella prov export --verikey sha256:abc123 --density standard --sign --output proof-signed.bundle
|
||||
```
|
||||
|
||||
### stella prov import
|
||||
|
||||
```bash
|
||||
# Import and verify
|
||||
stella prov import --input proof.bundle
|
||||
|
||||
# Import with lazy chunk fetch from remote
|
||||
stella prov import --input proof-lite.json --lazy-fetch --backend https://stellaops.example.com
|
||||
|
||||
# Import with offline chunk directory
|
||||
stella prov import --input proof-lite.json --chunks-dir /mnt/usb/chunks/
|
||||
```
|
||||
|
||||
### stella prov verify
|
||||
|
||||
```bash
|
||||
# Verify proof without importing
|
||||
stella prov verify --input proof.bundle
|
||||
|
||||
# Verify signature
|
||||
stella prov verify --input proof-signed.bundle --signer-cert ca.pem
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Message Contracts
|
||||
|
||||
### SignerRevokedEvent
|
||||
|
||||
```csharp
|
||||
public sealed record SignerRevokedEvent
|
||||
{
|
||||
public required string SignerId { get; init; }
|
||||
public required string SignerSetHash { get; init; }
|
||||
public required string CertificateSerial { get; init; }
|
||||
public required string Reason { get; init; }
|
||||
public required string Actor { get; init; }
|
||||
public required DateTimeOffset RevokedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### FeedEpochAdvancedEvent
|
||||
|
||||
```csharp
|
||||
public sealed record FeedEpochAdvancedEvent
|
||||
{
|
||||
public required string FeedId { get; init; } // "cve", "ghsa", "nvd"
|
||||
public required string PreviousEpoch { get; init; } // "2024-W51"
|
||||
public required string CurrentEpoch { get; init; } // "2024-W52"
|
||||
public required int AdvisoriesAdded { get; init; }
|
||||
public required int AdvisoriesModified { get; init; }
|
||||
public required DateTimeOffset AdvancedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evidence Density Levels
|
||||
|
||||
| Level | Contents | Typical Size | Use Case |
|
||||
|-------|----------|--------------|----------|
|
||||
| `lite` | DecisionDigest + ProofRoot + ChunkManifest | ~2 KB | Quick verification, high-trust networks |
|
||||
| `standard` | Above + first 3 chunks | ~200 KB | Normal air-gap, auditor preview |
|
||||
| `strict` | Above + all chunks | Variable | Full audit, compliance evidence |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-6 | Signer revocation | Revocation events invalidate cache |
|
||||
| **Wave 1** | 7-13 | Feed epoch binding | Epoch advance invalidates cache |
|
||||
| **Wave 2** | 14-21 | Evidence chunking | Large evidence splits/reassembles |
|
||||
| **Wave 3** | 22-26 | Proof paging API | Paged chunk retrieval works |
|
||||
| **Wave 4** | 27-34 | Minimal export | Density levels export correctly |
|
||||
| **Wave 5** | 35-43 | CLI commands | Export/import/verify work e2e |
|
||||
| **Wave 6** | 44-48 | Lazy fetch | Connected + disconnected modes |
|
||||
| **Wave 7** | 49-52 | Revocation ledger | Audit trail for invalidations |
|
||||
| **Wave 8** | 53-56 | Documentation | All workflows documented |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint |
|
||||
|-----------|-------------|----------------|
|
||||
| Authority key revocation | `KeyRotationService.RevokeKey()` must emit event | Authority module |
|
||||
| Concelier epoch advance | Merge reconcile job must emit event | Concelier module |
|
||||
| DSSE signing | Export signing uses Signer infrastructure | Signer module |
|
||||
| Bundle format | Must be compatible with existing OfflineKit | AirGap module |
|
||||
| Chunk LRU | Evidence chunks subject to retention policy | Evidence module |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| 64KB default chunk size | Balance between HTTP efficiency and granularity |
|
||||
| Lazy fetch via manifest | Enables minimal initial transfer, on-demand detail |
|
||||
| Three density levels | Clear trade-off between size and completeness |
|
||||
| Revocation ledger | Audit trail for compliance, replay for catch-up |
|
||||
| Epoch string format | ISO week or timestamp for deterministic comparison |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Revocation event loss | Stale cache entries | Durable messaging; revocation ledger replay | Platform Guild |
|
||||
| Chunk verification failure | Data corruption | Re-fetch from source; multiple chunk sources | AirGap Guild |
|
||||
| Large evidence OOM | Service crash | Streaming chunk processing | Platform Guild |
|
||||
| Epoch race conditions | Inconsistent invalidation | Ordered event processing; epoch comparison | Concelier Guild |
|
||||
| CLI export interruption | Partial bundle | Atomic writes; resume support | CLI Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Provcache advisory gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,451 @@
|
||||
# Sprint 8200.0001.0003 · Provcache UX & Observability
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Deliver **user-facing visibility** and **operational observability** for the Provcache layer. This sprint enables users and operators to understand provenance caching behavior and trust decisions. This sprint delivers:
|
||||
|
||||
1. **UI "Provenance-Cached" Badge**: Visual indicator in Timeline/Findings when decisions are cached.
|
||||
2. **Proof Tree Viewer**: Interactive visualization of the evidence tree behind a decision.
|
||||
3. **Input Manifest Display**: Show exact inputs (SBOM, VEX, policy) that formed a cached decision.
|
||||
4. **Cache Metrics Dashboard**: Grafana dashboards for cache performance monitoring.
|
||||
5. **Trust Score Visualization**: Display trust scores with breakdown by evidence type.
|
||||
6. **OCI Attestation Attachment**: Emit DecisionDigest as OCI-attached attestation on images.
|
||||
|
||||
**Working directory:** `src/Web/StellaOps.Web/` (Angular frontend), `src/__Libraries/StellaOps.Provcache/` (metrics), `src/ExportCenter/` (OCI attachment).
|
||||
|
||||
**Evidence:** UI badge visible on cached decisions; proof tree renders correctly; Grafana dashboards operational; OCI attestations verifiable with `cosign`.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0001.0001 (Provcache Core Backend), Sprint 8200.0001.0002 (Invalidation & Air-Gap).
|
||||
- **Frontend depends on:** Angular v17 patterns, existing Findings/Timeline components.
|
||||
- **Recommended to land after:** Core backend and invalidation are stable.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/provcache/README.md` (from Sprint 8200.0001.0001)
|
||||
- `docs/modules/findings/README.md`
|
||||
- `src/Web/StellaOps.Web/README.md`
|
||||
- Grafana dashboard patterns in `deploy/grafana/`
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Provenance Badge States
|
||||
|
||||
| State | Icon | Tooltip | Meaning |
|
||||
|-------|------|---------|---------|
|
||||
| `cached` | ⚡ | "Provenance-cached" | Decision from cache, fast path |
|
||||
| `computed` | 🔄 | "Freshly computed" | Decision computed this request |
|
||||
| `stale` | ⏳ | "Stale - recomputing" | Cache expired, recomputation in progress |
|
||||
| `unknown` | ❓ | "Unknown provenance" | Legacy data, no cache metadata |
|
||||
|
||||
### Trust Score Breakdown
|
||||
|
||||
The trust score (0-100) is composed from:
|
||||
|
||||
| Component | Weight | Source |
|
||||
|-----------|--------|--------|
|
||||
| Reachability evidence | 25% | Call graph / static analysis |
|
||||
| SBOM completeness | 20% | Package coverage, license data |
|
||||
| VEX statement coverage | 20% | Vendor statements, OpenVEX |
|
||||
| Policy freshness | 15% | Last policy update timestamp |
|
||||
| Signer trust | 20% | Signer reputation, key age |
|
||||
|
||||
### Proof Tree Structure
|
||||
|
||||
```
|
||||
DecisionDigest
|
||||
├── VeriKey
|
||||
│ ├── Source Hash (artifact)
|
||||
│ ├── SBOM Hash
|
||||
│ ├── VEX Hash Set
|
||||
│ ├── Policy Hash
|
||||
│ ├── Signer Set Hash
|
||||
│ └── Time Window
|
||||
├── Verdicts
|
||||
│ ├── CVE-2024-1234 → MITIGATED
|
||||
│ ├── CVE-2024-5678 → AFFECTED
|
||||
│ └── ...
|
||||
├── Evidence Tree (Merkle)
|
||||
│ ├── Reachability [chunk 0-2]
|
||||
│ ├── VEX Statements [chunk 3-5]
|
||||
│ └── Policy Rules [chunk 6]
|
||||
└── Metadata
|
||||
├── Trust Score: 85
|
||||
├── Created: 2025-12-24T12:00:00Z
|
||||
└── Expires: 2025-12-25T12:00:00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (API Extensions)** | | | | | |
|
||||
| 0 | PROV-8200-200 | TODO | Sprint 0001 | Platform Guild | Add `cacheSource` field to policy evaluation response. |
|
||||
| 1 | PROV-8200-201 | TODO | Task 0 | Platform Guild | Add `trustScoreBreakdown` to DecisionDigest response. |
|
||||
| 2 | PROV-8200-202 | TODO | Task 0 | Platform Guild | Add `inputManifest` endpoint for VeriKey components. |
|
||||
| 3 | PROV-8200-203 | TODO | Tasks 0-2 | QA Guild | Add API contract tests for new response fields. |
|
||||
| **Wave 1 (Provenance Badge Component)** | | | | | |
|
||||
| 4 | PROV-8200-204 | TODO | Tasks 0-2 | Frontend Guild | Create `ProvenanceBadgeComponent` Angular component. |
|
||||
| 5 | PROV-8200-205 | TODO | Task 4 | Frontend Guild | Implement badge state icons (cached/computed/stale/unknown). |
|
||||
| 6 | PROV-8200-206 | TODO | Task 4 | Frontend Guild | Implement tooltip with cache details. |
|
||||
| 7 | PROV-8200-207 | TODO | Task 4 | Frontend Guild | Add badge to `FindingRowComponent`. |
|
||||
| 8 | PROV-8200-208 | TODO | Task 4 | Frontend Guild | Add badge to `TimelineEventComponent`. |
|
||||
| 9 | PROV-8200-209 | TODO | Tasks 4-8 | QA Guild | Add Storybook stories for all badge states. |
|
||||
| **Wave 2 (Trust Score Display)** | | | | | |
|
||||
| 10 | PROV-8200-210 | TODO | Task 1 | Frontend Guild | Create `TrustScoreComponent` Angular component. |
|
||||
| 11 | PROV-8200-211 | TODO | Task 10 | Frontend Guild | Implement donut chart visualization. |
|
||||
| 12 | PROV-8200-212 | TODO | Task 10 | Frontend Guild | Implement breakdown tooltip with component percentages. |
|
||||
| 13 | PROV-8200-213 | TODO | Task 10 | Frontend Guild | Add color coding (green/yellow/red thresholds). |
|
||||
| 14 | PROV-8200-214 | TODO | Task 10 | Frontend Guild | Integrate into FindingDetailComponent. |
|
||||
| 15 | PROV-8200-215 | TODO | Tasks 10-14 | QA Guild | Add Storybook stories for score ranges. |
|
||||
| **Wave 3 (Proof Tree Viewer)** | | | | | |
|
||||
| 16 | PROV-8200-216 | TODO | Sprint 0002 | Frontend Guild | Create `ProofTreeComponent` Angular component. |
|
||||
| 17 | PROV-8200-217 | TODO | Task 16 | Frontend Guild | Implement collapsible tree visualization. |
|
||||
| 18 | PROV-8200-218 | TODO | Task 16 | Frontend Guild | Implement VeriKey component display. |
|
||||
| 19 | PROV-8200-219 | TODO | Task 16 | Frontend Guild | Implement verdict list with status colors. |
|
||||
| 20 | PROV-8200-220 | TODO | Task 16 | Frontend Guild | Implement Merkle tree visualization with chunk links. |
|
||||
| 21 | PROV-8200-221 | TODO | Task 16 | Frontend Guild | Implement chunk download on click (lazy fetch). |
|
||||
| 22 | PROV-8200-222 | TODO | Task 16 | Frontend Guild | Add "Verify Proof" button with Merkle verification. |
|
||||
| 23 | PROV-8200-223 | TODO | Tasks 16-22 | QA Guild | Add Storybook stories and interaction tests. |
|
||||
| **Wave 4 (Input Manifest Panel)** | | | | | |
|
||||
| 24 | PROV-8200-224 | TODO | Task 2 | Frontend Guild | Create `InputManifestComponent` Angular component. |
|
||||
| 25 | PROV-8200-225 | TODO | Task 24 | Frontend Guild | Display source artifact info (image, digest). |
|
||||
| 26 | PROV-8200-226 | TODO | Task 24 | Frontend Guild | Display SBOM info (format, package count). |
|
||||
| 27 | PROV-8200-227 | TODO | Task 24 | Frontend Guild | Display VEX statement summary (count, sources). |
|
||||
| 28 | PROV-8200-228 | TODO | Task 24 | Frontend Guild | Display policy info (name, version, hash). |
|
||||
| 29 | PROV-8200-229 | TODO | Task 24 | Frontend Guild | Display signer info (certificates, expiry). |
|
||||
| 30 | PROV-8200-230 | TODO | Task 24 | Frontend Guild | Integrate into FindingDetailComponent via tab. |
|
||||
| 31 | PROV-8200-231 | TODO | Tasks 24-30 | QA Guild | Add Storybook stories and snapshot tests. |
|
||||
| **Wave 5 (Metrics & Telemetry)** | | | | | |
|
||||
| 32 | PROV-8200-232 | TODO | Sprint 0001 | Platform Guild | Add Prometheus counter: `provcache_requests_total`. |
|
||||
| 33 | PROV-8200-233 | TODO | Task 32 | Platform Guild | Add Prometheus counter: `provcache_hits_total`. |
|
||||
| 34 | PROV-8200-234 | TODO | Task 32 | Platform Guild | Add Prometheus counter: `provcache_misses_total`. |
|
||||
| 35 | PROV-8200-235 | TODO | Task 32 | Platform Guild | Add Prometheus histogram: `provcache_latency_seconds`. |
|
||||
| 36 | PROV-8200-236 | TODO | Task 32 | Platform Guild | Add Prometheus gauge: `provcache_items_count`. |
|
||||
| 37 | PROV-8200-237 | TODO | Task 32 | Platform Guild | Add Prometheus counter: `provcache_invalidations_total`. |
|
||||
| 38 | PROV-8200-238 | TODO | Task 32 | Platform Guild | Add labels: `source` (valkey/postgres), `reason` (hit/miss/expired). |
|
||||
| 39 | PROV-8200-239 | TODO | Tasks 32-38 | QA Guild | Add metrics emission tests. |
|
||||
| **Wave 6 (Grafana Dashboards)** | | | | | |
|
||||
| 40 | PROV-8200-240 | TODO | Tasks 32-38 | DevOps Guild | Create `provcache-overview.json` dashboard. |
|
||||
| 41 | PROV-8200-241 | TODO | Task 40 | DevOps Guild | Add cache hit rate panel (percentage over time). |
|
||||
| 42 | PROV-8200-242 | TODO | Task 40 | DevOps Guild | Add latency percentiles panel (p50, p95, p99). |
|
||||
| 43 | PROV-8200-243 | TODO | Task 40 | DevOps Guild | Add invalidation rate panel. |
|
||||
| 44 | PROV-8200-244 | TODO | Task 40 | DevOps Guild | Add cache size panel (items, bytes). |
|
||||
| 45 | PROV-8200-245 | TODO | Task 40 | DevOps Guild | Add trust score distribution histogram. |
|
||||
| 46 | PROV-8200-246 | TODO | Tasks 40-45 | QA Guild | Validate dashboards against sample metrics. |
|
||||
| **Wave 7 (OCI Attestation Attachment)** | | | | | |
|
||||
| 47 | PROV-8200-247 | TODO | Sprint 0002 | ExportCenter Guild | Define `stella.ops/provcache@v1` predicate type. |
|
||||
| 48 | PROV-8200-248 | TODO | Task 47 | ExportCenter Guild | Implement OCI attestation builder for DecisionDigest. |
|
||||
| 49 | PROV-8200-249 | TODO | Task 48 | ExportCenter Guild | Integrate with OCI push workflow. |
|
||||
| 50 | PROV-8200-250 | TODO | Task 49 | ExportCenter Guild | Add configuration for automatic attestation attachment. |
|
||||
| 51 | PROV-8200-251 | TODO | Task 49 | ExportCenter Guild | Add `cosign verify-attestation` compatibility test. |
|
||||
| 52 | PROV-8200-252 | TODO | Tasks 47-51 | QA Guild | Add OCI attestation e2e tests. |
|
||||
| **Wave 8 (Documentation)** | | | | | |
|
||||
| 53 | PROV-8200-253 | TODO | All prior | Docs Guild | Document UI components and usage. |
|
||||
| 54 | PROV-8200-254 | TODO | All prior | Docs Guild | Document metrics and alerting recommendations. |
|
||||
| 55 | PROV-8200-255 | TODO | All prior | Docs Guild | Document OCI attestation verification. |
|
||||
| 56 | PROV-8200-256 | TODO | All prior | Docs Guild | Add Grafana dashboard to `deploy/grafana/`. |
|
||||
|
||||
---
|
||||
|
||||
## Angular Component Specifications
|
||||
|
||||
### ProvenanceBadgeComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stellaops-provenance-badge',
|
||||
template: `
|
||||
<span class="provenance-badge" [class]="state" [matTooltip]="tooltip">
|
||||
<mat-icon>{{ icon }}</mat-icon>
|
||||
<span class="label">{{ label }}</span>
|
||||
</span>
|
||||
`
|
||||
})
|
||||
export class ProvenanceBadgeComponent {
|
||||
@Input() state: 'cached' | 'computed' | 'stale' | 'unknown' = 'unknown';
|
||||
@Input() cacheDetails?: CacheDetails;
|
||||
|
||||
get icon(): string {
|
||||
return {
|
||||
cached: 'bolt',
|
||||
computed: 'refresh',
|
||||
stale: 'hourglass_empty',
|
||||
unknown: 'help_outline'
|
||||
}[this.state];
|
||||
}
|
||||
|
||||
get tooltip(): string {
|
||||
if (this.state === 'cached' && this.cacheDetails) {
|
||||
return `Cached ${this.cacheDetails.ageSeconds}s ago, trust score: ${this.cacheDetails.trustScore}`;
|
||||
}
|
||||
return {
|
||||
cached: 'Provenance-cached decision',
|
||||
computed: 'Freshly computed decision',
|
||||
stale: 'Cache expired, recomputing...',
|
||||
unknown: 'Unknown provenance state'
|
||||
}[this.state];
|
||||
}
|
||||
}
|
||||
|
||||
interface CacheDetails {
|
||||
veriKey: string;
|
||||
ageSeconds: number;
|
||||
trustScore: number;
|
||||
expiresAt: string;
|
||||
}
|
||||
```
|
||||
|
||||
### TrustScoreComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stellaops-trust-score',
|
||||
template: `
|
||||
<div class="trust-score-container">
|
||||
<div class="donut-chart" [style.--score]="score">
|
||||
<span class="score-value">{{ score }}</span>
|
||||
</div>
|
||||
<div class="breakdown" *ngIf="showBreakdown">
|
||||
<div *ngFor="let item of breakdown" class="breakdown-item">
|
||||
<span class="component-name">{{ item.name }}</span>
|
||||
<span class="component-score" [class]="item.status">{{ item.score }}%</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
`
|
||||
})
|
||||
export class TrustScoreComponent {
|
||||
@Input() score: number = 0;
|
||||
@Input() breakdown?: TrustScoreBreakdown[];
|
||||
@Input() showBreakdown: boolean = false;
|
||||
|
||||
get scoreClass(): string {
|
||||
if (this.score >= 80) return 'high';
|
||||
if (this.score >= 50) return 'medium';
|
||||
return 'low';
|
||||
}
|
||||
}
|
||||
|
||||
interface TrustScoreBreakdown {
|
||||
name: string; // 'Reachability', 'SBOM', 'VEX', 'Policy', 'Signer'
|
||||
score: number; // 0-100 for this component
|
||||
weight: number; // Weight percentage
|
||||
status: 'good' | 'warning' | 'poor';
|
||||
}
|
||||
```
|
||||
|
||||
### ProofTreeComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stellaops-proof-tree',
|
||||
template: `
|
||||
<mat-tree [dataSource]="dataSource" [treeControl]="treeControl">
|
||||
<mat-tree-node *matTreeNodeDef="let node" matTreeNodePadding>
|
||||
<button mat-icon-button disabled></button>
|
||||
<mat-icon [class]="node.type">{{ getIcon(node.type) }}</mat-icon>
|
||||
<span class="node-label">{{ node.label }}</span>
|
||||
<span class="node-value" *ngIf="node.value">{{ node.value }}</span>
|
||||
<button mat-icon-button *ngIf="node.downloadable" (click)="download(node)">
|
||||
<mat-icon>download</mat-icon>
|
||||
</button>
|
||||
</mat-tree-node>
|
||||
|
||||
<mat-nested-tree-node *matTreeNodeDef="let node; when: hasChild">
|
||||
<div class="mat-tree-node">
|
||||
<button mat-icon-button matTreeNodeToggle>
|
||||
<mat-icon>{{ treeControl.isExpanded(node) ? 'expand_more' : 'chevron_right' }}</mat-icon>
|
||||
</button>
|
||||
<mat-icon [class]="node.type">{{ getIcon(node.type) }}</mat-icon>
|
||||
<span class="node-label">{{ node.label }}</span>
|
||||
</div>
|
||||
<div [class.hidden]="!treeControl.isExpanded(node)">
|
||||
<ng-container matTreeNodeOutlet></ng-container>
|
||||
</div>
|
||||
</mat-nested-tree-node>
|
||||
</mat-tree>
|
||||
|
||||
<div class="actions">
|
||||
<button mat-raised-button (click)="verifyProof()" [disabled]="verifying">
|
||||
<mat-icon>verified</mat-icon>
|
||||
Verify Merkle Proof
|
||||
</button>
|
||||
<mat-progress-spinner *ngIf="verifying" mode="indeterminate" diameter="20"></mat-progress-spinner>
|
||||
<span *ngIf="verificationResult" [class]="verificationResult.valid ? 'valid' : 'invalid'">
|
||||
{{ verificationResult.message }}
|
||||
</span>
|
||||
</div>
|
||||
`
|
||||
})
|
||||
export class ProofTreeComponent {
|
||||
@Input() decisionDigest!: DecisionDigest;
|
||||
@Input() proofRoot!: string;
|
||||
|
||||
// Tree control and data source setup...
|
||||
|
||||
async verifyProof(): Promise<void> {
|
||||
this.verifying = true;
|
||||
try {
|
||||
const result = await this.provcacheService.verifyMerkleProof(this.proofRoot);
|
||||
this.verificationResult = result;
|
||||
} finally {
|
||||
this.verifying = false;
|
||||
}
|
||||
}
|
||||
|
||||
async download(node: ProofTreeNode): Promise<void> {
|
||||
if (node.chunkIndex !== undefined) {
|
||||
const blob = await this.provcacheService.downloadChunk(this.proofRoot, node.chunkIndex);
|
||||
// Trigger download...
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics Specification
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
```
|
||||
# Counter: Total cache requests
|
||||
provcache_requests_total{source="valkey|postgres", result="hit|miss|expired"}
|
||||
|
||||
# Counter: Cache hits
|
||||
provcache_hits_total{source="valkey|postgres"}
|
||||
|
||||
# Counter: Cache misses
|
||||
provcache_misses_total{reason="not_found|expired|invalidated"}
|
||||
|
||||
# Histogram: Latency in seconds
|
||||
provcache_latency_seconds{operation="get|set|invalidate", source="valkey|postgres"}
|
||||
|
||||
# Gauge: Current item count
|
||||
provcache_items_count{source="valkey|postgres"}
|
||||
|
||||
# Counter: Invalidations
|
||||
provcache_invalidations_total{reason="signer_revoked|epoch_advanced|ttl_expired|manual"}
|
||||
|
||||
# Gauge: Average trust score
|
||||
provcache_trust_score_average
|
||||
|
||||
# Histogram: Trust score distribution
|
||||
provcache_trust_score_bucket{le="20|40|60|80|100"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OCI Attestation Format
|
||||
|
||||
### Predicate Type
|
||||
|
||||
`stella.ops/provcache@v1`
|
||||
|
||||
### Predicate Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "stella.ops/provcache@v1",
|
||||
"veriKey": "sha256:abc123...",
|
||||
"decision": {
|
||||
"digestVersion": "v1",
|
||||
"verdictHash": "sha256:def456...",
|
||||
"proofRoot": "sha256:789abc...",
|
||||
"trustScore": 85,
|
||||
"createdAt": "2025-12-24T12:00:00Z",
|
||||
"expiresAt": "2025-12-25T12:00:00Z"
|
||||
},
|
||||
"inputs": {
|
||||
"sourceDigest": "sha256:image...",
|
||||
"sbomDigest": "sha256:sbom...",
|
||||
"policyDigest": "sha256:policy...",
|
||||
"feedEpoch": "2024-W52"
|
||||
},
|
||||
"verdicts": {
|
||||
"CVE-2024-1234": "mitigated",
|
||||
"CVE-2024-5678": "affected"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
# Verify attestation with cosign
|
||||
cosign verify-attestation \
|
||||
--type stella.ops/provcache@v1 \
|
||||
--certificate-identity-regexp '.*@stellaops\.example\.com' \
|
||||
--certificate-oidc-issuer https://auth.stellaops.example.com \
|
||||
registry.example.com/app:v1.2.3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-3 | API extensions | New fields in responses |
|
||||
| **Wave 1** | 4-9 | Provenance badge | Badge visible in UI |
|
||||
| **Wave 2** | 10-15 | Trust score display | Score visualization works |
|
||||
| **Wave 3** | 16-23 | Proof tree viewer | Tree renders, chunks downloadable |
|
||||
| **Wave 4** | 24-31 | Input manifest | Manifest panel displays correctly |
|
||||
| **Wave 5** | 32-39 | Metrics | Prometheus metrics exposed |
|
||||
| **Wave 6** | 40-46 | Grafana dashboards | Dashboards operational |
|
||||
| **Wave 7** | 47-52 | OCI attestation | cosign verification passes |
|
||||
| **Wave 8** | 53-56 | Documentation | All components documented |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint |
|
||||
|-----------|-------------|----------------|
|
||||
| Angular patterns | Follow existing component patterns | Frontend standards |
|
||||
| Grafana provisioning | Dashboards auto-deployed via Helm | DevOps |
|
||||
| OCI push integration | ExportCenter handles image push | ExportCenter module |
|
||||
| cosign compatibility | Attestation format must be verifiable | Signer module |
|
||||
| Theme support | Components must support light/dark | Frontend standards |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Material Design icons | Consistent with existing UI |
|
||||
| Donut chart for trust score | Familiar visualization, shows proportion |
|
||||
| Lazy chunk fetch in UI | Avoid loading full evidence upfront |
|
||||
| OCI attestation as optional | Not all images need provenance attached |
|
||||
| Prometheus metrics | Standard observability stack |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Large proof tree performance | UI lag | Virtual scrolling, lazy loading | Frontend Guild |
|
||||
| Metric cardinality explosion | Storage bloat | Limit label values | Platform Guild |
|
||||
| OCI attestation size limits | Push failure | Compress, use minimal predicate | ExportCenter Guild |
|
||||
| Dashboard query performance | Slow load | Pre-aggregate metrics | DevOps Guild |
|
||||
| Theme inconsistency | Visual bugs | Use theme CSS variables | Frontend Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Provcache advisory gap analysis | Project Mgmt |
|
||||
181
docs/implplan/SPRINT_8200_0001_0003_sbom_schema_validation_ci.md
Normal file
181
docs/implplan/SPRINT_8200_0001_0003_sbom_schema_validation_ci.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Sprint 8200.0001.0003 · SBOM Schema Validation in CI
|
||||
|
||||
## Priority
|
||||
**P2 - HIGH** | Estimated Effort: 1 day
|
||||
|
||||
## Topic & Scope
|
||||
- Integrate CycloneDX sbom-utility for independent schema validation in CI.
|
||||
- Add SPDX 3.0.1 schema validation.
|
||||
- Fail CI on schema/version drift before diff or policy evaluation.
|
||||
- Validate golden fixtures on every PR.
|
||||
- **Working directory:** `.gitea/workflows/`, `docs/schemas/`, `scripts/`
|
||||
- **Evidence:** CI fails on invalid SBOM; all golden fixtures validate; schema versions pinned.
|
||||
|
||||
## Problem Statement
|
||||
Current state:
|
||||
- CycloneDX 1.6 and SPDX 3.0.1 fixtures exist in `bench/golden-corpus/`
|
||||
- No external validator confirms schema compliance
|
||||
- Schema drift could go unnoticed until runtime
|
||||
|
||||
Required:
|
||||
- Use `sbom-utility validate` (or equivalent) as independent check
|
||||
- Validate all SBOM outputs against official schemas
|
||||
- Fail fast on version/format mismatches
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: None (independent CI improvement)
|
||||
- Blocks: None
|
||||
- Safe to run in parallel with: All other sprints
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (Schema Versions section)
|
||||
- CycloneDX sbom-utility: https://github.com/CycloneDX/sbom-utility
|
||||
- SPDX tools: https://github.com/spdx/tools-python
|
||||
- Product Advisory: §1 Golden fixtures & schema gates
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Schema Files** | | | | | |
|
||||
| 1 | SCHEMA-8200-001 | TODO | None | Scanner Guild | Download and commit CycloneDX 1.6 JSON schema to `docs/schemas/`. |
|
||||
| 2 | SCHEMA-8200-002 | TODO | None | Scanner Guild | Download and commit SPDX 3.0.1 JSON schema to `docs/schemas/`. |
|
||||
| 3 | SCHEMA-8200-003 | TODO | None | Scanner Guild | Download and commit OpenVEX 0.2.0 schema to `docs/schemas/`. |
|
||||
| **Validation Scripts** | | | | | |
|
||||
| 4 | SCHEMA-8200-004 | TODO | Task 1-3 | Scanner Guild | Create `scripts/validate-sbom.sh` wrapper for sbom-utility. |
|
||||
| 5 | SCHEMA-8200-005 | TODO | Task 4 | Scanner Guild | Create `scripts/validate-spdx.sh` wrapper for SPDX validation. |
|
||||
| 6 | SCHEMA-8200-006 | TODO | Task 4 | Scanner Guild | Create `scripts/validate-vex.sh` wrapper for OpenVEX validation. |
|
||||
| **CI Workflow** | | | | | |
|
||||
| 7 | SCHEMA-8200-007 | TODO | Task 4-6 | Platform Guild | Create `.gitea/workflows/schema-validation.yml` workflow. |
|
||||
| 8 | SCHEMA-8200-008 | TODO | Task 7 | Platform Guild | Add job to validate all CycloneDX fixtures in `bench/golden-corpus/`. |
|
||||
| 9 | SCHEMA-8200-009 | TODO | Task 7 | Platform Guild | Add job to validate all SPDX fixtures in `bench/golden-corpus/`. |
|
||||
| 10 | SCHEMA-8200-010 | TODO | Task 7 | Platform Guild | Add job to validate all VEX fixtures. |
|
||||
| 11 | SCHEMA-8200-011 | TODO | Task 7 | Platform Guild | Configure workflow to run on PR and push to main. |
|
||||
| **Integration** | | | | | |
|
||||
| 12 | SCHEMA-8200-012 | TODO | Task 11 | Platform Guild | Add schema validation as required check for PR merge. |
|
||||
| 13 | SCHEMA-8200-013 | TODO | Task 11 | Platform Guild | Add validation step to `determinism-gate.yml` workflow. |
|
||||
| **Testing & Negative Cases** | | | | | |
|
||||
| 14 | SCHEMA-8200-014 | TODO | Task 11 | Scanner Guild | Add test fixture with intentionally invalid CycloneDX (wrong version). |
|
||||
| 15 | SCHEMA-8200-015 | TODO | Task 11 | Scanner Guild | Verify CI fails on invalid fixture (negative test). |
|
||||
| **Documentation** | | | | | |
|
||||
| 16 | SCHEMA-8200-016 | TODO | Task 15 | Scanner Guild | Document schema validation in `docs/testing/schema-validation.md`. |
|
||||
| 17 | SCHEMA-8200-017 | TODO | Task 15 | Scanner Guild | Add troubleshooting guide for schema validation failures. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### CI Workflow
|
||||
```yaml
|
||||
# .gitea/workflows/schema-validation.yml
|
||||
name: Schema Validation
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'bench/golden-corpus/**'
|
||||
- 'src/Scanner/**'
|
||||
- 'docs/schemas/**'
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
validate-cyclonedx:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install sbom-utility
|
||||
run: |
|
||||
curl -sSfL https://github.com/CycloneDX/sbom-utility/releases/download/v0.16.0/sbom-utility-v0.16.0-linux-amd64.tar.gz | tar xz
|
||||
sudo mv sbom-utility /usr/local/bin/
|
||||
|
||||
- name: Validate CycloneDX fixtures
|
||||
run: |
|
||||
find bench/golden-corpus -name '*cyclonedx*.json' | while read file; do
|
||||
echo "Validating: $file"
|
||||
sbom-utility validate --input-file "$file" --schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||
done
|
||||
|
||||
validate-spdx:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install SPDX tools
|
||||
run: pip install spdx-tools
|
||||
|
||||
- name: Validate SPDX fixtures
|
||||
run: |
|
||||
find bench/golden-corpus -name '*spdx*.json' | while read file; do
|
||||
echo "Validating: $file"
|
||||
pyspdxtools validate "$file"
|
||||
done
|
||||
|
||||
validate-vex:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Validate OpenVEX fixtures
|
||||
run: |
|
||||
find bench/golden-corpus -name '*vex*.json' | while read file; do
|
||||
echo "Validating: $file"
|
||||
# Use ajv or similar JSON schema validator
|
||||
npx ajv validate -s docs/schemas/openvex-0.2.0.schema.json -d "$file"
|
||||
done
|
||||
```
|
||||
|
||||
### Validation Script
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/validate-sbom.sh
|
||||
set -euo pipefail
|
||||
|
||||
SCHEMA_DIR="docs/schemas"
|
||||
SBOM_FILE="$1"
|
||||
FORMAT="${2:-auto}"
|
||||
|
||||
case "$FORMAT" in
|
||||
cyclonedx|auto)
|
||||
if grep -q '"bomFormat".*"CycloneDX"' "$SBOM_FILE"; then
|
||||
sbom-utility validate --input-file "$SBOM_FILE" --schema "$SCHEMA_DIR/cyclonedx-bom-1.6.schema.json"
|
||||
fi
|
||||
;;
|
||||
spdx)
|
||||
pyspdxtools validate "$SBOM_FILE"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown format: $FORMAT"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `docs/schemas/cyclonedx-bom-1.6.schema.json` | Download from CycloneDX |
|
||||
| `docs/schemas/spdx-3.0.1.schema.json` | Download from SPDX |
|
||||
| `docs/schemas/openvex-0.2.0.schema.json` | Download from OpenVEX |
|
||||
| `scripts/validate-sbom.sh` | Create |
|
||||
| `scripts/validate-spdx.sh` | Create |
|
||||
| `scripts/validate-vex.sh` | Create |
|
||||
| `.gitea/workflows/schema-validation.yml` | Create |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] CI validates all CycloneDX 1.6 fixtures
|
||||
2. [ ] CI validates all SPDX 3.0.1 fixtures
|
||||
3. [ ] CI validates all OpenVEX fixtures
|
||||
4. [ ] CI fails on schema violation (negative test passes)
|
||||
5. [ ] Schema validation is a required PR check
|
||||
6. [ ] Documentation explains how to fix validation errors
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| sbom-utility version changes behavior | Low | Pin version in CI | Platform Guild |
|
||||
| Schema download fails in CI | Low | Commit schemas to repo; don't download at runtime | Scanner Guild |
|
||||
| False positives from strict validation | Medium | Use official schemas; document known edge cases | Scanner Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P2 priority - quick win for early validation. | Project Mgmt |
|
||||
217
docs/implplan/SPRINT_8200_0001_0004_e2e_reproducibility_test.md
Normal file
217
docs/implplan/SPRINT_8200_0001_0004_e2e_reproducibility_test.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Sprint 8200.0001.0004 · Full E2E Reproducibility Test
|
||||
|
||||
## Priority
|
||||
**P3 - HIGH** | Estimated Effort: 5 days
|
||||
|
||||
## Topic & Scope
|
||||
- Implement comprehensive end-to-end reproducibility test covering the full pipeline.
|
||||
- Pipeline: ingest → normalize → diff → decide → attest → bundle → reverify.
|
||||
- Verify identical inputs produce identical verdict hashes on fresh runners.
|
||||
- Compare bundle manifests byte-for-byte across runs.
|
||||
- **Working directory:** `tests/integration/StellaOps.Integration.E2E/`, `.gitea/workflows/`
|
||||
- **Evidence:** E2E test passes; verdict hash matches across runs; bundle manifest identical.
|
||||
|
||||
## Problem Statement
|
||||
Current state:
|
||||
- `ProofChainIntegrationTests` covers scan → manifest → score → proof → verify
|
||||
- Missing: advisory ingestion, normalization, VEX integration phases
|
||||
- No "clean runner" verification
|
||||
|
||||
Required:
|
||||
- Full pipeline test: `ingest → normalize → diff → decide → attest → bundle`
|
||||
- Re-run on fresh environment and compare:
|
||||
- Verdict hash (must match)
|
||||
- Bundle manifest (must match)
|
||||
- Artifact digests (must match)
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 8200.0001.0001 (VerdictId content-addressing)
|
||||
- Depends on: Sprint 8200.0001.0002 (DSSE round-trip testing)
|
||||
- Blocks: None
|
||||
- Safe to run in parallel with: Sprint 8200.0001.0003 (Schema validation)
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (Replay Procedure section)
|
||||
- `tests/integration/StellaOps.Integration.ProofChain/` (existing partial E2E)
|
||||
- `docs/testing/determinism-verification.md`
|
||||
- Product Advisory: §5 End-to-end reproducibility test
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Test Infrastructure** | | | | | |
|
||||
| 1 | E2E-8200-001 | TODO | None | Platform Guild | Create `tests/integration/StellaOps.Integration.E2E/` project. |
|
||||
| 2 | E2E-8200-002 | TODO | Task 1 | Platform Guild | Create `E2EReproducibilityTestFixture` with full service composition. |
|
||||
| 3 | E2E-8200-003 | TODO | Task 2 | Platform Guild | Add helper to snapshot all inputs (feeds, policies, VEX) with hashes. |
|
||||
| 4 | E2E-8200-004 | TODO | Task 2 | Platform Guild | Add helper to compare verdict manifests byte-for-byte. |
|
||||
| **Pipeline Stages** | | | | | |
|
||||
| 5 | E2E-8200-005 | TODO | Task 2 | Concelier Guild | Implement ingest stage: load advisory feeds from fixtures. |
|
||||
| 6 | E2E-8200-006 | TODO | Task 5 | Concelier Guild | Implement normalize stage: merge advisories, deduplicate. |
|
||||
| 7 | E2E-8200-007 | TODO | Task 6 | Scanner Guild | Implement diff stage: compare SBOM against advisories. |
|
||||
| 8 | E2E-8200-008 | TODO | Task 7 | Policy Guild | Implement decide stage: evaluate policy, compute verdict. |
|
||||
| 9 | E2E-8200-009 | TODO | Task 8 | Attestor Guild | Implement attest stage: create DSSE envelope. |
|
||||
| 10 | E2E-8200-010 | TODO | Task 9 | Attestor Guild | Implement bundle stage: package into Sigstore bundle. |
|
||||
| **Reproducibility Tests** | | | | | |
|
||||
| 11 | E2E-8200-011 | TODO | Task 10 | Platform Guild | Add test: run pipeline twice → identical verdict hash. |
|
||||
| 12 | E2E-8200-012 | TODO | Task 11 | Platform Guild | Add test: run pipeline twice → identical bundle manifest. |
|
||||
| 13 | E2E-8200-013 | TODO | Task 11 | Platform Guild | Add test: run pipeline with frozen clock → identical timestamps. |
|
||||
| 14 | E2E-8200-014 | TODO | Task 11 | Platform Guild | Add test: parallel execution (10 concurrent) → all identical. |
|
||||
| **Cross-Environment Tests** | | | | | |
|
||||
| 15 | E2E-8200-015 | TODO | Task 12 | Platform Guild | Add CI job: run on ubuntu-latest, compare hashes. |
|
||||
| 16 | E2E-8200-016 | TODO | Task 15 | Platform Guild | Add CI job: run on windows-latest, compare hashes. |
|
||||
| 17 | E2E-8200-017 | TODO | Task 15 | Platform Guild | Add CI job: run on macos-latest, compare hashes. |
|
||||
| 18 | E2E-8200-018 | TODO | Task 17 | Platform Guild | Add cross-platform hash comparison matrix job. |
|
||||
| **Golden Baseline** | | | | | |
|
||||
| 19 | E2E-8200-019 | TODO | Task 18 | Platform Guild | Create golden baseline fixtures with expected hashes. |
|
||||
| 20 | E2E-8200-020 | TODO | Task 19 | Platform Guild | Add CI assertion: current run matches golden baseline. |
|
||||
| 21 | E2E-8200-021 | TODO | Task 20 | Platform Guild | Document baseline update procedure for intentional changes. |
|
||||
| **CI Workflow** | | | | | |
|
||||
| 22 | E2E-8200-022 | TODO | Task 18 | Platform Guild | Create `.gitea/workflows/e2e-reproducibility.yml`. |
|
||||
| 23 | E2E-8200-023 | TODO | Task 22 | Platform Guild | Add nightly schedule for full reproducibility suite. |
|
||||
| 24 | E2E-8200-024 | TODO | Task 22 | Platform Guild | Add reproducibility gate as required PR check. |
|
||||
| **Documentation** | | | | | |
|
||||
| 25 | E2E-8200-025 | TODO | Task 24 | Platform Guild | Document E2E test structure in `docs/testing/e2e-reproducibility.md`. |
|
||||
| 26 | E2E-8200-026 | TODO | Task 24 | Platform Guild | Add troubleshooting guide for reproducibility failures. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### E2E Test Structure
|
||||
```csharp
|
||||
public class E2EReproducibilityTests : IClassFixture<E2EReproducibilityTestFixture>
|
||||
{
|
||||
private readonly E2EReproducibilityTestFixture _fixture;
|
||||
|
||||
[Fact]
|
||||
public async Task FullPipeline_ProducesIdenticalVerdictHash_AcrossRuns()
|
||||
{
|
||||
// Arrange - Snapshot inputs
|
||||
var inputSnapshot = await _fixture.SnapshotInputsAsync();
|
||||
|
||||
// Act - Run pipeline twice
|
||||
var result1 = await RunFullPipelineAsync(inputSnapshot);
|
||||
var result2 = await RunFullPipelineAsync(inputSnapshot);
|
||||
|
||||
// Assert - Identical outputs
|
||||
Assert.Equal(result1.VerdictHash, result2.VerdictHash);
|
||||
Assert.Equal(result1.BundleManifestHash, result2.BundleManifestHash);
|
||||
Assert.Equal(result1.DsseEnvelopeHash, result2.DsseEnvelopeHash);
|
||||
}
|
||||
|
||||
private async Task<PipelineResult> RunFullPipelineAsync(InputSnapshot inputs)
|
||||
{
|
||||
// Stage 1: Ingest
|
||||
var advisories = await _fixture.IngestAdvisoriesAsync(inputs.FeedSnapshot);
|
||||
|
||||
// Stage 2: Normalize
|
||||
var normalized = await _fixture.NormalizeAdvisoriesAsync(advisories);
|
||||
|
||||
// Stage 3: Diff
|
||||
var diff = await _fixture.ComputeDiffAsync(inputs.Sbom, normalized);
|
||||
|
||||
// Stage 4: Decide
|
||||
var verdict = await _fixture.EvaluatePolicyAsync(diff, inputs.PolicyPack);
|
||||
|
||||
// Stage 5: Attest
|
||||
var envelope = await _fixture.CreateAttestationAsync(verdict);
|
||||
|
||||
// Stage 6: Bundle
|
||||
var bundle = await _fixture.CreateBundleAsync(envelope);
|
||||
|
||||
return new PipelineResult
|
||||
{
|
||||
VerdictHash = verdict.VerdictId,
|
||||
BundleManifestHash = ComputeHash(bundle.Manifest),
|
||||
DsseEnvelopeHash = ComputeHash(envelope.Serialize())
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### CI Workflow
|
||||
```yaml
|
||||
# .gitea/workflows/e2e-reproducibility.yml
|
||||
name: E2E Reproducibility
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'src/**'
|
||||
- 'tests/integration/**'
|
||||
schedule:
|
||||
- cron: '0 2 * * *' # Nightly at 2am UTC
|
||||
|
||||
jobs:
|
||||
reproducibility:
|
||||
strategy:
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup .NET
|
||||
uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
dotnet-version: '10.0.x'
|
||||
|
||||
- name: Run E2E Reproducibility Tests
|
||||
run: |
|
||||
dotnet test tests/integration/StellaOps.Integration.E2E \
|
||||
--filter "Category=Reproducibility" \
|
||||
--logger "trx;LogFileName=results-${{ matrix.os }}.trx"
|
||||
|
||||
- name: Upload Results
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: reproducibility-${{ matrix.os }}
|
||||
path: |
|
||||
**/results-*.trx
|
||||
**/verdict-hashes.json
|
||||
|
||||
compare:
|
||||
needs: reproducibility
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download All Results
|
||||
uses: actions/download-artifact@v4
|
||||
|
||||
- name: Compare Hashes Across Platforms
|
||||
run: |
|
||||
# Extract verdict hashes from each platform
|
||||
for os in ubuntu-latest windows-latest macos-latest; do
|
||||
cat reproducibility-$os/verdict-hashes.json
|
||||
done | jq -s '.[0] == .[1] and .[1] == .[2]' | grep -q 'true'
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `tests/integration/StellaOps.Integration.E2E/StellaOps.Integration.E2E.csproj` | Create |
|
||||
| `tests/integration/StellaOps.Integration.E2E/E2EReproducibilityTestFixture.cs` | Create |
|
||||
| `tests/integration/StellaOps.Integration.E2E/E2EReproducibilityTests.cs` | Create |
|
||||
| `tests/integration/StellaOps.Integration.E2E/PipelineStages/` | Create directory |
|
||||
| `.gitea/workflows/e2e-reproducibility.yml` | Create |
|
||||
| `bench/e2e-baselines/` | Create directory for golden baselines |
|
||||
| `docs/testing/e2e-reproducibility.md` | Create |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] Full pipeline test passes (ingest → bundle)
|
||||
2. [ ] Identical inputs → identical verdict hash (100% match)
|
||||
3. [ ] Identical inputs → identical bundle manifest (100% match)
|
||||
4. [ ] Cross-platform reproducibility verified (Linux, Windows, macOS)
|
||||
5. [ ] Golden baseline comparison implemented
|
||||
6. [ ] CI workflow runs nightly and on PR
|
||||
7. [ ] Documentation complete
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Platform-specific differences (line endings, paths) | High | Use canonical serialization; normalize paths | Platform Guild |
|
||||
| Floating-point precision differences | Medium | Use fixed-precision decimals; avoid floats | Platform Guild |
|
||||
| Parallel execution race conditions | Medium | Use deterministic ordering; thread-safe collections | Platform Guild |
|
||||
| Clock drift between pipeline stages | Medium | Freeze clock for entire pipeline run | Platform Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P3 priority - validates full reproducibility chain. | Project Mgmt |
|
||||
@@ -0,0 +1,196 @@
|
||||
# Sprint 8200.0001.0005 · Sigstore Bundle Implementation
|
||||
|
||||
## Priority
|
||||
**P4 - MEDIUM** | Estimated Effort: 3 days
|
||||
|
||||
## Topic & Scope
|
||||
- Implement Sigstore Bundle v0.3 marshalling and unmarshalling.
|
||||
- Package DSSE envelope + certificates + Rekor proof into self-contained bundle.
|
||||
- Enable offline verification with all necessary material.
|
||||
- Add cosign bundle compatibility verification.
|
||||
- **Working directory:** `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/`, `src/ExportCenter/`
|
||||
- **Evidence:** Sigstore bundles serialize/deserialize correctly; bundles verifiable by cosign; offline verification works.
|
||||
|
||||
## Problem Statement
|
||||
Current state:
|
||||
- `OciArtifactTypes.SigstoreBundle` constant defined
|
||||
- DSSE envelopes created correctly
|
||||
- No Sigstore bundle serialization/deserialization
|
||||
|
||||
Required:
|
||||
- Implement bundle format per https://github.com/sigstore/protobuf-specs
|
||||
- Package: DSSE envelope + certificate chain + Rekor entry + inclusion proof
|
||||
- Enable: `cosign verify-attestation --bundle bundle.json`
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 8200.0001.0002 (DSSE round-trip testing)
|
||||
- Blocks: None
|
||||
- Safe to run in parallel with: Sprint 8200.0001.0004 (E2E test - can mock bundle)
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (Sigstore Bundle Format section)
|
||||
- Sigstore Bundle Spec: https://github.com/sigstore/cosign/blob/main/specs/BUNDLE_SPEC.md
|
||||
- Sigstore Protobuf: https://github.com/sigstore/protobuf-specs
|
||||
- Product Advisory: §2 DSSE attestations & bundle round-trips
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Models** | | | | | |
|
||||
| 1 | BUNDLE-8200-001 | TODO | None | Attestor Guild | Create `SigstoreBundle` record matching v0.3 schema. |
|
||||
| 2 | BUNDLE-8200-002 | TODO | Task 1 | Attestor Guild | Create `VerificationMaterial` model (certificate, tlog entries). |
|
||||
| 3 | BUNDLE-8200-003 | TODO | Task 1 | Attestor Guild | Create `TransparencyLogEntry` model (logId, logIndex, inclusionProof). |
|
||||
| 4 | BUNDLE-8200-004 | TODO | Task 1 | Attestor Guild | Create `InclusionProof` model (Merkle proof data). |
|
||||
| **Serialization** | | | | | |
|
||||
| 5 | BUNDLE-8200-005 | TODO | Task 4 | Attestor Guild | Implement `SigstoreBundleSerializer.Serialize()` to JSON. |
|
||||
| 6 | BUNDLE-8200-006 | TODO | Task 5 | Attestor Guild | Implement `SigstoreBundleSerializer.Deserialize()` from JSON. |
|
||||
| 7 | BUNDLE-8200-007 | TODO | Task 6 | Attestor Guild | Add protobuf support if required for binary format. |
|
||||
| **Builder** | | | | | |
|
||||
| 8 | BUNDLE-8200-008 | TODO | Task 5 | Attestor Guild | Create `SigstoreBundleBuilder` to construct bundles from components. |
|
||||
| 9 | BUNDLE-8200-009 | TODO | Task 8 | Attestor Guild | Add certificate chain packaging to builder. |
|
||||
| 10 | BUNDLE-8200-010 | TODO | Task 8 | Attestor Guild | Add Rekor entry packaging to builder. |
|
||||
| 11 | BUNDLE-8200-011 | TODO | Task 8 | Attestor Guild | Add DSSE envelope packaging to builder. |
|
||||
| **Verification** | | | | | |
|
||||
| 12 | BUNDLE-8200-012 | TODO | Task 6 | Attestor Guild | Create `SigstoreBundleVerifier` for offline verification. |
|
||||
| 13 | BUNDLE-8200-013 | TODO | Task 12 | Attestor Guild | Implement certificate chain validation. |
|
||||
| 14 | BUNDLE-8200-014 | TODO | Task 12 | Attestor Guild | Implement Merkle inclusion proof verification. |
|
||||
| 15 | BUNDLE-8200-015 | TODO | Task 12 | Attestor Guild | Implement DSSE signature verification. |
|
||||
| **Integration** | | | | | |
|
||||
| 16 | BUNDLE-8200-016 | TODO | Task 11 | Attestor Guild | Integrate bundle creation into `AttestorBundleService`. |
|
||||
| 17 | BUNDLE-8200-017 | TODO | Task 16 | ExportCenter Guild | Add bundle export to Export Center. |
|
||||
| 18 | BUNDLE-8200-018 | TODO | Task 16 | CLI Guild | Add `stella attest bundle` command. |
|
||||
| **Testing** | | | | | |
|
||||
| 19 | BUNDLE-8200-019 | TODO | Task 6 | Attestor Guild | Add unit test: serialize → deserialize round-trip. |
|
||||
| 20 | BUNDLE-8200-020 | TODO | Task 12 | Attestor Guild | Add unit test: verify valid bundle. |
|
||||
| 21 | BUNDLE-8200-021 | TODO | Task 12 | Attestor Guild | Add unit test: verify fails with tampered bundle. |
|
||||
| 22 | BUNDLE-8200-022 | TODO | Task 18 | Attestor Guild | Add integration test: bundle verifiable by `cosign verify-attestation --bundle`. |
|
||||
| **Documentation** | | | | | |
|
||||
| 23 | BUNDLE-8200-023 | TODO | Task 22 | Attestor Guild | Document bundle format in `docs/modules/attestor/bundle-format.md`. |
|
||||
| 24 | BUNDLE-8200-024 | TODO | Task 22 | Attestor Guild | Add cosign verification examples to docs. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### Sigstore Bundle Model
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Sigstore Bundle v0.3 format for offline verification.
|
||||
/// </summary>
|
||||
public sealed record SigstoreBundle
|
||||
{
|
||||
/// <summary>Media type: application/vnd.dev.sigstore.bundle.v0.3+json</summary>
|
||||
[JsonPropertyName("mediaType")]
|
||||
public string MediaType => "application/vnd.dev.sigstore.bundle.v0.3+json";
|
||||
|
||||
/// <summary>Verification material (certs + tlog entries).</summary>
|
||||
[JsonPropertyName("verificationMaterial")]
|
||||
public required VerificationMaterial VerificationMaterial { get; init; }
|
||||
|
||||
/// <summary>The signed DSSE envelope.</summary>
|
||||
[JsonPropertyName("dsseEnvelope")]
|
||||
public required DsseEnvelope DsseEnvelope { get; init; }
|
||||
}
|
||||
|
||||
public sealed record VerificationMaterial
|
||||
{
|
||||
[JsonPropertyName("certificate")]
|
||||
public CertificateInfo? Certificate { get; init; }
|
||||
|
||||
[JsonPropertyName("tlogEntries")]
|
||||
public IReadOnlyList<TransparencyLogEntry>? TlogEntries { get; init; }
|
||||
|
||||
[JsonPropertyName("timestampVerificationData")]
|
||||
public TimestampVerificationData? TimestampVerificationData { get; init; }
|
||||
}
|
||||
|
||||
public sealed record TransparencyLogEntry
|
||||
{
|
||||
[JsonPropertyName("logIndex")]
|
||||
public required string LogIndex { get; init; }
|
||||
|
||||
[JsonPropertyName("logId")]
|
||||
public required LogId LogId { get; init; }
|
||||
|
||||
[JsonPropertyName("kindVersion")]
|
||||
public required KindVersion KindVersion { get; init; }
|
||||
|
||||
[JsonPropertyName("integratedTime")]
|
||||
public required string IntegratedTime { get; init; }
|
||||
|
||||
[JsonPropertyName("inclusionPromise")]
|
||||
public InclusionPromise? InclusionPromise { get; init; }
|
||||
|
||||
[JsonPropertyName("inclusionProof")]
|
||||
public InclusionProof? InclusionProof { get; init; }
|
||||
|
||||
[JsonPropertyName("canonicalizedBody")]
|
||||
public required string CanonicalizedBody { get; init; }
|
||||
}
|
||||
|
||||
public sealed record InclusionProof
|
||||
{
|
||||
[JsonPropertyName("logIndex")]
|
||||
public required string LogIndex { get; init; }
|
||||
|
||||
[JsonPropertyName("rootHash")]
|
||||
public required string RootHash { get; init; }
|
||||
|
||||
[JsonPropertyName("treeSize")]
|
||||
public required string TreeSize { get; init; }
|
||||
|
||||
[JsonPropertyName("hashes")]
|
||||
public required IReadOnlyList<string> Hashes { get; init; }
|
||||
|
||||
[JsonPropertyName("checkpoint")]
|
||||
public required Checkpoint Checkpoint { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Bundle Builder Usage
|
||||
```csharp
|
||||
var bundle = new SigstoreBundleBuilder()
|
||||
.WithDsseEnvelope(envelope)
|
||||
.WithCertificateChain(certChain)
|
||||
.WithRekorEntry(rekorEntry)
|
||||
.WithInclusionProof(proof)
|
||||
.Build();
|
||||
|
||||
var json = SigstoreBundleSerializer.Serialize(bundle);
|
||||
File.WriteAllText("attestation.bundle", json);
|
||||
|
||||
// Verify with cosign:
|
||||
// cosign verify-attestation --bundle attestation.bundle --certificate-identity=... image:tag
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/StellaOps.Attestor.Bundle.csproj` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Models/SigstoreBundle.cs` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Models/VerificationMaterial.cs` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Models/TransparencyLogEntry.cs` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Serialization/SigstoreBundleSerializer.cs` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Builder/SigstoreBundleBuilder.cs` | Create |
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.Bundle/Verification/SigstoreBundleVerifier.cs` | Create |
|
||||
| `src/Attestor/__Tests/StellaOps.Attestor.Bundle.Tests/` | Create test project |
|
||||
| `docs/modules/attestor/bundle-format.md` | Create |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] SigstoreBundle model matches v0.3 spec
|
||||
2. [ ] Serialize/deserialize round-trip works
|
||||
3. [ ] Bundle includes all verification material
|
||||
4. [ ] Offline verification works without network
|
||||
5. [ ] `cosign verify-attestation --bundle` succeeds
|
||||
6. [ ] Integration with AttestorBundleService complete
|
||||
7. [ ] CLI command added
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Sigstore spec changes | Medium | Pin to v0.3; monitor upstream | Attestor Guild |
|
||||
| Protobuf dependency complexity | Low | Use JSON format; protobuf optional | Attestor Guild |
|
||||
| Certificate chain validation complexity | Medium | Use existing crypto libraries; test thoroughly | Attestor Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P4 priority - enables offline verification. | Project Mgmt |
|
||||
@@ -0,0 +1,227 @@
|
||||
# Sprint 8200.0001.0006 · Budget Threshold Attestation
|
||||
|
||||
## Priority
|
||||
**P6 - MEDIUM** | Estimated Effort: 2 days
|
||||
|
||||
## Topic & Scope
|
||||
- Attest unknown budget thresholds in DSSE verdict bundles.
|
||||
- Create `BudgetCheckPredicate` to capture policy configuration at decision time.
|
||||
- Include budget check results in verdict attestations.
|
||||
- Enable auditors to verify what thresholds were enforced.
|
||||
- **Working directory:** `src/Policy/StellaOps.Policy.Engine/Attestation/`, `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`
|
||||
- **Evidence:** Budget thresholds attested in verdict bundles; predicate includes environment, limits, actual counts.
|
||||
|
||||
## Problem Statement
|
||||
Current state:
|
||||
- `UnknownsBudgetGate` enforces budgets correctly
|
||||
- `VerdictPredicateBuilder` creates verdict attestations
|
||||
- Budget configuration NOT included in attestations
|
||||
|
||||
Required:
|
||||
- Auditors need to know what thresholds were applied
|
||||
- Reproducibility requires attesting all inputs including policy config
|
||||
- Advisory §4: "Make thresholds environment-aware and attest them in the bundle"
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 8200.0001.0001 (VerdictId content-addressing)
|
||||
- Blocks: None
|
||||
- Safe to run in parallel with: Sprint 8200.0001.0004 (E2E test)
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/reproducibility.md` (Unknown Budget Attestation section)
|
||||
- `src/Policy/__Libraries/StellaOps.Policy.Unknowns/` (existing budget models)
|
||||
- `src/Policy/StellaOps.Policy.Engine/Attestation/VerdictPredicateBuilder.cs`
|
||||
- Product Advisory: §4 Policy engine: unknown-budget gates
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Models** | | | | | |
|
||||
| 1 | BUDGET-8200-001 | TODO | None | Policy Guild | Create `BudgetCheckPredicate` record with environment, limits, counts, result. |
|
||||
| 2 | BUDGET-8200-002 | TODO | Task 1 | Policy Guild | Create `BudgetCheckPredicateType` URI constant. |
|
||||
| 3 | BUDGET-8200-003 | TODO | Task 1 | Policy Guild | Add `ConfigHash` field for budget configuration hash. |
|
||||
| **Integration** | | | | | |
|
||||
| 4 | BUDGET-8200-004 | TODO | Task 3 | Policy Guild | Modify `UnknownBudgetService` to return `BudgetCheckResult` with details. |
|
||||
| 5 | BUDGET-8200-005 | TODO | Task 4 | Policy Guild | Add `BudgetCheckResult` to `PolicyGateContext`. |
|
||||
| 6 | BUDGET-8200-006 | TODO | Task 5 | Policy Guild | Modify `VerdictPredicateBuilder` to include `BudgetCheckPredicate`. |
|
||||
| 7 | BUDGET-8200-007 | TODO | Task 6 | Policy Guild | Compute budget config hash for determinism proof. |
|
||||
| **Attestation** | | | | | |
|
||||
| 8 | BUDGET-8200-008 | TODO | Task 6 | Attestor Guild | Create `BudgetCheckStatement` extending `InTotoStatement`. |
|
||||
| 9 | BUDGET-8200-009 | TODO | Task 8 | Attestor Guild | Integrate budget statement into `PolicyDecisionAttestationService`. |
|
||||
| 10 | BUDGET-8200-010 | TODO | Task 9 | Attestor Guild | Add budget predicate to verdict DSSE envelope. |
|
||||
| **Testing** | | | | | |
|
||||
| 11 | BUDGET-8200-011 | TODO | Task 10 | Policy Guild | Add unit test: budget predicate included in verdict attestation. |
|
||||
| 12 | BUDGET-8200-012 | TODO | Task 11 | Policy Guild | Add unit test: budget config hash is deterministic. |
|
||||
| 13 | BUDGET-8200-013 | TODO | Task 11 | Policy Guild | Add unit test: different environments produce different predicates. |
|
||||
| 14 | BUDGET-8200-014 | TODO | Task 11 | Policy Guild | Add integration test: extract budget predicate from DSSE envelope. |
|
||||
| **Verification** | | | | | |
|
||||
| 15 | BUDGET-8200-015 | TODO | Task 10 | Policy Guild | Add verification rule: budget predicate matches current config. |
|
||||
| 16 | BUDGET-8200-016 | TODO | Task 15 | Policy Guild | Add alert if budget thresholds were changed since attestation. |
|
||||
| **Documentation** | | | | | |
|
||||
| 17 | BUDGET-8200-017 | TODO | Task 16 | Policy Guild | Document budget predicate format in `docs/modules/policy/budget-attestation.md`. |
|
||||
| 18 | BUDGET-8200-018 | TODO | Task 17 | Policy Guild | Add examples of extracting budget info from attestation. |
|
||||
|
||||
## Technical Specification
|
||||
|
||||
### BudgetCheckPredicate Model
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Predicate capturing unknown budget enforcement at decision time.
|
||||
/// </summary>
|
||||
public sealed record BudgetCheckPredicate
|
||||
{
|
||||
public const string PredicateTypeUri = "https://stellaops.io/attestation/budget-check/v1";
|
||||
|
||||
/// <summary>Environment for which budget was evaluated.</summary>
|
||||
[JsonPropertyName("environment")]
|
||||
public required string Environment { get; init; }
|
||||
|
||||
/// <summary>Budget configuration applied.</summary>
|
||||
[JsonPropertyName("budgetConfig")]
|
||||
public required BudgetConfig BudgetConfig { get; init; }
|
||||
|
||||
/// <summary>Actual unknown counts at evaluation time.</summary>
|
||||
[JsonPropertyName("actualCounts")]
|
||||
public required BudgetActualCounts ActualCounts { get; init; }
|
||||
|
||||
/// <summary>Budget check result: pass, warn, fail.</summary>
|
||||
[JsonPropertyName("result")]
|
||||
public required string Result { get; init; }
|
||||
|
||||
/// <summary>SHA-256 hash of budget configuration for determinism.</summary>
|
||||
[JsonPropertyName("configHash")]
|
||||
public required string ConfigHash { get; init; }
|
||||
|
||||
/// <summary>Violations if any limits exceeded.</summary>
|
||||
[JsonPropertyName("violations")]
|
||||
public IReadOnlyList<BudgetViolation>? Violations { get; init; }
|
||||
}
|
||||
|
||||
public sealed record BudgetConfig
|
||||
{
|
||||
[JsonPropertyName("maxUnknownCount")]
|
||||
public int MaxUnknownCount { get; init; }
|
||||
|
||||
[JsonPropertyName("maxCumulativeUncertainty")]
|
||||
public double MaxCumulativeUncertainty { get; init; }
|
||||
|
||||
[JsonPropertyName("reasonLimits")]
|
||||
public IReadOnlyDictionary<string, int>? ReasonLimits { get; init; }
|
||||
|
||||
[JsonPropertyName("action")]
|
||||
public string Action { get; init; } = "warn";
|
||||
}
|
||||
|
||||
public sealed record BudgetActualCounts
|
||||
{
|
||||
[JsonPropertyName("total")]
|
||||
public int Total { get; init; }
|
||||
|
||||
[JsonPropertyName("cumulativeUncertainty")]
|
||||
public double CumulativeUncertainty { get; init; }
|
||||
|
||||
[JsonPropertyName("byReason")]
|
||||
public IReadOnlyDictionary<string, int>? ByReason { get; init; }
|
||||
}
|
||||
|
||||
public sealed record BudgetViolation
|
||||
{
|
||||
[JsonPropertyName("type")]
|
||||
public required string Type { get; init; }
|
||||
|
||||
[JsonPropertyName("limit")]
|
||||
public int Limit { get; init; }
|
||||
|
||||
[JsonPropertyName("actual")]
|
||||
public int Actual { get; init; }
|
||||
|
||||
[JsonPropertyName("reason")]
|
||||
public string? Reason { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Integration into VerdictPredicateBuilder
|
||||
```csharp
|
||||
public class VerdictPredicateBuilder
|
||||
{
|
||||
public VerdictPredicate Build(PolicyEvaluationResult result, PolicyGateContext context)
|
||||
{
|
||||
var budgetPredicate = CreateBudgetCheckPredicate(context);
|
||||
|
||||
return new VerdictPredicate
|
||||
{
|
||||
VerdictId = result.VerdictId,
|
||||
Status = result.Status,
|
||||
Gate = result.RecommendedGate,
|
||||
Evidence = result.Evidence,
|
||||
BudgetCheck = budgetPredicate, // NEW
|
||||
DeterminismHash = ComputeDeterminismHash(result, budgetPredicate)
|
||||
};
|
||||
}
|
||||
|
||||
private BudgetCheckPredicate CreateBudgetCheckPredicate(PolicyGateContext context)
|
||||
{
|
||||
var budgetResult = context.BudgetCheckResult;
|
||||
|
||||
return new BudgetCheckPredicate
|
||||
{
|
||||
Environment = context.Environment,
|
||||
BudgetConfig = new BudgetConfig
|
||||
{
|
||||
MaxUnknownCount = budgetResult.Budget.MaxUnknownCount,
|
||||
MaxCumulativeUncertainty = budgetResult.Budget.MaxCumulativeUncertainty,
|
||||
ReasonLimits = budgetResult.Budget.ReasonLimits,
|
||||
Action = budgetResult.Budget.Action.ToString()
|
||||
},
|
||||
ActualCounts = new BudgetActualCounts
|
||||
{
|
||||
Total = budgetResult.ActualCount,
|
||||
CumulativeUncertainty = budgetResult.ActualCumulativeUncertainty,
|
||||
ByReason = budgetResult.CountsByReason
|
||||
},
|
||||
Result = budgetResult.Passed ? "pass" : budgetResult.Budget.Action.ToString(),
|
||||
ConfigHash = ComputeBudgetConfigHash(budgetResult.Budget),
|
||||
Violations = budgetResult.Violations?.ToList()
|
||||
};
|
||||
}
|
||||
|
||||
private static string ComputeBudgetConfigHash(UnknownBudget budget)
|
||||
{
|
||||
var json = JsonSerializer.Serialize(budget, CanonicalJsonOptions);
|
||||
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(json));
|
||||
return $"sha256:{Convert.ToHexString(hash).ToLowerInvariant()}";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Predicates/BudgetCheckPredicate.cs` | Create |
|
||||
| `src/Policy/__Libraries/StellaOps.Policy.Unknowns/Models/BudgetCheckResult.cs` | Create/Enhance |
|
||||
| `src/Policy/__Libraries/StellaOps.Policy.Unknowns/Services/UnknownBudgetService.cs` | Modify to return BudgetCheckResult |
|
||||
| `src/Policy/__Libraries/StellaOps.Policy/Gates/PolicyGateContext.cs` | Add BudgetCheckResult field |
|
||||
| `src/Policy/StellaOps.Policy.Engine/Attestation/VerdictPredicateBuilder.cs` | Add budget predicate |
|
||||
| `src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Attestation/BudgetCheckPredicateTests.cs` | Create |
|
||||
| `docs/modules/policy/budget-attestation.md` | Create |
|
||||
|
||||
## Acceptance Criteria
|
||||
1. [ ] BudgetCheckPredicate model created
|
||||
2. [ ] Budget config hash is deterministic
|
||||
3. [ ] Predicate included in verdict attestation
|
||||
4. [ ] Environment, limits, counts, and result captured
|
||||
5. [ ] Violations listed when budget exceeded
|
||||
6. [ ] Tests verify predicate extraction from DSSE
|
||||
7. [ ] Documentation complete
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Budget config changes frequently | Low | Config hash tracks changes; document drift handling | Policy Guild |
|
||||
| Predicate size bloat | Low | Only include essential fields; violations optional | Policy Guild |
|
||||
| Breaking existing attestation consumers | Medium | Add as new field; don't remove existing fields | Policy Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory gap analysis. P6 priority - completes attestation story. | Project Mgmt |
|
||||
508
docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md
Normal file
508
docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md
Normal file
@@ -0,0 +1,508 @@
|
||||
# Feedser Implementation Master Plan
|
||||
|
||||
## Epic: Federated Learning Cache with Provenance-Scoped Deduplication
|
||||
|
||||
**Epoch:** 8200
|
||||
**Module:** FEEDSER (Concelier evolution)
|
||||
**Status:** PLANNING
|
||||
**Created:** 2025-12-24
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Transform Concelier into a **federated, learning cache** with **provenance-scoped deduplication** where:
|
||||
- The same CVE across distros collapses into one signed canonical record
|
||||
- Only advisories that matter to your builds persist (learning from SBOM/VEX/runtime)
|
||||
- Multiple Feedser nodes can share normalized advisories via signed, pull-only sync
|
||||
|
||||
### Expected Outcomes
|
||||
|
||||
| Metric | Target | Mechanism |
|
||||
|--------|--------|-----------|
|
||||
| Duplicate reduction | 40-60% | Semantic merge_hash collapses distro variants |
|
||||
| Read latency (p99) | <20ms | Valkey front-cache for hot advisories |
|
||||
| Relevant dataset | ~5K from 200K+ | Interest scoring + stub degradation |
|
||||
| Federation sync | Pull-only, air-gap friendly | Signed delta bundles with cursors |
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis Summary
|
||||
|
||||
Based on comprehensive codebase analysis, the following gaps were identified:
|
||||
|
||||
### Phase A: Deterministic Core
|
||||
|
||||
| # | Gap | Current State | Implementation |
|
||||
|---|-----|---------------|----------------|
|
||||
| A1 | **Semantic merge_hash** | `CanonicalHashCalculator` computes SHA256 over full JSON | New identity-based hash: `hash(cve + purl + version-range + weakness + patch_lineage)` |
|
||||
| A2 | **advisory_canonical + source_edge** | Single `vuln.advisories` table | Two-table structure for multi-source attribution |
|
||||
| A3 | **DSSE per source edge** | Dual-sign exists but not on edges | Each source edge carries signature |
|
||||
|
||||
### Phase B: Learning Cache
|
||||
|
||||
| # | Gap | Current State | Implementation |
|
||||
|---|-----|---------------|----------------|
|
||||
| B1 | **interest_score table** | No per-advisory scoring | Score based on SBOM/VEX/runtime intersection |
|
||||
| B2 | **SBOM intersection scoring** | Scanner has BOM Index | `/learn/sbom` endpoint updates scores |
|
||||
| B3 | **Valkey advisory cache** | Valkey used for Gateway messaging only | Hot keys `advisory:{merge_hash}`, `rank:hot` |
|
||||
| B4 | **Stub degradation** | No concept | Low-score advisories become lightweight stubs |
|
||||
|
||||
### Phase C: Federation
|
||||
|
||||
| # | Gap | Current State | Implementation |
|
||||
|---|-----|---------------|----------------|
|
||||
| C1 | **sync_ledger table** | None | Track site_id, cursor, bundle_hash |
|
||||
| C2 | **Delta bundle export** | `AirgapBundleBuilder` exists, no cursors | Add cursor-based delta export |
|
||||
| C3 | **Bundle import/merge** | Import exists, no merge | Add verify + apply + merge logic |
|
||||
|
||||
### Phase D: Backport Precision
|
||||
|
||||
| # | Gap | Current State | Implementation |
|
||||
|---|-----|---------------|----------------|
|
||||
| D1 | **provenance_scope table** | None | Track backport_semver, patch_id, evidence |
|
||||
| D2 | **BackportProofService integration** | 4-tier evidence exists separately | Wire into canonical merge decision |
|
||||
|
||||
---
|
||||
|
||||
## Sprint Roadmap
|
||||
|
||||
```
|
||||
Phase A (Weeks 1-4): Deterministic Core
|
||||
├── SPRINT_8200_0012_0001_CONCEL_merge_hash_library
|
||||
├── SPRINT_8200_0012_0002_DB_canonical_source_edge_schema
|
||||
└── SPRINT_8200_0012_0003_CONCEL_canonical_advisory_service
|
||||
|
||||
Phase B (Weeks 5-8): Learning Cache
|
||||
├── SPRINT_8200_0013_0001_GW_valkey_advisory_cache
|
||||
├── SPRINT_8200_0013_0002_CONCEL_interest_scoring
|
||||
└── SPRINT_8200_0013_0003_SCAN_sbom_intersection_scoring
|
||||
|
||||
Phase C (Weeks 9-12): Federation
|
||||
├── SPRINT_8200_0014_0001_DB_sync_ledger_schema
|
||||
├── SPRINT_8200_0014_0002_CONCEL_delta_bundle_export
|
||||
└── SPRINT_8200_0014_0003_CONCEL_bundle_import_merge
|
||||
|
||||
Phase D (Weeks 13-14): Backport Precision
|
||||
├── SPRINT_8200_0015_0001_DB_provenance_scope_schema
|
||||
└── SPRINT_8200_0015_0002_CONCEL_backport_integration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema Overview
|
||||
|
||||
### New Tables (vuln schema)
|
||||
|
||||
```sql
|
||||
-- Phase A: Canonical/Source Edge Model
|
||||
CREATE TABLE vuln.advisory_canonical (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
cve TEXT NOT NULL,
|
||||
affects_key TEXT NOT NULL, -- normalized purl|cpe
|
||||
version_range JSONB, -- structured range
|
||||
weakness TEXT[], -- CWE set
|
||||
merge_hash TEXT NOT NULL UNIQUE, -- deterministic identity hash
|
||||
status TEXT DEFAULT 'active' CHECK (status IN ('active', 'stub')),
|
||||
epss_score NUMERIC(5,4), -- optional EPSS
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE vuln.advisory_source_edge (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
source_id UUID NOT NULL REFERENCES vuln.sources(id),
|
||||
vendor_status TEXT CHECK (vendor_status IN ('affected', 'not_affected', 'fixed', 'under_investigation')),
|
||||
source_doc_hash TEXT NOT NULL, -- SHA256 of source document
|
||||
dsse_envelope JSONB, -- DSSE signature envelope
|
||||
precedence_rank INT NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (canonical_id, source_id, source_doc_hash)
|
||||
);
|
||||
|
||||
-- Phase B: Interest Scoring
|
||||
CREATE TABLE vuln.interest_score (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE UNIQUE,
|
||||
score NUMERIC(3,2) NOT NULL CHECK (score >= 0 AND score <= 1),
|
||||
reasons JSONB NOT NULL DEFAULT '[]', -- ['in_sbom', 'reachable', 'deployed']
|
||||
last_seen_in_build UUID, -- FK to scanner.scan_manifest
|
||||
computed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_interest_score_score ON vuln.interest_score(score DESC);
|
||||
CREATE INDEX idx_interest_score_canonical ON vuln.interest_score(canonical_id);
|
||||
|
||||
-- Phase C: Sync Ledger
|
||||
CREATE TABLE vuln.sync_ledger (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
site_id TEXT NOT NULL,
|
||||
cursor TEXT NOT NULL,
|
||||
bundle_hash TEXT NOT NULL,
|
||||
signed_at TIMESTAMPTZ NOT NULL,
|
||||
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
items_count INT NOT NULL DEFAULT 0,
|
||||
UNIQUE (site_id, cursor)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sync_ledger_site ON vuln.sync_ledger(site_id, signed_at DESC);
|
||||
|
||||
-- Phase D: Provenance Scope
|
||||
CREATE TABLE vuln.provenance_scope (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
distro_release TEXT, -- e.g., 'debian:bookworm', 'rhel:9'
|
||||
backport_semver TEXT, -- distro-specific backported version
|
||||
patch_id TEXT, -- upstream commit/patch reference
|
||||
evidence_ref UUID, -- FK to proofchain.proof_entries
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (canonical_id, distro_release)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_provenance_scope_canonical ON vuln.provenance_scope(canonical_id);
|
||||
CREATE INDEX idx_provenance_scope_distro ON vuln.provenance_scope(distro_release);
|
||||
```
|
||||
|
||||
### Valkey Keys (Phase B)
|
||||
|
||||
```
|
||||
advisory:{merge_hash} -> JSON canonical advisory (TTL by score)
|
||||
rank:hot -> ZSET of merge_hash by interest_score
|
||||
by:purl:{normalized_purl} -> SET of merge_hash affecting this purl
|
||||
by:cve:{cve_id} -> merge_hash for this CVE
|
||||
cache:ttl:high -> 24h (score >= 0.7)
|
||||
cache:ttl:medium -> 4h (score >= 0.4)
|
||||
cache:ttl:low -> 1h (score < 0.4)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Phase A Endpoints
|
||||
|
||||
```yaml
|
||||
# Advisory canonical read
|
||||
GET /api/v1/advisories/{canonical_id}
|
||||
Response: CanonicalAdvisory + SourceEdges + ProvenanceScopes
|
||||
|
||||
GET /api/v1/advisories?artifact_id={purl|cpe}
|
||||
Response: Deduplicated set of relevant canonical advisories
|
||||
|
||||
# Ingest with merge decision
|
||||
POST /api/v1/ingest/{source_type} # osv, rhsa, dsa, ghsa, nvd
|
||||
Request: Raw advisory document
|
||||
Response: { canonical_id, merge_decision, signature_ref }
|
||||
```
|
||||
|
||||
### Phase B Endpoints
|
||||
|
||||
```yaml
|
||||
# SBOM learning
|
||||
POST /api/v1/learn/sbom
|
||||
Request: { artifact_id, sbom_digest }
|
||||
Response: { updated_count, new_scores[] }
|
||||
|
||||
# Runtime signal learning
|
||||
POST /api/v1/learn/runtime
|
||||
Request: { artifact_id, signals[] }
|
||||
Response: { updated_count }
|
||||
|
||||
# Hot advisory query
|
||||
GET /api/v1/advisories/hot?limit=100
|
||||
Response: Top N by interest_score
|
||||
```
|
||||
|
||||
### Phase C Endpoints
|
||||
|
||||
```yaml
|
||||
# Bundle export with cursor
|
||||
GET /api/v1/federation/export?since_cursor={cursor}
|
||||
Response: Delta bundle (ZST) + new cursor
|
||||
|
||||
# Bundle import
|
||||
POST /api/v1/federation/import
|
||||
Request: Signed bundle
|
||||
Response: { imported, updated, skipped, cursor }
|
||||
|
||||
# Site status
|
||||
GET /api/v1/federation/sites
|
||||
Response: Known sites + cursors
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Merge Hash Algorithm
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Computes deterministic identity hash for canonical advisory deduplication.
|
||||
/// Same CVE + same affected package + same version semantics = same hash.
|
||||
/// </summary>
|
||||
public static string ComputeMergeHash(
|
||||
string cve,
|
||||
string affectsKey, // normalized purl or cpe
|
||||
VersionRange? versionRange,
|
||||
IReadOnlyList<string> weaknesses,
|
||||
string? patchLineage) // upstream patch provenance
|
||||
{
|
||||
// Normalize inputs
|
||||
var normalizedCve = cve.ToUpperInvariant().Trim();
|
||||
var normalizedAffects = NormalizePurlOrCpe(affectsKey);
|
||||
var normalizedRange = NormalizeVersionRange(versionRange);
|
||||
var normalizedWeaknesses = weaknesses
|
||||
.Select(w => w.ToUpperInvariant().Trim())
|
||||
.OrderBy(w => w, StringComparer.Ordinal)
|
||||
.ToArray();
|
||||
var normalizedLineage = NormalizePatchLineage(patchLineage);
|
||||
|
||||
// Build canonical string
|
||||
var builder = new StringBuilder();
|
||||
builder.Append(normalizedCve);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedAffects);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedRange);
|
||||
builder.Append('|');
|
||||
builder.Append(string.Join(",", normalizedWeaknesses));
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedLineage ?? "");
|
||||
|
||||
// SHA256 hash
|
||||
var bytes = Encoding.UTF8.GetBytes(builder.ToString());
|
||||
var hash = SHA256.HashData(bytes);
|
||||
return Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interest Scoring Algorithm
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Computes interest score for advisory based on org-specific signals.
|
||||
/// </summary>
|
||||
public static InterestScore ComputeInterestScore(
|
||||
Guid canonicalId,
|
||||
IReadOnlyList<SbomMatch> sbomMatches,
|
||||
IReadOnlyList<RuntimeSignal> runtimeSignals,
|
||||
IReadOnlyList<VexStatement> vexStatements,
|
||||
DateTimeOffset? lastSeenInBuild)
|
||||
{
|
||||
var reasons = new List<string>();
|
||||
var weights = new Dictionary<string, double>
|
||||
{
|
||||
["in_sbom"] = 0.30,
|
||||
["reachable"] = 0.25,
|
||||
["deployed"] = 0.20,
|
||||
["no_vex_na"] = 0.15,
|
||||
["age_decay"] = 0.10
|
||||
};
|
||||
|
||||
double score = 0.0;
|
||||
|
||||
// Factor 1: In SBOM (30%)
|
||||
if (sbomMatches.Any())
|
||||
{
|
||||
score += weights["in_sbom"];
|
||||
reasons.Add("in_sbom");
|
||||
}
|
||||
|
||||
// Factor 2: Reachable (25%)
|
||||
var reachableMatches = sbomMatches.Where(m => m.IsReachable).ToList();
|
||||
if (reachableMatches.Any())
|
||||
{
|
||||
score += weights["reachable"];
|
||||
reasons.Add("reachable");
|
||||
}
|
||||
|
||||
// Factor 3: Deployed (20%)
|
||||
var deployedMatches = sbomMatches.Where(m => m.IsDeployed).ToList();
|
||||
if (deployedMatches.Any())
|
||||
{
|
||||
score += weights["deployed"];
|
||||
reasons.Add("deployed");
|
||||
}
|
||||
|
||||
// Factor 4: No VEX Not-Affected (15%)
|
||||
var hasNotAffected = vexStatements.Any(v => v.Status == VexStatus.NotAffected);
|
||||
if (!hasNotAffected)
|
||||
{
|
||||
score += weights["no_vex_na"];
|
||||
reasons.Add("no_vex_na");
|
||||
}
|
||||
|
||||
// Factor 5: Age decay (10%) - newer is better
|
||||
if (lastSeenInBuild.HasValue)
|
||||
{
|
||||
var age = DateTimeOffset.UtcNow - lastSeenInBuild.Value;
|
||||
var decayFactor = Math.Max(0, 1 - (age.TotalDays / 365)); // Linear decay over 1 year
|
||||
score += weights["age_decay"] * decayFactor;
|
||||
if (decayFactor > 0.5) reasons.Add("recent");
|
||||
}
|
||||
|
||||
return new InterestScore
|
||||
{
|
||||
CanonicalId = canonicalId,
|
||||
Score = Math.Round(score, 2),
|
||||
Reasons = reasons.ToArray(),
|
||||
ComputedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Golden Corpora (Phase A)
|
||||
|
||||
| Corpus | Purpose | Source |
|
||||
|--------|---------|--------|
|
||||
| `dedup-debian-rhel-cve-2024.json` | Same CVE, different distro packaging | Debian DSA + RHSA for same CVE |
|
||||
| `dedup-backport-variants.json` | Backport-aware merging | Alpine/SUSE backports |
|
||||
| `dedup-alias-collision.json` | Alias-driven vs merge_hash dedup | GHSA → CVE mapping conflicts |
|
||||
|
||||
### Determinism Tests
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(GoldenCorpora))]
|
||||
public void MergeHash_SameInputs_SameOutput(GoldenCorpusItem item)
|
||||
{
|
||||
// Arrange: Parse advisories from different sources
|
||||
var advisory1 = ParseAdvisory(item.Source1);
|
||||
var advisory2 = ParseAdvisory(item.Source2);
|
||||
|
||||
// Act: Compute merge hashes
|
||||
var hash1 = MergeHashCalculator.Compute(advisory1);
|
||||
var hash2 = MergeHashCalculator.Compute(advisory2);
|
||||
|
||||
// Assert: Same identity = same hash
|
||||
if (item.ExpectedSameCanonical)
|
||||
{
|
||||
Assert.Equal(hash1, hash2);
|
||||
}
|
||||
else
|
||||
{
|
||||
Assert.NotEqual(hash1, hash2);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Federation Replay Tests
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task BundleImport_ProducesDeterministicState()
|
||||
{
|
||||
// Arrange: Export bundle from Site A
|
||||
var bundleA = await _siteA.ExportBundleAsync(cursor: null);
|
||||
|
||||
// Act: Import to Site B (empty)
|
||||
await _siteB.ImportBundleAsync(bundleA);
|
||||
|
||||
// Assert: Sites have identical canonical advisories
|
||||
var advisoriesA = await _siteA.GetAllCanonicalsAsync();
|
||||
var advisoriesB = await _siteB.GetAllCanonicalsAsync();
|
||||
|
||||
Assert.Equal(
|
||||
advisoriesA.Select(a => a.MergeHash).OrderBy(h => h),
|
||||
advisoriesB.Select(a => a.MergeHash).OrderBy(h => h));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### External Dependencies
|
||||
|
||||
| Dependency | Version | Purpose |
|
||||
|------------|---------|---------|
|
||||
| `StackExchange.Redis` | 2.8+ | Valkey client |
|
||||
| `ZstdSharp` | 0.8+ | Bundle compression |
|
||||
| `Microsoft.AspNetCore.OutputCaching` | 10.0 | Response caching |
|
||||
|
||||
### Internal Dependencies
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `StellaOps.Concelier.Core` | Base advisory models |
|
||||
| `StellaOps.Concelier.Merge` | Existing merge infrastructure |
|
||||
| `StellaOps.Concelier.ProofService` | BackportProofService |
|
||||
| `StellaOps.Attestor.Envelope` | DSSE envelope handling |
|
||||
| `StellaOps.Scanner.Core` | SBOM models, BOM Index |
|
||||
| `StellaOps.Excititor.Core` | VEX observation models |
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase A Complete When
|
||||
|
||||
- [ ] `MergeHashCalculator` produces deterministic hashes for golden corpus
|
||||
- [ ] `advisory_canonical` + `advisory_source_edge` tables created and populated
|
||||
- [ ] Existing advisories migrated to canonical model
|
||||
- [ ] Source edges carry DSSE signatures
|
||||
- [ ] API returns deduplicated canonicals
|
||||
|
||||
### Phase B Complete When
|
||||
|
||||
- [ ] Valkey advisory cache operational with TTL-by-score
|
||||
- [ ] `/learn/sbom` updates interest scores
|
||||
- [ ] Interest scores affect cache TTL
|
||||
- [ ] Stub degradation working for low-score advisories
|
||||
- [ ] p99 read latency < 20ms from Valkey
|
||||
|
||||
### Phase C Complete When
|
||||
|
||||
- [ ] `sync_ledger` tracks federation state
|
||||
- [ ] Delta bundle export with cursors working
|
||||
- [ ] Bundle import verifies + merges correctly
|
||||
- [ ] Two test sites can sync bidirectionally
|
||||
- [ ] Air-gap bundle transfer works via file
|
||||
|
||||
### Phase D Complete When
|
||||
|
||||
- [ ] `provenance_scope` tracks distro backports
|
||||
- [ ] `BackportProofService` evidence flows into merge decisions
|
||||
- [ ] Backport-aware dedup produces correct results
|
||||
- [ ] Policy lattice configurable for vendor vs distro precedence
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Merge hash breaks existing identity | Data migration failure | Shadow-write both hashes during transition; validate before cutover |
|
||||
| Valkey unavailable | Read latency spike | Fallback to Postgres with degraded TTL |
|
||||
| Federation merge conflicts | Data divergence | Deterministic conflict resolution; audit log all decisions |
|
||||
| Interest scoring bias | Wrong advisories prioritized | Configurable weights; audit score changes |
|
||||
| Backport evidence incomplete | False negatives | Multi-tier fallback (advisory → changelog → patch → binary) |
|
||||
|
||||
---
|
||||
|
||||
## Owners
|
||||
|
||||
| Role | Team | Responsibilities |
|
||||
|------|------|------------------|
|
||||
| Technical Lead | Concelier Guild | Architecture decisions, merge algorithm design |
|
||||
| Database Engineer | Platform Guild | Schema migrations, query optimization |
|
||||
| Backend Engineer | Concelier Guild | Service implementation, API design |
|
||||
| Integration Engineer | Scanner Guild | SBOM scoring integration |
|
||||
| QA Engineer | QA Guild | Golden corpus, determinism tests |
|
||||
| Docs Engineer | Docs Guild | API documentation, migration guide |
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- `docs/modules/concelier/README.md` - Module architecture
|
||||
- `docs/modules/concelier/operations/connectors/` - Connector runbooks
|
||||
- `docs/db/SPECIFICATION.md` - Database specification
|
||||
- `docs/24_OFFLINE_KIT.md` - Air-gap operations
|
||||
- `SPRINT_8100_0011_0003_gateway_valkey_messaging_transport.md` - Valkey infrastructure
|
||||
261
docs/implplan/SPRINT_8200_0012_0001_CONCEL_merge_hash_library.md
Normal file
261
docs/implplan/SPRINT_8200_0012_0001_CONCEL_merge_hash_library.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Sprint 8200.0012.0001 - Merge Hash Library
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement the **deterministic semantic merge_hash** algorithm that enables provenance-scoped deduplication. This sprint delivers:
|
||||
|
||||
1. **MergeHashCalculator**: Compute identity-based hash from `(cve + purl + version-range + weakness + patch_lineage)`
|
||||
2. **Normalization Helpers**: Canonicalize PURLs, CPEs, version ranges, and CWE identifiers
|
||||
3. **Golden Corpus Tests**: Validate determinism across Debian/RHEL/SUSE/Alpine/Astra variants
|
||||
4. **Migration Path**: Shadow-write merge_hash alongside existing content hash
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Merge/`
|
||||
|
||||
**Evidence:** Golden corpus tests pass; same CVE from different distros produces identical merge_hash when semantically equivalent.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Master plan approval, existing `CanonicalHashCalculator` implementation
|
||||
- **Blocks:** SPRINT_8200_0012_0002 (schema), SPRINT_8200_0012_0003 (service)
|
||||
- **Safe to run in parallel with:** Nothing (foundational)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- `src/Concelier/__Libraries/StellaOps.Concelier.Models/CANONICAL_RECORDS.md`
|
||||
- `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/CanonicalHashCalculator.cs`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Design & Setup** | | | | | |
|
||||
| 0 | MHASH-8200-000 | TODO | Master plan | Platform Guild | Review existing `CanonicalHashCalculator` and document differences from semantic merge_hash |
|
||||
| 1 | MHASH-8200-001 | TODO | Task 0 | Concelier Guild | Create `StellaOps.Concelier.Merge.Identity` namespace and project structure |
|
||||
| 2 | MHASH-8200-002 | TODO | Task 1 | Concelier Guild | Define `IMergeHashCalculator` interface with `ComputeMergeHash()` method |
|
||||
| **Wave 1: Normalization Helpers** | | | | | |
|
||||
| 3 | MHASH-8200-003 | TODO | Task 2 | Concelier Guild | Implement `PurlNormalizer.Normalize(string purl)` - lowercase, sort qualifiers, strip checksums |
|
||||
| 4 | MHASH-8200-004 | TODO | Task 2 | Concelier Guild | Implement `CpeNormalizer.Normalize(string cpe)` - canonical CPE 2.3 format |
|
||||
| 5 | MHASH-8200-005 | TODO | Task 2 | Concelier Guild | Implement `VersionRangeNormalizer.Normalize(VersionRange range)` - canonical range expression |
|
||||
| 6 | MHASH-8200-006 | TODO | Task 2 | Concelier Guild | Implement `CweNormalizer.Normalize(IEnumerable<string> cwes)` - uppercase, sorted, deduplicated |
|
||||
| 7 | MHASH-8200-007 | TODO | Task 2 | Concelier Guild | Implement `PatchLineageNormalizer.Normalize(string? lineage)` - extract upstream commit refs |
|
||||
| 8 | MHASH-8200-008 | TODO | Tasks 3-7 | QA Guild | Unit tests for each normalizer with edge cases (empty, malformed, unicode) |
|
||||
| **Wave 2: Core Hash Calculator** | | | | | |
|
||||
| 9 | MHASH-8200-009 | TODO | Tasks 3-7 | Concelier Guild | Implement `MergeHashCalculator.ComputeMergeHash()` combining all normalizers |
|
||||
| 10 | MHASH-8200-010 | TODO | Task 9 | Concelier Guild | Implement canonical string builder with deterministic field ordering |
|
||||
| 11 | MHASH-8200-011 | TODO | Task 10 | Concelier Guild | Implement SHA256 hash computation with hex encoding |
|
||||
| 12 | MHASH-8200-012 | TODO | Task 11 | QA Guild | Add unit tests for hash determinism (same inputs = same output across runs) |
|
||||
| **Wave 3: Golden Corpus Validation** | | | | | |
|
||||
| 13 | MHASH-8200-013 | TODO | Task 12 | QA Guild | Create `dedup-debian-rhel-cve-2024.json` corpus (10+ CVEs with both DSA and RHSA) |
|
||||
| 14 | MHASH-8200-014 | TODO | Task 12 | QA Guild | Create `dedup-backport-variants.json` corpus (Alpine/SUSE backports) |
|
||||
| 15 | MHASH-8200-015 | TODO | Task 12 | QA Guild | Create `dedup-alias-collision.json` corpus (GHSA→CVE mapping edge cases) |
|
||||
| 16 | MHASH-8200-016 | TODO | Tasks 13-15 | QA Guild | Implement `MergeHashGoldenCorpusTests` with expected hash assertions |
|
||||
| 17 | MHASH-8200-017 | TODO | Task 16 | QA Guild | Add fuzzing tests for malformed version ranges and unusual PURLs |
|
||||
| **Wave 4: Integration & Migration** | | | | | |
|
||||
| 18 | MHASH-8200-018 | TODO | Task 12 | Concelier Guild | Add `MergeHash` property to `Advisory` domain model (nullable during migration) |
|
||||
| 19 | MHASH-8200-019 | TODO | Task 18 | Concelier Guild | Modify `AdvisoryMergeService` to compute and store merge_hash during merge |
|
||||
| 20 | MHASH-8200-020 | TODO | Task 19 | Concelier Guild | Add shadow-write mode: compute merge_hash for existing advisories without changing identity |
|
||||
| 21 | MHASH-8200-021 | TODO | Task 20 | QA Guild | Integration test: ingest same CVE from two connectors, verify same merge_hash |
|
||||
| 22 | MHASH-8200-022 | TODO | Task 21 | Docs Guild | Document merge_hash algorithm in `CANONICAL_RECORDS.md` |
|
||||
|
||||
---
|
||||
|
||||
## API Design
|
||||
|
||||
### IMergeHashCalculator Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Merge.Identity;
|
||||
|
||||
/// <summary>
|
||||
/// Computes deterministic semantic merge hash for advisory deduplication.
|
||||
/// </summary>
|
||||
public interface IMergeHashCalculator
|
||||
{
|
||||
/// <summary>
|
||||
/// Compute merge hash from advisory identity components.
|
||||
/// </summary>
|
||||
string ComputeMergeHash(MergeHashInput input);
|
||||
|
||||
/// <summary>
|
||||
/// Compute merge hash directly from Advisory domain model.
|
||||
/// </summary>
|
||||
string ComputeMergeHash(Advisory advisory);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Input components for merge hash computation.
|
||||
/// </summary>
|
||||
public sealed record MergeHashInput
|
||||
{
|
||||
/// <summary>CVE identifier (e.g., "CVE-2024-1234"). Required.</summary>
|
||||
public required string Cve { get; init; }
|
||||
|
||||
/// <summary>Affected package identifier (PURL or CPE). Required.</summary>
|
||||
public required string AffectsKey { get; init; }
|
||||
|
||||
/// <summary>Affected version range. Optional.</summary>
|
||||
public VersionRange? VersionRange { get; init; }
|
||||
|
||||
/// <summary>Associated CWE identifiers. Optional.</summary>
|
||||
public IReadOnlyList<string> Weaknesses { get; init; } = [];
|
||||
|
||||
/// <summary>Upstream patch provenance (commit SHA, patch ID). Optional.</summary>
|
||||
public string? PatchLineage { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Normalizer Interfaces
|
||||
|
||||
```csharp
|
||||
public interface IPurlNormalizer
|
||||
{
|
||||
/// <summary>Normalize PURL to canonical form.</summary>
|
||||
string Normalize(string purl);
|
||||
}
|
||||
|
||||
public interface ICpeNormalizer
|
||||
{
|
||||
/// <summary>Normalize CPE to canonical CPE 2.3 format.</summary>
|
||||
string Normalize(string cpe);
|
||||
}
|
||||
|
||||
public interface IVersionRangeNormalizer
|
||||
{
|
||||
/// <summary>Normalize version range to canonical expression.</summary>
|
||||
string Normalize(VersionRange? range);
|
||||
}
|
||||
|
||||
public interface ICweNormalizer
|
||||
{
|
||||
/// <summary>Normalize CWE list to sorted, deduplicated, uppercase set.</summary>
|
||||
string Normalize(IEnumerable<string> cwes);
|
||||
}
|
||||
|
||||
public interface IPatchLineageNormalizer
|
||||
{
|
||||
/// <summary>Normalize patch lineage to canonical commit reference.</summary>
|
||||
string? Normalize(string? lineage);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Normalization Rules
|
||||
|
||||
### PURL Normalization
|
||||
|
||||
| Input | Output | Rule |
|
||||
|-------|--------|------|
|
||||
| `pkg:NPM/@angular/core@14.0.0` | `pkg:npm/%40angular/core@14.0.0` | Lowercase type, encode @ |
|
||||
| `pkg:maven/org.apache/commons@1.0?type=jar` | `pkg:maven/org.apache/commons@1.0` | Strip type qualifier |
|
||||
| `pkg:deb/debian/curl@7.68.0-1+deb10u1?arch=amd64` | `pkg:deb/debian/curl@7.68.0-1+deb10u1` | Strip arch qualifier |
|
||||
|
||||
### CPE Normalization
|
||||
|
||||
| Input | Output | Rule |
|
||||
|-------|--------|------|
|
||||
| `cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*` | `cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*` | Canonical CPE 2.3 |
|
||||
| `cpe:/a:vendor:product:1.0` | `cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*` | Convert CPE 2.2 |
|
||||
|
||||
### Version Range Normalization
|
||||
|
||||
| Input | Output | Rule |
|
||||
|-------|--------|------|
|
||||
| `[1.0.0, 2.0.0)` | `>=1.0.0,<2.0.0` | Canonical interval notation |
|
||||
| `< 1.5.0` | `<1.5.0` | Trim whitespace |
|
||||
| Fixed: 1.5.1 | `>=1.5.1` | Convert "fixed" to range |
|
||||
|
||||
### CWE Normalization
|
||||
|
||||
| Input | Output | Rule |
|
||||
|-------|--------|------|
|
||||
| `['cwe-79', 'CWE-89']` | `CWE-79,CWE-89` | Uppercase, sorted, comma-joined |
|
||||
| `['CWE-89', 'CWE-89']` | `CWE-89` | Deduplicated |
|
||||
|
||||
---
|
||||
|
||||
## Golden Corpus Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"corpus": "dedup-debian-rhel-cve-2024",
|
||||
"version": "1.0.0",
|
||||
"items": [
|
||||
{
|
||||
"id": "CVE-2024-1234-curl",
|
||||
"description": "Same curl CVE from Debian and RHEL",
|
||||
"sources": [
|
||||
{
|
||||
"source": "debian",
|
||||
"advisory_id": "DSA-5678-1",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:deb/debian/curl@7.68.0-1+deb10u1",
|
||||
"version_range": "<7.68.0-1+deb10u2",
|
||||
"weaknesses": ["CWE-120"]
|
||||
},
|
||||
{
|
||||
"source": "redhat",
|
||||
"advisory_id": "RHSA-2024:1234",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:rpm/redhat/curl@7.61.1-22.el8",
|
||||
"version_range": "<7.61.1-22.el8_6.1",
|
||||
"weaknesses": ["CWE-120"]
|
||||
}
|
||||
],
|
||||
"expected": {
|
||||
"same_canonical": true,
|
||||
"expected_merge_hash": "a1b2c3d4e5f6...",
|
||||
"rationale": "Same CVE, same CWE, both are curl packages affected by same upstream issue"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Evidence Requirements
|
||||
|
||||
| Test Category | Evidence |
|
||||
|---------------|----------|
|
||||
| Normalizer unit tests | All normalizers handle empty, null, malformed, unicode inputs |
|
||||
| Hash determinism | 100 runs of same input produce identical hash |
|
||||
| Golden corpus | All expected same_canonical pairs produce identical merge_hash |
|
||||
| Fuzz testing | 1000 random inputs don't cause exceptions |
|
||||
| Migration shadow-write | Existing advisories gain merge_hash without identity change |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Use semantic identity, not content hash | Content hash changes on any field; semantic hash is stable |
|
||||
| Include patch lineage in hash | Differentiates distro backports from upstream fixes |
|
||||
| Exclude CVSS from hash | CVSS varies by assessor; doesn't change advisory identity |
|
||||
| Exclude severity from hash | Derived from CVSS; not part of identity |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Normalization edge cases | Hash collision or divergence | Extensive golden corpus + fuzz testing |
|
||||
| PURL ecosystem variations | Different ecosystems need different normalization | Per-ecosystem normalizer with registry |
|
||||
| Backport detection failure | Wrong canonical grouping | Multi-tier evidence from BackportProofService |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,389 @@
|
||||
# Sprint 8200.0012.0001 · Evidence-Weighted Score Core Library
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement a **unified evidence-weighted scoring model** that combines six evidence dimensions (Reachability, Runtime, Backport, Exploit, Source Trust, Mitigations) into a single 0-100 score per vulnerability finding. This enables rapid triage by surfacing the most "real" risks with transparent, decomposable evidence.
|
||||
|
||||
This sprint delivers:
|
||||
|
||||
1. **EvidenceWeightedScoreCalculator**: Core formula implementation with configurable weights
|
||||
2. **Score Input Models**: Normalized 0-1 inputs for all six dimensions
|
||||
3. **Score Result Models**: API response shape with decomposition, flags, explanations, caps
|
||||
4. **Guardrails Engine**: Hard caps/floors based on evidence conditions
|
||||
5. **Weight Policy Configuration**: Environment-specific weight profiles (prod/staging/dev)
|
||||
6. **Determinism Guarantees**: Same inputs → same score, policy versioning with digest
|
||||
|
||||
**Working directory:** `src/Signals/StellaOps.Signals/EvidenceWeightedScore/` (new), `src/Signals/__Tests/StellaOps.Signals.Tests/EvidenceWeightedScore/` (tests)
|
||||
|
||||
**Evidence:** Formula produces deterministic 0-100 scores; guardrails apply correctly; weight policies load per-tenant; all property tests pass.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** None (new foundational module)
|
||||
- **Blocks:** Sprint 8200.0012.0002 (Normalizers), Sprint 8200.0012.0003 (Policy Integration), Sprint 8200.0012.0004 (API)
|
||||
- **Safe to run in parallel with:** None initially; foundational sprint
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/product-advisories/evidence-weighted-score-blueprint.md` (this advisory)
|
||||
- `docs/modules/signals/architecture.md` (to be created)
|
||||
- `docs/modules/policy/architecture.md` (existing confidence scoring context)
|
||||
|
||||
---
|
||||
|
||||
## Scoring Model Specification
|
||||
|
||||
### Formula
|
||||
|
||||
```
|
||||
Score = clamp01(
|
||||
W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT
|
||||
) * 100
|
||||
```
|
||||
|
||||
### Input Dimensions (all normalized 0-1)
|
||||
|
||||
| Dimension | Symbol | Description | Source |
|
||||
|-----------|--------|-------------|--------|
|
||||
| Reachability | RCH | Static/dynamic reachability confidence | Policy/ConfidenceCalculator |
|
||||
| Runtime | RTS | Runtime signal strength (eBPF/dyld/ETW hits, recency) | Policy/RuntimeEvidence |
|
||||
| Backport | BKP | Backport/patch evidence strength | Concelier/BackportProofService |
|
||||
| Exploit | XPL | Exploit likelihood (EPSS + KEV) | Scanner/EpssPriorityCalculator |
|
||||
| Source Trust | SRC | Source trust (vendor VEX > distro > community) | Excititor/TrustVector |
|
||||
| Mitigations | MIT | Active mitigations (feature flags, seccomp, isolation) | Policy/GateMultipliers |
|
||||
|
||||
### Default Weights
|
||||
|
||||
```yaml
|
||||
weights:
|
||||
rch: 0.30 # Reachability
|
||||
rts: 0.25 # Runtime
|
||||
bkp: 0.15 # Backport
|
||||
xpl: 0.15 # Exploit
|
||||
src: 0.10 # Source Trust
|
||||
mit: 0.10 # Mitigations (subtractive)
|
||||
```
|
||||
|
||||
### Guardrails
|
||||
|
||||
| Condition | Action | Rationale |
|
||||
|-----------|--------|-----------|
|
||||
| `BKP >= 1.0 && status == "not_affected" && RTS < 0.6` | `Score = min(Score, 15)` | Vendor says not affected, no runtime contradiction |
|
||||
| `RTS >= 0.8` | `Score = max(Score, 60)` | Strong live signal overrides other factors |
|
||||
| `RCH == 0 && RTS == 0` | `Score = min(Score, 45)` | Speculative finding with no proof |
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Project Setup)** | | | | | |
|
||||
| 0 | EWS-8200-000 | TODO | None | Platform Guild | Create `StellaOps.Signals` project structure with proper namespace and package references. |
|
||||
| 1 | EWS-8200-001 | TODO | Task 0 | Platform Guild | Create `StellaOps.Signals.Tests` test project with xUnit, FsCheck (property tests), Verify (snapshots). |
|
||||
| 2 | EWS-8200-002 | TODO | Task 0 | Platform Guild | Create `docs/modules/signals/architecture.md` with module purpose and design rationale. |
|
||||
| **Wave 1 (Input Models)** | | | | | |
|
||||
| 3 | EWS-8200-003 | TODO | Task 0 | Signals Guild | Define `EvidenceWeightedScoreInput` record with all six normalized dimensions (RCH, RTS, BKP, XPL, SRC, MIT). |
|
||||
| 4 | EWS-8200-004 | TODO | Task 3 | Signals Guild | Add input validation: all values clamped [0, 1], null handling with defaults. |
|
||||
| 5 | EWS-8200-005 | TODO | Task 3 | Signals Guild | Define `ReachabilityInput` with state enum, confidence, hop count, gate flags. |
|
||||
| 6 | EWS-8200-006 | TODO | Task 3 | Signals Guild | Define `RuntimeInput` with posture, observation count, recency, session digests. |
|
||||
| 7 | EWS-8200-007 | TODO | Task 3 | Signals Guild | Define `BackportInput` with evidence tier, proof ID, status (affected/not_affected/fixed). |
|
||||
| 8 | EWS-8200-008 | TODO | Task 3 | Signals Guild | Define `ExploitInput` with EPSS score, EPSS percentile, KEV status, KEV date. |
|
||||
| 9 | EWS-8200-009 | TODO | Task 3 | Signals Guild | Define `SourceTrustInput` with trust vector (provenance, coverage, replayability), issuer type. |
|
||||
| 10 | EWS-8200-010 | TODO | Task 3 | Signals Guild | Define `MitigationInput` with active mitigations list, combined effectiveness score. |
|
||||
| 11 | EWS-8200-011 | TODO | Tasks 5-10 | QA Guild | Add unit tests for all input models: validation, serialization, edge cases. |
|
||||
| **Wave 2 (Weight Configuration)** | | | | | |
|
||||
| 12 | EWS-8200-012 | TODO | Task 0 | Signals Guild | Define `EvidenceWeightPolicy` record with weight values and policy version. |
|
||||
| 13 | EWS-8200-013 | TODO | Task 12 | Signals Guild | Define `EvidenceWeightPolicyOptions` for DI configuration with environment profiles. |
|
||||
| 14 | EWS-8200-014 | TODO | Task 12 | Signals Guild | Implement `IEvidenceWeightPolicyProvider` interface with `GetPolicy(tenantId, environment)`. |
|
||||
| 15 | EWS-8200-015 | TODO | Task 14 | Signals Guild | Implement `FileEvidenceWeightPolicyProvider` loading from YAML with hot-reload support. |
|
||||
| 16 | EWS-8200-016 | TODO | Task 14 | Signals Guild | Implement `InMemoryEvidenceWeightPolicyProvider` for testing. |
|
||||
| 17 | EWS-8200-017 | TODO | Task 12 | Signals Guild | Implement weight normalization: ensure weights sum to 1.0 (excluding MIT which is subtractive). |
|
||||
| 18 | EWS-8200-018 | TODO | Task 12 | Signals Guild | Implement policy digest computation (canonical JSON → SHA256) for determinism tracking. |
|
||||
| 19 | EWS-8200-019 | TODO | Tasks 12-18 | QA Guild | Add unit tests for weight policy: loading, validation, normalization, digest stability. |
|
||||
| **Wave 3 (Core Calculator)** | | | | | |
|
||||
| 20 | EWS-8200-020 | TODO | Tasks 3, 12 | Signals Guild | Define `IEvidenceWeightedScoreCalculator` interface with `Calculate(input, policy)`. |
|
||||
| 21 | EWS-8200-021 | TODO | Task 20 | Signals Guild | Implement `EvidenceWeightedScoreCalculator`: apply formula `W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT`. |
|
||||
| 22 | EWS-8200-022 | TODO | Task 21 | Signals Guild | Implement clamping: result clamped to [0, 1] before multiplying by 100. |
|
||||
| 23 | EWS-8200-023 | TODO | Task 21 | Signals Guild | Implement factor breakdown: return per-dimension contribution for UI decomposition. |
|
||||
| 24 | EWS-8200-024 | TODO | Task 21 | Signals Guild | Implement explanation generation: human-readable summary of top contributing factors. |
|
||||
| 25 | EWS-8200-025 | TODO | Tasks 20-24 | QA Guild | Add unit tests for calculator: formula correctness, edge cases (all zeros, all ones, negatives). |
|
||||
| 26 | EWS-8200-026 | TODO | Tasks 20-24 | QA Guild | Add property tests: score monotonicity (increasing inputs → increasing score), commutativity. |
|
||||
| **Wave 4 (Guardrails)** | | | | | |
|
||||
| 27 | EWS-8200-027 | TODO | Task 21 | Signals Guild | Define `ScoreGuardrailConfig` with cap/floor conditions and thresholds. |
|
||||
| 28 | EWS-8200-028 | TODO | Task 27 | Signals Guild | Implement "not_affected cap": if BKP=1 + not_affected + RTS<0.6 → cap at 15. |
|
||||
| 29 | EWS-8200-029 | TODO | Task 27 | Signals Guild | Implement "runtime floor": if RTS >= 0.8 → floor at 60. |
|
||||
| 30 | EWS-8200-030 | TODO | Task 27 | Signals Guild | Implement "speculative cap": if RCH=0 + RTS=0 → cap at 45. |
|
||||
| 31 | EWS-8200-031 | TODO | Task 27 | Signals Guild | Implement guardrail application order (caps before floors) and conflict resolution. |
|
||||
| 32 | EWS-8200-032 | TODO | Task 27 | Signals Guild | Add `AppliedGuardrails` to result: which caps/floors were triggered and why. |
|
||||
| 33 | EWS-8200-033 | TODO | Tasks 27-32 | QA Guild | Add unit tests for all guardrail conditions and edge cases. |
|
||||
| 34 | EWS-8200-034 | TODO | Tasks 27-32 | QA Guild | Add property tests: guardrails never produce score outside [0, 100]. |
|
||||
| **Wave 5 (Result Models)** | | | | | |
|
||||
| 35 | EWS-8200-035 | TODO | Tasks 21, 27 | Signals Guild | Define `EvidenceWeightedScoreResult` record matching API shape specification. |
|
||||
| 36 | EWS-8200-036 | TODO | Task 35 | Signals Guild | Add `Inputs` property with normalized dimension values (rch, rts, bkp, xpl, src, mit). |
|
||||
| 37 | EWS-8200-037 | TODO | Task 35 | Signals Guild | Add `Weights` property echoing policy weights used for calculation. |
|
||||
| 38 | EWS-8200-038 | TODO | Task 35 | Signals Guild | Add `Flags` property: ["live-signal", "proven-path", "vendor-na", "speculative"]. |
|
||||
| 39 | EWS-8200-039 | TODO | Task 35 | Signals Guild | Add `Explanations` property: list of human-readable evidence explanations. |
|
||||
| 40 | EWS-8200-040 | TODO | Task 35 | Signals Guild | Add `Caps` property: { speculative_cap, not_affected_cap, runtime_floor }. |
|
||||
| 41 | EWS-8200-041 | TODO | Task 35 | Signals Guild | Add `PolicyDigest` property for determinism verification. |
|
||||
| 42 | EWS-8200-042 | TODO | Tasks 35-41 | QA Guild | Add snapshot tests for result JSON structure (canonical format). |
|
||||
| **Wave 6 (Bucket Classification)** | | | | | |
|
||||
| 43 | EWS-8200-043 | TODO | Task 35 | Signals Guild | Define `ScoreBucket` enum: ActNow (90-100), ScheduleNext (70-89), Investigate (40-69), Watchlist (0-39). |
|
||||
| 44 | EWS-8200-044 | TODO | Task 43 | Signals Guild | Implement `GetBucket(score)` with configurable thresholds. |
|
||||
| 45 | EWS-8200-045 | TODO | Task 43 | Signals Guild | Add bucket to result model and explanation. |
|
||||
| 46 | EWS-8200-046 | TODO | Tasks 43-45 | QA Guild | Add unit tests for bucket classification boundary conditions. |
|
||||
| **Wave 7 (DI & Integration)** | | | | | |
|
||||
| 47 | EWS-8200-047 | TODO | All above | Signals Guild | Implement `AddEvidenceWeightedScoring()` extension method for IServiceCollection. |
|
||||
| 48 | EWS-8200-048 | TODO | Task 47 | Signals Guild | Wire policy provider, calculator, and configuration into DI container. |
|
||||
| 49 | EWS-8200-049 | TODO | Task 47 | Signals Guild | Add `IOptionsMonitor<EvidenceWeightPolicyOptions>` for hot-reload support. |
|
||||
| 50 | EWS-8200-050 | TODO | Tasks 47-49 | QA Guild | Add integration tests for full DI pipeline. |
|
||||
| **Wave 8 (Determinism & Quality Gates)** | | | | | |
|
||||
| 51 | EWS-8200-051 | TODO | All above | QA Guild | Add determinism test: same inputs + same policy → identical score and digest. |
|
||||
| 52 | EWS-8200-052 | TODO | All above | QA Guild | Add ordering independence test: input order doesn't affect result. |
|
||||
| 53 | EWS-8200-053 | TODO | All above | QA Guild | Add concurrent calculation test: thread-safe scoring. |
|
||||
| 54 | EWS-8200-054 | TODO | All above | Platform Guild | Add benchmark tests: calculate 10K scores in <1s. |
|
||||
|
||||
---
|
||||
|
||||
## API Design Specification
|
||||
|
||||
### EvidenceWeightedScoreInput
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Normalized inputs for evidence-weighted score calculation.
|
||||
/// All values are [0, 1] where higher = stronger evidence.
|
||||
/// </summary>
|
||||
public sealed record EvidenceWeightedScoreInput
|
||||
{
|
||||
/// <summary>Finding identifier (CVE@PURL format).</summary>
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
/// <summary>Reachability confidence [0, 1]. Higher = more reachable.</summary>
|
||||
public required double Rch { get; init; }
|
||||
|
||||
/// <summary>Runtime signal strength [0, 1]. Higher = stronger live signal.</summary>
|
||||
public required double Rts { get; init; }
|
||||
|
||||
/// <summary>Backport evidence [0, 1]. Higher = stronger patch proof.</summary>
|
||||
public required double Bkp { get; init; }
|
||||
|
||||
/// <summary>Exploit likelihood [0, 1]. Higher = more likely to be exploited.</summary>
|
||||
public required double Xpl { get; init; }
|
||||
|
||||
/// <summary>Source trust [0, 1]. Higher = more trustworthy source.</summary>
|
||||
public required double Src { get; init; }
|
||||
|
||||
/// <summary>Mitigation effectiveness [0, 1]. Higher = stronger mitigations.</summary>
|
||||
public required double Mit { get; init; }
|
||||
|
||||
/// <summary>VEX status for backport guardrail evaluation.</summary>
|
||||
public string? VexStatus { get; init; }
|
||||
|
||||
/// <summary>Detailed inputs for explanation generation.</summary>
|
||||
public ReachabilityInput? ReachabilityDetails { get; init; }
|
||||
public RuntimeInput? RuntimeDetails { get; init; }
|
||||
public BackportInput? BackportDetails { get; init; }
|
||||
public ExploitInput? ExploitDetails { get; init; }
|
||||
public SourceTrustInput? SourceTrustDetails { get; init; }
|
||||
public MitigationInput? MitigationDetails { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### EvidenceWeightedScoreResult
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Result of evidence-weighted score calculation.
|
||||
/// </summary>
|
||||
public sealed record EvidenceWeightedScoreResult
|
||||
{
|
||||
/// <summary>Finding identifier.</summary>
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
/// <summary>Final score [0, 100]. Higher = more evidence of real risk.</summary>
|
||||
public required int Score { get; init; }
|
||||
|
||||
/// <summary>Score bucket for quick triage.</summary>
|
||||
public required ScoreBucket Bucket { get; init; }
|
||||
|
||||
/// <summary>Normalized input values used.</summary>
|
||||
public required EvidenceInputs Inputs { get; init; }
|
||||
|
||||
/// <summary>Weight values used.</summary>
|
||||
public required EvidenceWeights Weights { get; init; }
|
||||
|
||||
/// <summary>Active flags for badges.</summary>
|
||||
public required IReadOnlyList<string> Flags { get; init; }
|
||||
|
||||
/// <summary>Human-readable explanations.</summary>
|
||||
public required IReadOnlyList<string> Explanations { get; init; }
|
||||
|
||||
/// <summary>Applied guardrails (caps/floors).</summary>
|
||||
public required AppliedGuardrails Caps { get; init; }
|
||||
|
||||
/// <summary>Policy digest for determinism verification.</summary>
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
/// <summary>Calculation timestamp (UTC ISO-8601).</summary>
|
||||
public required DateTimeOffset CalculatedAt { get; init; }
|
||||
}
|
||||
|
||||
public sealed record EvidenceInputs(
|
||||
double Rch, double Rts, double Bkp,
|
||||
double Xpl, double Src, double Mit);
|
||||
|
||||
public sealed record EvidenceWeights(
|
||||
double Rch, double Rts, double Bkp,
|
||||
double Xpl, double Src, double Mit);
|
||||
|
||||
public sealed record AppliedGuardrails(
|
||||
bool SpeculativeCap,
|
||||
bool NotAffectedCap,
|
||||
bool RuntimeFloor);
|
||||
|
||||
public enum ScoreBucket
|
||||
{
|
||||
/// <summary>90-100: Act now (live + reachable or KEV).</summary>
|
||||
ActNow = 0,
|
||||
|
||||
/// <summary>70-89: Likely real; schedule next sprint.</summary>
|
||||
ScheduleNext = 1,
|
||||
|
||||
/// <summary>40-69: Investigate when touching component.</summary>
|
||||
Investigate = 2,
|
||||
|
||||
/// <summary>0-39: Low/insufficient evidence; watchlist.</summary>
|
||||
Watchlist = 3
|
||||
}
|
||||
```
|
||||
|
||||
### Weight Policy YAML Schema
|
||||
|
||||
```yaml
|
||||
# score-policy.yaml
|
||||
version: "ews.v1"
|
||||
profile: production
|
||||
|
||||
weights:
|
||||
rch: 0.30
|
||||
rts: 0.25
|
||||
bkp: 0.15
|
||||
xpl: 0.15
|
||||
src: 0.10
|
||||
mit: 0.10
|
||||
|
||||
guardrails:
|
||||
not_affected_cap:
|
||||
enabled: true
|
||||
max_score: 15
|
||||
requires_bkp_min: 1.0
|
||||
requires_rts_max: 0.6
|
||||
runtime_floor:
|
||||
enabled: true
|
||||
min_score: 60
|
||||
requires_rts_min: 0.8
|
||||
speculative_cap:
|
||||
enabled: true
|
||||
max_score: 45
|
||||
requires_rch_max: 0.0
|
||||
requires_rts_max: 0.0
|
||||
|
||||
buckets:
|
||||
act_now_min: 90
|
||||
schedule_next_min: 70
|
||||
investigate_min: 40
|
||||
# Below 40 = watchlist
|
||||
|
||||
environments:
|
||||
production:
|
||||
weights:
|
||||
rch: 0.35
|
||||
rts: 0.30
|
||||
bkp: 0.10
|
||||
xpl: 0.15
|
||||
src: 0.05
|
||||
mit: 0.05
|
||||
development:
|
||||
weights:
|
||||
rch: 0.20
|
||||
rts: 0.15
|
||||
bkp: 0.20
|
||||
xpl: 0.20
|
||||
src: 0.15
|
||||
mit: 0.10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-2 | Project setup | Projects compile, architecture doc exists |
|
||||
| **Wave 1** | 3-11 | Input models | All input types defined, validated, tested |
|
||||
| **Wave 2** | 12-19 | Weight configuration | Policy loading, normalization, digest works |
|
||||
| **Wave 3** | 20-26 | Core calculator | Formula correct, breakdown works, property tests pass |
|
||||
| **Wave 4** | 27-34 | Guardrails | All three guardrails work, edge cases covered |
|
||||
| **Wave 5** | 35-42 | Result models | API shape complete, snapshots stable |
|
||||
| **Wave 6** | 43-46 | Bucket classification | Thresholds correct, boundaries tested |
|
||||
| **Wave 7** | 47-50 | DI integration | Full pipeline works via DI |
|
||||
| **Wave 8** | 51-54 | Determinism gates | All quality gates pass, benchmarks meet target |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint |
|
||||
|-----------|-------------|----------------|
|
||||
| Normalizer inputs | Calculator consumes normalized 0-1 values from Sprint 0002 normalizers | 8200.0012.0002 |
|
||||
| Policy integration | Score result feeds into Policy verdict system | 8200.0012.0003 |
|
||||
| API exposure | Score endpoint returns EvidenceWeightedScoreResult | 8200.0012.0004 |
|
||||
| Determinism | Must match existing determinism guarantees in Policy module | Policy architecture |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming Checkpoints
|
||||
|
||||
| Date (UTC) | Milestone | Evidence |
|
||||
|------------|-----------|----------|
|
||||
| 2026-01-13 | Wave 0-2 complete | Project structure, input models defined |
|
||||
| 2026-01-27 | Wave 3-4 complete | Calculator works, guardrails applied |
|
||||
| 2026-02-10 | Wave 5-6 complete | Result models, buckets working |
|
||||
| 2026-02-24 | Wave 7-8 complete | DI integration, determinism tests pass |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Six-dimension model (RCH, RTS, BKP, XPL, SRC, MIT) | Covers all evidence types from existing infrastructure |
|
||||
| MIT is subtractive | Mitigations reduce risk; they shouldn't contribute positively |
|
||||
| Guardrails are hard caps/floors | Encode domain expertise; prevent edge case scoring |
|
||||
| Policy-driven weights | Different environments have different priorities |
|
||||
| Deterministic by design | Same inputs + policy → same score always |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Weight tuning requires iteration | Suboptimal prioritization | Start with conservative defaults; add telemetry | Signals Guild |
|
||||
| Guardrail conflicts | Unexpected scores | Define clear application order; test extensively | Signals Guild |
|
||||
| Performance at scale | Latency | Benchmark early; optimize hot paths | Platform Guild |
|
||||
| Integration complexity | Sprint delays | Clear interface contracts; mock providers | Project Mgmt |
|
||||
| Existing scoring migration | User confusion | Gradual rollout; feature flag; docs | Product Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from evidence-weighted score product advisory gap analysis. | Project Mgmt |
|
||||
@@ -0,0 +1,440 @@
|
||||
# Sprint 8200.0012.0002 - Canonical Source Edge Schema
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement the **database schema** for the canonical advisory + source edge model. This sprint delivers:
|
||||
|
||||
1. **advisory_canonical table**: Stores deduplicated canonical advisories with merge_hash
|
||||
2. **advisory_source_edge table**: Links canonical records to source documents with DSSE signatures
|
||||
3. **Migration scripts**: Create tables, indexes, and constraints
|
||||
4. **Data migration**: Populate from existing `vuln.advisories` table
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/`
|
||||
|
||||
**Evidence:** Tables created with all constraints; existing advisories migrated; queries execute with expected performance.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0001 (merge_hash library)
|
||||
- **Blocks:** SPRINT_8200_0012_0003 (service layer)
|
||||
- **Safe to run in parallel with:** Nothing (schema must be stable first)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/db/SPECIFICATION.md`
|
||||
- `docs/db/schemas/vuln.sql`
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Schema Design Review** | | | | | |
|
||||
| 0 | SCHEMA-8200-000 | TODO | Master plan | Platform Guild | Review existing `vuln.advisories` schema and document field mapping to canonical model |
|
||||
| 1 | SCHEMA-8200-001 | TODO | Task 0 | Platform Guild | Finalize `advisory_canonical` table design with DBA review |
|
||||
| 2 | SCHEMA-8200-002 | TODO | Task 0 | Platform Guild | Finalize `advisory_source_edge` table design with DSSE envelope storage |
|
||||
| **Wave 1: Migration Scripts** | | | | | |
|
||||
| 3 | SCHEMA-8200-003 | TODO | Tasks 1-2 | Platform Guild | Create migration `20250101000001_CreateAdvisoryCanonical.sql` |
|
||||
| 4 | SCHEMA-8200-004 | TODO | Task 3 | Platform Guild | Create migration `20250101000002_CreateAdvisorySourceEdge.sql` |
|
||||
| 5 | SCHEMA-8200-005 | TODO | Task 4 | Platform Guild | Create migration `20250101000003_CreateCanonicalIndexes.sql` |
|
||||
| 6 | SCHEMA-8200-006 | TODO | Tasks 3-5 | QA Guild | Validate migrations in test environment (create/rollback/recreate) |
|
||||
| **Wave 2: Entity Models** | | | | | |
|
||||
| 7 | SCHEMA-8200-007 | TODO | Task 3 | Concelier Guild | Create `AdvisoryCanonicalEntity` record with all properties |
|
||||
| 8 | SCHEMA-8200-008 | TODO | Task 4 | Concelier Guild | Create `AdvisorySourceEdgeEntity` record with DSSE envelope property |
|
||||
| 9 | SCHEMA-8200-009 | TODO | Tasks 7-8 | Concelier Guild | Create `IAdvisoryCanonicalRepository` interface |
|
||||
| 10 | SCHEMA-8200-010 | TODO | Task 9 | Concelier Guild | Implement `PostgresAdvisoryCanonicalRepository` with CRUD operations |
|
||||
| 11 | SCHEMA-8200-011 | TODO | Task 10 | QA Guild | Unit tests for repository (CRUD, unique constraints, cascade delete) |
|
||||
| **Wave 3: Data Migration** | | | | | |
|
||||
| 12 | SCHEMA-8200-012 | TODO | Tasks 10-11 | Platform Guild | Create data migration script to populate `advisory_canonical` from `vuln.advisories` |
|
||||
| 13 | SCHEMA-8200-013 | TODO | Task 12 | Platform Guild | Create script to create `advisory_source_edge` from existing provenance data |
|
||||
| 14 | SCHEMA-8200-014 | TODO | Task 13 | Platform Guild | Create verification queries to compare record counts and data integrity |
|
||||
| 15 | SCHEMA-8200-015 | TODO | Task 14 | QA Guild | Run data migration in staging environment; validate results |
|
||||
| **Wave 4: Query Optimization** | | | | | |
|
||||
| 16 | SCHEMA-8200-016 | TODO | Task 15 | Platform Guild | Create covering index for `advisory_canonical(merge_hash)` lookups |
|
||||
| 17 | SCHEMA-8200-017 | TODO | Task 15 | Platform Guild | Create index for `advisory_source_edge(canonical_id, source_id)` joins |
|
||||
| 18 | SCHEMA-8200-018 | TODO | Task 15 | Platform Guild | Create partial index for `status = 'active'` queries |
|
||||
| 19 | SCHEMA-8200-019 | TODO | Tasks 16-18 | QA Guild | Benchmark queries: <10ms for merge_hash lookup, <50ms for source edge join |
|
||||
| 20 | SCHEMA-8200-020 | TODO | Task 19 | Docs Guild | Document schema in `docs/db/schemas/vuln.sql` |
|
||||
|
||||
---
|
||||
|
||||
## Schema Specification
|
||||
|
||||
### vuln.advisory_canonical
|
||||
|
||||
```sql
|
||||
-- Migration: 20250101000001_CreateAdvisoryCanonical.sql
|
||||
|
||||
CREATE TABLE vuln.advisory_canonical (
|
||||
-- Identity
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
|
||||
-- Merge key components (used to compute merge_hash)
|
||||
cve TEXT NOT NULL,
|
||||
affects_key TEXT NOT NULL, -- normalized purl or cpe
|
||||
version_range JSONB, -- structured: { introduced, fixed, last_affected }
|
||||
weakness TEXT[] NOT NULL DEFAULT '{}', -- sorted CWE array
|
||||
|
||||
-- Computed identity
|
||||
merge_hash TEXT NOT NULL, -- SHA256 of normalized (cve|affects|range|weakness|lineage)
|
||||
|
||||
-- Metadata
|
||||
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'stub', 'withdrawn')),
|
||||
severity TEXT, -- normalized: critical, high, medium, low, none
|
||||
epss_score NUMERIC(5,4), -- EPSS probability (0.0000-1.0000)
|
||||
exploit_known BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
|
||||
-- Content (for stub degradation)
|
||||
title TEXT,
|
||||
summary TEXT,
|
||||
|
||||
-- Audit
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- Constraints
|
||||
CONSTRAINT uq_advisory_canonical_merge_hash UNIQUE (merge_hash)
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX idx_advisory_canonical_cve ON vuln.advisory_canonical(cve);
|
||||
CREATE INDEX idx_advisory_canonical_affects ON vuln.advisory_canonical(affects_key);
|
||||
CREATE INDEX idx_advisory_canonical_status ON vuln.advisory_canonical(status) WHERE status = 'active';
|
||||
CREATE INDEX idx_advisory_canonical_severity ON vuln.advisory_canonical(severity);
|
||||
CREATE INDEX idx_advisory_canonical_updated ON vuln.advisory_canonical(updated_at DESC);
|
||||
|
||||
-- Trigger for updated_at
|
||||
CREATE TRIGGER trg_advisory_canonical_updated
|
||||
BEFORE UPDATE ON vuln.advisory_canonical
|
||||
FOR EACH ROW EXECUTE FUNCTION vuln.update_timestamp();
|
||||
|
||||
COMMENT ON TABLE vuln.advisory_canonical IS 'Deduplicated canonical advisories with semantic merge_hash';
|
||||
COMMENT ON COLUMN vuln.advisory_canonical.merge_hash IS 'Deterministic hash of (cve, affects_key, version_range, weakness, patch_lineage)';
|
||||
COMMENT ON COLUMN vuln.advisory_canonical.status IS 'active=full record, stub=minimal for low interest, withdrawn=no longer valid';
|
||||
```
|
||||
|
||||
### vuln.advisory_source_edge
|
||||
|
||||
```sql
|
||||
-- Migration: 20250101000002_CreateAdvisorySourceEdge.sql
|
||||
|
||||
CREATE TABLE vuln.advisory_source_edge (
|
||||
-- Identity
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
|
||||
-- Relationships
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
source_id UUID NOT NULL REFERENCES vuln.sources(id) ON DELETE RESTRICT,
|
||||
|
||||
-- Source document
|
||||
source_advisory_id TEXT NOT NULL, -- vendor's advisory ID (DSA-5678, RHSA-2024:1234)
|
||||
source_doc_hash TEXT NOT NULL, -- SHA256 of raw source document
|
||||
|
||||
-- VEX-style status
|
||||
vendor_status TEXT CHECK (vendor_status IN (
|
||||
'affected', 'not_affected', 'fixed', 'under_investigation'
|
||||
)),
|
||||
|
||||
-- Precedence (lower = higher priority)
|
||||
precedence_rank INT NOT NULL DEFAULT 100,
|
||||
|
||||
-- DSSE signature envelope
|
||||
dsse_envelope JSONB, -- { payloadType, payload, signatures[] }
|
||||
|
||||
-- Content snapshot
|
||||
raw_payload JSONB, -- original advisory document
|
||||
|
||||
-- Audit
|
||||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- Constraints
|
||||
CONSTRAINT uq_advisory_source_edge_unique
|
||||
UNIQUE (canonical_id, source_id, source_doc_hash)
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX idx_source_edge_canonical ON vuln.advisory_source_edge(canonical_id);
|
||||
CREATE INDEX idx_source_edge_source ON vuln.advisory_source_edge(source_id);
|
||||
CREATE INDEX idx_source_edge_advisory_id ON vuln.advisory_source_edge(source_advisory_id);
|
||||
CREATE INDEX idx_source_edge_fetched ON vuln.advisory_source_edge(fetched_at DESC);
|
||||
|
||||
-- GIN index for JSONB queries on dsse_envelope
|
||||
CREATE INDEX idx_source_edge_dsse_gin ON vuln.advisory_source_edge
|
||||
USING GIN (dsse_envelope jsonb_path_ops);
|
||||
|
||||
COMMENT ON TABLE vuln.advisory_source_edge IS 'Links canonical advisories to source documents with signatures';
|
||||
COMMENT ON COLUMN vuln.advisory_source_edge.precedence_rank IS 'Source priority: vendor=10, distro=20, osv=30, nvd=40';
|
||||
COMMENT ON COLUMN vuln.advisory_source_edge.dsse_envelope IS 'DSSE envelope with signature over raw_payload';
|
||||
```
|
||||
|
||||
### Supporting Functions
|
||||
|
||||
```sql
|
||||
-- Migration: 20250101000003_CreateCanonicalFunctions.sql
|
||||
|
||||
-- Function to get canonical by merge_hash (most common lookup)
|
||||
CREATE OR REPLACE FUNCTION vuln.get_canonical_by_hash(p_merge_hash TEXT)
|
||||
RETURNS vuln.advisory_canonical
|
||||
LANGUAGE sql STABLE
|
||||
AS $$
|
||||
SELECT * FROM vuln.advisory_canonical
|
||||
WHERE merge_hash = p_merge_hash;
|
||||
$$;
|
||||
|
||||
-- Function to get all source edges for a canonical
|
||||
CREATE OR REPLACE FUNCTION vuln.get_source_edges(p_canonical_id UUID)
|
||||
RETURNS SETOF vuln.advisory_source_edge
|
||||
LANGUAGE sql STABLE
|
||||
AS $$
|
||||
SELECT * FROM vuln.advisory_source_edge
|
||||
WHERE canonical_id = p_canonical_id
|
||||
ORDER BY precedence_rank ASC, fetched_at DESC;
|
||||
$$;
|
||||
|
||||
-- Function to upsert canonical with merge_hash dedup
|
||||
CREATE OR REPLACE FUNCTION vuln.upsert_canonical(
|
||||
p_cve TEXT,
|
||||
p_affects_key TEXT,
|
||||
p_version_range JSONB,
|
||||
p_weakness TEXT[],
|
||||
p_merge_hash TEXT,
|
||||
p_severity TEXT DEFAULT NULL,
|
||||
p_title TEXT DEFAULT NULL,
|
||||
p_summary TEXT DEFAULT NULL
|
||||
)
|
||||
RETURNS UUID
|
||||
LANGUAGE plpgsql
|
||||
AS $$
|
||||
DECLARE
|
||||
v_id UUID;
|
||||
BEGIN
|
||||
INSERT INTO vuln.advisory_canonical (
|
||||
cve, affects_key, version_range, weakness, merge_hash,
|
||||
severity, title, summary
|
||||
)
|
||||
VALUES (
|
||||
p_cve, p_affects_key, p_version_range, p_weakness, p_merge_hash,
|
||||
p_severity, p_title, p_summary
|
||||
)
|
||||
ON CONFLICT (merge_hash) DO UPDATE SET
|
||||
severity = COALESCE(EXCLUDED.severity, vuln.advisory_canonical.severity),
|
||||
title = COALESCE(EXCLUDED.title, vuln.advisory_canonical.title),
|
||||
summary = COALESCE(EXCLUDED.summary, vuln.advisory_canonical.summary),
|
||||
updated_at = NOW()
|
||||
RETURNING id INTO v_id;
|
||||
|
||||
RETURN v_id;
|
||||
END;
|
||||
$$;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entity Models
|
||||
|
||||
### AdvisoryCanonicalEntity
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Models;
|
||||
|
||||
/// <summary>
|
||||
/// Entity representing a deduplicated canonical advisory.
|
||||
/// </summary>
|
||||
public sealed record AdvisoryCanonicalEntity
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public required string Cve { get; init; }
|
||||
public required string AffectsKey { get; init; }
|
||||
public JsonDocument? VersionRange { get; init; }
|
||||
public string[] Weakness { get; init; } = [];
|
||||
public required string MergeHash { get; init; }
|
||||
public string Status { get; init; } = "active";
|
||||
public string? Severity { get; init; }
|
||||
public decimal? EpssScore { get; init; }
|
||||
public bool ExploitKnown { get; init; }
|
||||
public string? Title { get; init; }
|
||||
public string? Summary { get; init; }
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
public DateTimeOffset UpdatedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### AdvisorySourceEdgeEntity
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Entity linking canonical advisory to source document.
|
||||
/// </summary>
|
||||
public sealed record AdvisorySourceEdgeEntity
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public Guid CanonicalId { get; init; }
|
||||
public Guid SourceId { get; init; }
|
||||
public required string SourceAdvisoryId { get; init; }
|
||||
public required string SourceDocHash { get; init; }
|
||||
public string? VendorStatus { get; init; }
|
||||
public int PrecedenceRank { get; init; } = 100;
|
||||
public JsonDocument? DsseEnvelope { get; init; }
|
||||
public JsonDocument? RawPayload { get; init; }
|
||||
public DateTimeOffset FetchedAt { get; init; }
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repository Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Storage.Advisories;
|
||||
|
||||
public interface IAdvisoryCanonicalRepository
|
||||
{
|
||||
// Read operations
|
||||
Task<AdvisoryCanonicalEntity?> GetByIdAsync(Guid id, CancellationToken ct = default);
|
||||
Task<AdvisoryCanonicalEntity?> GetByMergeHashAsync(string mergeHash, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<AdvisoryCanonicalEntity>> GetByCveAsync(string cve, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<AdvisoryCanonicalEntity>> GetByAffectsKeyAsync(string affectsKey, CancellationToken ct = default);
|
||||
|
||||
// Write operations
|
||||
Task<Guid> UpsertAsync(AdvisoryCanonicalEntity entity, CancellationToken ct = default);
|
||||
Task UpdateStatusAsync(Guid id, string status, CancellationToken ct = default);
|
||||
Task DeleteAsync(Guid id, CancellationToken ct = default);
|
||||
|
||||
// Source edge operations
|
||||
Task<IReadOnlyList<AdvisorySourceEdgeEntity>> GetSourceEdgesAsync(Guid canonicalId, CancellationToken ct = default);
|
||||
Task AddSourceEdgeAsync(AdvisorySourceEdgeEntity edge, CancellationToken ct = default);
|
||||
|
||||
// Bulk operations
|
||||
Task<int> CountAsync(CancellationToken ct = default);
|
||||
IAsyncEnumerable<AdvisoryCanonicalEntity> StreamActiveAsync(CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Migration Strategy
|
||||
|
||||
### Phase 1: Shadow Tables (Non-Breaking)
|
||||
|
||||
```sql
|
||||
-- Create new tables alongside existing
|
||||
-- No changes to vuln.advisories
|
||||
|
||||
-- Populate advisory_canonical from existing advisories
|
||||
INSERT INTO vuln.advisory_canonical (
|
||||
cve, affects_key, version_range, weakness, merge_hash,
|
||||
severity, title, summary, created_at
|
||||
)
|
||||
SELECT
|
||||
a.primary_vuln_id,
|
||||
COALESCE(aa.package_purl, 'unknown'),
|
||||
aa.version_ranges,
|
||||
COALESCE(w.cwes, '{}'),
|
||||
-- Compute merge_hash via application code
|
||||
'PLACEHOLDER_' || a.id::TEXT,
|
||||
a.severity,
|
||||
a.title,
|
||||
a.summary,
|
||||
a.created_at
|
||||
FROM vuln.advisories a
|
||||
LEFT JOIN vuln.advisory_affected aa ON aa.advisory_id = a.id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT array_agg(weakness_id) as cwes
|
||||
FROM vuln.advisory_weaknesses
|
||||
WHERE advisory_id = a.id
|
||||
) w ON TRUE
|
||||
WHERE a.state = 'active';
|
||||
```
|
||||
|
||||
### Phase 2: Backfill merge_hash
|
||||
|
||||
```csharp
|
||||
// Application-side migration job
|
||||
public async Task BackfillMergeHashesAsync(CancellationToken ct)
|
||||
{
|
||||
await foreach (var canonical in _repository.StreamAllAsync(ct))
|
||||
{
|
||||
if (canonical.MergeHash.StartsWith("PLACEHOLDER_"))
|
||||
{
|
||||
var input = new MergeHashInput
|
||||
{
|
||||
Cve = canonical.Cve,
|
||||
AffectsKey = canonical.AffectsKey,
|
||||
VersionRange = ParseVersionRange(canonical.VersionRange),
|
||||
Weaknesses = canonical.Weakness
|
||||
};
|
||||
|
||||
var mergeHash = _hashCalculator.ComputeMergeHash(input);
|
||||
await _repository.UpdateMergeHashAsync(canonical.Id, mergeHash, ct);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Create Source Edges
|
||||
|
||||
```sql
|
||||
-- Create source edges from existing provenance
|
||||
INSERT INTO vuln.advisory_source_edge (
|
||||
canonical_id, source_id, source_advisory_id, source_doc_hash,
|
||||
precedence_rank, raw_payload, fetched_at
|
||||
)
|
||||
SELECT
|
||||
c.id,
|
||||
s.source_id,
|
||||
snap.source_advisory_id,
|
||||
snap.payload_hash,
|
||||
CASE s.source_type
|
||||
WHEN 'vendor' THEN 10
|
||||
WHEN 'oval' THEN 20
|
||||
WHEN 'osv' THEN 30
|
||||
WHEN 'nvd' THEN 40
|
||||
ELSE 100
|
||||
END,
|
||||
snap.raw_payload,
|
||||
snap.created_at
|
||||
FROM vuln.advisory_canonical c
|
||||
JOIN vuln.advisories a ON a.primary_vuln_id = c.cve
|
||||
JOIN vuln.advisory_snapshots snap ON snap.advisory_id = a.id
|
||||
JOIN vuln.sources s ON s.id = snap.source_id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Evidence Requirements
|
||||
|
||||
| Test | Evidence |
|
||||
|------|----------|
|
||||
| Migration up/down | Tables created, dropped, recreated cleanly |
|
||||
| Unique constraint | Duplicate merge_hash rejected with appropriate error |
|
||||
| Cascade delete | Deleting canonical removes all source edges |
|
||||
| DSSE storage | JSONB envelope stored and retrieved correctly |
|
||||
| Index performance | merge_hash lookup < 10ms with 1M rows |
|
||||
| Data migration | Record counts match after migration |
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Migration data loss | Critical | Full backup before migration; reversible steps |
|
||||
| Duplicate merge_hash during migration | Constraint violation | Compute hashes before insert; handle conflicts |
|
||||
| Performance regression | User impact | Benchmark queries before/after; add indexes |
|
||||
| DSSE envelope size | Storage bloat | Optional compression; consider external storage |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
387
docs/implplan/SPRINT_8200_0012_0002_evidence_normalizers.md
Normal file
387
docs/implplan/SPRINT_8200_0012_0002_evidence_normalizers.md
Normal file
@@ -0,0 +1,387 @@
|
||||
# Sprint 8200.0012.0002 · Evidence Dimension Normalizers
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **normalizers** that convert raw evidence from existing modules into the normalized 0-1 inputs required by the Evidence-Weighted Score calculator. Each normalizer bridges an existing data source to the unified scoring model.
|
||||
|
||||
This sprint delivers:
|
||||
|
||||
1. **BackportEvidenceNormalizer**: Convert `ProofBlob` confidence → 0-1 BKP score
|
||||
2. **ExploitLikelihoodNormalizer**: Combine EPSS score/percentile + KEV → 0-1 XPL score
|
||||
3. **MitigationNormalizer**: Convert gate multipliers → 0-1 MIT score
|
||||
4. **ReachabilityNormalizer**: Convert `ReachabilityState` + confidence → 0-1 RCH score
|
||||
5. **RuntimeSignalNormalizer**: Convert `RuntimeEvidence` → 0-1 RTS score
|
||||
6. **SourceTrustNormalizer**: Convert `TrustVector` → 0-1 SRC score
|
||||
7. **Aggregate Normalizer Service**: Compose all normalizers into single evidence input
|
||||
|
||||
**Working directory:** `src/Signals/StellaOps.Signals/EvidenceWeightedScore/Normalizers/` (new), tests in `src/Signals/__Tests/StellaOps.Signals.Tests/EvidenceWeightedScore/Normalizers/`
|
||||
|
||||
**Evidence:** All normalizers produce valid [0, 1] outputs; edge cases handled; integration tests pass with real data from existing modules.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0012.0001 (Core input models)
|
||||
- **Blocks:** Sprint 8200.0012.0003 (Policy Integration), Sprint 8200.0012.0004 (API)
|
||||
- **Safe to run in parallel with:** None (depends on core sprint)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/signals/architecture.md` (from Sprint 0001)
|
||||
- `docs/modules/concelier/backport-detection.md` (existing)
|
||||
- `docs/modules/scanner/epss-enrichment.md` (existing)
|
||||
- `docs/modules/excititor/trust-vector.md` (existing)
|
||||
- `docs/modules/policy/reachability-analysis.md` (existing)
|
||||
|
||||
---
|
||||
|
||||
## Normalization Specifications
|
||||
|
||||
### BKP (Backport Evidence) Normalization
|
||||
|
||||
**Source:** `Concelier/BackportProofService.GenerateProofAsync()` → `ProofBlob`
|
||||
|
||||
| Evidence Tier | Confidence Range | BKP Value |
|
||||
|--------------|------------------|-----------|
|
||||
| No evidence | - | 0.00 |
|
||||
| Tier 1: Distro advisory text only | 0.0-0.5 | 0.40-0.55 |
|
||||
| Tier 1: Distro advisory with version | 0.5-0.8 | 0.55-0.70 |
|
||||
| Tier 2: Changelog mention | 0.3-0.6 | 0.45-0.60 |
|
||||
| Tier 3: Patch header match | 0.6-0.9 | 0.70-0.85 |
|
||||
| Tier 3: HunkSig match | 0.7-0.95 | 0.80-0.92 |
|
||||
| Tier 4: Binary fingerprint match | 0.85-1.0 | 0.90-1.00 |
|
||||
| Multiple tiers combined | Aggregated | max(individual) + 0.05 bonus |
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
BKP = evidence.Count == 0 ? 0.0
|
||||
: Math.Min(1.0, MaxTierScore(evidence) + CombinationBonus(evidence));
|
||||
```
|
||||
|
||||
### XPL (Exploit Likelihood) Normalization
|
||||
|
||||
**Source:** `Scanner/EpssPriorityCalculator` + `Concelier/VendorRiskSignalExtractor.KevStatus`
|
||||
|
||||
| Signal | XPL Contribution |
|
||||
|--------|-----------------|
|
||||
| KEV present | +0.40 (floor) |
|
||||
| EPSS percentile >= 0.99 (top 1%) | 0.90-1.00 |
|
||||
| EPSS percentile >= 0.95 (top 5%) | 0.70-0.89 |
|
||||
| EPSS percentile >= 0.75 (top 25%) | 0.40-0.69 |
|
||||
| EPSS percentile < 0.75 | 0.20-0.39 |
|
||||
| No EPSS data | 0.30 (neutral) |
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
XPL = Math.Max(
|
||||
kevPresent ? 0.40 : 0.0,
|
||||
epssPercentile.HasValue
|
||||
? MapPercentileToScore(epssPercentile.Value)
|
||||
: 0.30
|
||||
);
|
||||
```
|
||||
|
||||
### MIT (Mitigation) Normalization
|
||||
|
||||
**Source:** `Policy/GateMultipliersBps` + runtime environment
|
||||
|
||||
| Mitigation | MIT Contribution |
|
||||
|------------|-----------------|
|
||||
| Feature flag disabled | 0.20-0.40 |
|
||||
| Auth required (non-admin) | 0.10-0.20 |
|
||||
| Admin only | 0.15-0.25 |
|
||||
| Non-default config required | 0.15-0.30 |
|
||||
| Seccomp profile active | 0.10-0.25 |
|
||||
| AppArmor/SELinux confined | 0.10-0.20 |
|
||||
| Network isolation | 0.05-0.15 |
|
||||
| Read-only filesystem | 0.05-0.10 |
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
MIT = Math.Min(1.0, Sum(ActiveMitigations.Select(m => m.Effectiveness)));
|
||||
```
|
||||
|
||||
### RCH (Reachability) Normalization
|
||||
|
||||
**Source:** `Policy/ConfidenceCalculator.CalculateReachabilityFactor()`
|
||||
|
||||
| State | Confidence | RCH Value |
|
||||
|-------|------------|-----------|
|
||||
| ConfirmedReachable | 1.0 | 0.95-1.00 |
|
||||
| StaticReachable | 0.7-1.0 | 0.70-0.90 |
|
||||
| StaticReachable | 0.3-0.7 | 0.40-0.70 |
|
||||
| Unknown | - | 0.50 |
|
||||
| StaticUnreachable | 0.7-1.0 | 0.10-0.25 |
|
||||
| ConfirmedUnreachable | 1.0 | 0.00-0.05 |
|
||||
|
||||
**Note:** RCH represents "risk of reachability" — higher = more likely reachable = more risk.
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
RCH = state switch
|
||||
{
|
||||
ConfirmedReachable => 0.95 + (confidence * 0.05),
|
||||
StaticReachable => 0.40 + (confidence * 0.50),
|
||||
Unknown => 0.50,
|
||||
StaticUnreachable => 0.25 - (confidence * 0.20),
|
||||
ConfirmedUnreachable => 0.05 - (confidence * 0.05),
|
||||
_ => 0.50
|
||||
};
|
||||
```
|
||||
|
||||
### RTS (Runtime Signal) Normalization
|
||||
|
||||
**Source:** `Policy/ConfidenceCalculator.CalculateRuntimeFactor()`
|
||||
|
||||
| Posture | Observations | Recency | RTS Value |
|
||||
|---------|-------------|---------|-----------|
|
||||
| Supports | 10+ / 24h | < 1h | 0.90-1.00 |
|
||||
| Supports | 5-10 / 24h | < 6h | 0.75-0.89 |
|
||||
| Supports | 1-5 / 24h | < 24h | 0.60-0.74 |
|
||||
| Supports | Any | > 24h | 0.50-0.60 |
|
||||
| Unknown | - | - | 0.00 |
|
||||
| Contradicts | Any | Any | 0.05-0.15 |
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
RTS = posture switch
|
||||
{
|
||||
Supports => CalculateSupportScore(observationCount, recencyHours),
|
||||
Unknown => 0.0,
|
||||
Contradicts => 0.10
|
||||
};
|
||||
|
||||
double CalculateSupportScore(int count, double recencyHours)
|
||||
{
|
||||
var baseScore = count >= 10 ? 0.90 : count >= 5 ? 0.75 : count >= 1 ? 0.60 : 0.50;
|
||||
var recencyBonus = recencyHours < 1 ? 0.10 : recencyHours < 6 ? 0.05 : 0.0;
|
||||
return Math.Min(1.0, baseScore + recencyBonus);
|
||||
}
|
||||
```
|
||||
|
||||
### SRC (Source Trust) Normalization
|
||||
|
||||
**Source:** `Excititor/TrustVector.ComputeBaseTrust()`
|
||||
|
||||
| Issuer Type | Trust Vector | SRC Value |
|
||||
|-------------|--------------|-----------|
|
||||
| Vendor VEX (signed) | 0.9-1.0 | 0.90-1.00 |
|
||||
| Vendor VEX (unsigned) | 0.7-0.9 | 0.70-0.85 |
|
||||
| Distro advisory (signed) | 0.7-0.85 | 0.70-0.85 |
|
||||
| Distro advisory (unsigned) | 0.5-0.7 | 0.50-0.70 |
|
||||
| Community/OSV | 0.4-0.6 | 0.40-0.60 |
|
||||
| Unknown/unverified | 0.0-0.3 | 0.20-0.30 |
|
||||
|
||||
**Formula:**
|
||||
```csharp
|
||||
SRC = trustVector.ComputeBaseTrust(defaultWeights) * issuerTypeMultiplier;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Interface Definitions)** | | | | | |
|
||||
| 0 | NORM-8200-000 | TODO | Sprint 0001 | Signals Guild | Define `IEvidenceNormalizer<TInput>` interface with `Normalize(TInput) → double`. |
|
||||
| 1 | NORM-8200-001 | TODO | Task 0 | Signals Guild | Define `INormalizerAggregator` interface with `Aggregate(finding) → EvidenceWeightedScoreInput`. |
|
||||
| 2 | NORM-8200-002 | TODO | Task 0 | Signals Guild | Define normalization configuration options (thresholds, tier weights). |
|
||||
| **Wave 1 (Backport Normalizer)** | | | | | |
|
||||
| 3 | NORM-8200-003 | TODO | Task 0 | Signals Guild | Implement `BackportEvidenceNormalizer`: consume `ProofBlob`, output BKP [0, 1]. |
|
||||
| 4 | NORM-8200-004 | TODO | Task 3 | Signals Guild | Implement tier-based scoring: distro < changelog < patch < binary. |
|
||||
| 5 | NORM-8200-005 | TODO | Task 3 | Signals Guild | Implement combination bonus: multiple evidence tiers increase confidence. |
|
||||
| 6 | NORM-8200-006 | TODO | Task 3 | Signals Guild | Handle "not_affected" status: set flag for guardrail consumption. |
|
||||
| 7 | NORM-8200-007 | TODO | Tasks 3-6 | QA Guild | Add unit tests: all tiers, combinations, edge cases, no evidence. |
|
||||
| **Wave 2 (Exploit Likelihood Normalizer)** | | | | | |
|
||||
| 8 | NORM-8200-008 | TODO | Task 0 | Signals Guild | Implement `ExploitLikelihoodNormalizer`: consume EPSS + KEV, output XPL [0, 1]. |
|
||||
| 9 | NORM-8200-009 | TODO | Task 8 | Signals Guild | Implement EPSS percentile → score mapping (linear interpolation within bands). |
|
||||
| 10 | NORM-8200-010 | TODO | Task 8 | Signals Guild | Implement KEV floor: if KEV present, minimum XPL = 0.40. |
|
||||
| 11 | NORM-8200-011 | TODO | Task 8 | Signals Guild | Handle missing EPSS data: neutral score 0.30. |
|
||||
| 12 | NORM-8200-012 | TODO | Tasks 8-11 | QA Guild | Add unit tests: percentile boundaries, KEV override, missing data. |
|
||||
| **Wave 3 (Mitigation Normalizer)** | | | | | |
|
||||
| 13 | NORM-8200-013 | TODO | Task 0 | Signals Guild | Implement `MitigationNormalizer`: consume gate flags + runtime env, output MIT [0, 1]. |
|
||||
| 14 | NORM-8200-014 | TODO | Task 13 | Signals Guild | Convert `GateMultipliersBps` to mitigation effectiveness scores. |
|
||||
| 15 | NORM-8200-015 | TODO | Task 13 | Signals Guild | Add seccomp/AppArmor detection via container metadata. |
|
||||
| 16 | NORM-8200-016 | TODO | Task 13 | Signals Guild | Add network isolation detection via network policy annotations. |
|
||||
| 17 | NORM-8200-017 | TODO | Task 13 | Signals Guild | Implement combination: sum mitigations, cap at 1.0. |
|
||||
| 18 | NORM-8200-018 | TODO | Tasks 13-17 | QA Guild | Add unit tests: individual mitigations, combinations, cap behavior. |
|
||||
| **Wave 4 (Reachability Normalizer)** | | | | | |
|
||||
| 19 | NORM-8200-019 | TODO | Task 0 | Signals Guild | Implement `ReachabilityNormalizer`: consume `ReachabilityEvidence`, output RCH [0, 1]. |
|
||||
| 20 | NORM-8200-020 | TODO | Task 19 | Signals Guild | Map `ReachabilityState` enum to base scores. |
|
||||
| 21 | NORM-8200-021 | TODO | Task 19 | Signals Guild | Apply `AnalysisConfidence` modifier within state range. |
|
||||
| 22 | NORM-8200-022 | TODO | Task 19 | Signals Guild | Handle unknown state: neutral 0.50. |
|
||||
| 23 | NORM-8200-023 | TODO | Tasks 19-22 | QA Guild | Add unit tests: all states, confidence variations, unknown handling. |
|
||||
| **Wave 5 (Runtime Signal Normalizer)** | | | | | |
|
||||
| 24 | NORM-8200-024 | TODO | Task 0 | Signals Guild | Implement `RuntimeSignalNormalizer`: consume `RuntimeEvidence`, output RTS [0, 1]. |
|
||||
| 25 | NORM-8200-025 | TODO | Task 24 | Signals Guild | Map `RuntimePosture` to base scores. |
|
||||
| 26 | NORM-8200-026 | TODO | Task 24 | Signals Guild | Implement observation count scaling (1-5 → 5-10 → 10+). |
|
||||
| 27 | NORM-8200-027 | TODO | Task 24 | Signals Guild | Implement recency bonus: more recent = higher score. |
|
||||
| 28 | NORM-8200-028 | TODO | Task 24 | Signals Guild | Handle "Contradicts" posture: low score but non-zero. |
|
||||
| 29 | NORM-8200-029 | TODO | Tasks 24-28 | QA Guild | Add unit tests: postures, counts, recency, edge cases. |
|
||||
| **Wave 6 (Source Trust Normalizer)** | | | | | |
|
||||
| 30 | NORM-8200-030 | TODO | Task 0 | Signals Guild | Implement `SourceTrustNormalizer`: consume `TrustVector` + issuer metadata, output SRC [0, 1]. |
|
||||
| 31 | NORM-8200-031 | TODO | Task 30 | Signals Guild | Call `TrustVector.ComputeBaseTrust()` with default weights. |
|
||||
| 32 | NORM-8200-032 | TODO | Task 30 | Signals Guild | Apply issuer type multiplier (vendor > distro > community). |
|
||||
| 33 | NORM-8200-033 | TODO | Task 30 | Signals Guild | Apply signature status modifier (signed > unsigned). |
|
||||
| 34 | NORM-8200-034 | TODO | Tasks 30-33 | QA Guild | Add unit tests: issuer types, signatures, trust vector variations. |
|
||||
| **Wave 7 (Aggregator Service)** | | | | | |
|
||||
| 35 | NORM-8200-035 | TODO | All above | Signals Guild | Implement `NormalizerAggregator`: orchestrate all normalizers for a finding. |
|
||||
| 36 | NORM-8200-036 | TODO | Task 35 | Signals Guild | Define finding data retrieval strategy (lazy vs eager loading). |
|
||||
| 37 | NORM-8200-037 | TODO | Task 35 | Signals Guild | Implement parallel normalization for performance. |
|
||||
| 38 | NORM-8200-038 | TODO | Task 35 | Signals Guild | Handle partial evidence: use defaults for missing dimensions. |
|
||||
| 39 | NORM-8200-039 | TODO | Task 35 | Signals Guild | Return fully populated `EvidenceWeightedScoreInput`. |
|
||||
| 40 | NORM-8200-040 | TODO | Tasks 35-39 | QA Guild | Add integration tests: full aggregation with real evidence data. |
|
||||
| **Wave 8 (DI & Integration)** | | | | | |
|
||||
| 41 | NORM-8200-041 | TODO | All above | Signals Guild | Implement `AddEvidenceNormalizers()` extension method. |
|
||||
| 42 | NORM-8200-042 | TODO | Task 41 | Signals Guild | Wire all normalizers + aggregator into DI container. |
|
||||
| 43 | NORM-8200-043 | TODO | Task 41 | Signals Guild | Add configuration binding for normalization options. |
|
||||
| 44 | NORM-8200-044 | TODO | Tasks 41-43 | QA Guild | Add integration tests for full DI pipeline. |
|
||||
| **Wave 9 (Cross-Module Integration Tests)** | | | | | |
|
||||
| 45 | NORM-8200-045 | TODO | All above | QA Guild | Add integration test: `BackportProofService` → `BackportNormalizer` → BKP. |
|
||||
| 46 | NORM-8200-046 | TODO | All above | QA Guild | Add integration test: `EpssPriorityCalculator` + KEV → `ExploitNormalizer` → XPL. |
|
||||
| 47 | NORM-8200-047 | TODO | All above | QA Guild | Add integration test: `ConfidenceCalculator` evidence → normalizers → full input. |
|
||||
| 48 | NORM-8200-048 | TODO | All above | QA Guild | Add end-to-end test: real finding → aggregator → calculator → score. |
|
||||
|
||||
---
|
||||
|
||||
## Interface Definitions
|
||||
|
||||
### IEvidenceNormalizer
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Normalizes raw evidence to [0, 1] score.
|
||||
/// </summary>
|
||||
/// <typeparam name="TInput">Raw evidence type</typeparam>
|
||||
public interface IEvidenceNormalizer<TInput>
|
||||
{
|
||||
/// <summary>
|
||||
/// Normalize evidence to [0, 1] score.
|
||||
/// </summary>
|
||||
double Normalize(TInput input);
|
||||
|
||||
/// <summary>
|
||||
/// Normalize with detailed breakdown.
|
||||
/// </summary>
|
||||
NormalizationResult NormalizeWithDetails(TInput input);
|
||||
}
|
||||
|
||||
public sealed record NormalizationResult(
|
||||
double Score,
|
||||
string Dimension,
|
||||
string Explanation,
|
||||
IReadOnlyDictionary<string, double> Components);
|
||||
```
|
||||
|
||||
### INormalizerAggregator
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Aggregates all normalizers to produce unified input.
|
||||
/// </summary>
|
||||
public interface INormalizerAggregator
|
||||
{
|
||||
/// <summary>
|
||||
/// Aggregate all evidence for a finding into normalized input.
|
||||
/// </summary>
|
||||
Task<EvidenceWeightedScoreInput> AggregateAsync(
|
||||
string findingId,
|
||||
CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Aggregate from pre-loaded evidence.
|
||||
/// </summary>
|
||||
EvidenceWeightedScoreInput Aggregate(FindingEvidence evidence);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pre-loaded evidence for a finding.
|
||||
/// </summary>
|
||||
public sealed record FindingEvidence(
|
||||
string FindingId,
|
||||
ReachabilityEvidence? Reachability,
|
||||
RuntimeEvidence? Runtime,
|
||||
ProofBlob? BackportProof,
|
||||
EpssData? Epss,
|
||||
bool IsInKev,
|
||||
VexStatement? BestVexStatement,
|
||||
IReadOnlyList<ActiveMitigation> Mitigations);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-2 | Interfaces | All interfaces defined, config options ready |
|
||||
| **Wave 1** | 3-7 | Backport normalizer | BKP normalization works with all tiers |
|
||||
| **Wave 2** | 8-12 | Exploit normalizer | XPL combines EPSS + KEV correctly |
|
||||
| **Wave 3** | 13-18 | Mitigation normalizer | MIT reflects active mitigations |
|
||||
| **Wave 4** | 19-23 | Reachability normalizer | RCH maps states correctly |
|
||||
| **Wave 5** | 24-29 | Runtime normalizer | RTS reflects observation strength |
|
||||
| **Wave 6** | 30-34 | Source trust normalizer | SRC combines trust vector + issuer |
|
||||
| **Wave 7** | 35-40 | Aggregator | Full input generation works |
|
||||
| **Wave 8** | 41-44 | DI integration | All normalizers wired via DI |
|
||||
| **Wave 9** | 45-48 | Cross-module tests | Real data flows through pipeline |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint/Module |
|
||||
|-----------|-------------|----------------------|
|
||||
| ProofBlob structure | Backport normalizer consumes existing ProofBlob | Concelier/BackportProofService |
|
||||
| EPSS data access | Exploit normalizer needs EPSS score + percentile | Scanner/EpssPriorityCalculator |
|
||||
| KEV status access | Exploit normalizer needs KEV flag | Concelier/VendorRiskSignalExtractor |
|
||||
| TrustVector API | Source trust normalizer calls ComputeBaseTrust | Excititor/TrustVector |
|
||||
| ReachabilityEvidence | Reachability normalizer consumes Policy types | Policy/ConfidenceCalculator |
|
||||
| RuntimeEvidence | Runtime normalizer consumes Policy types | Policy/ConfidenceCalculator |
|
||||
| Core input models | All normalizers produce inputs for Sprint 0001 | 8200.0012.0001 |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming Checkpoints
|
||||
|
||||
| Date (UTC) | Milestone | Evidence |
|
||||
|------------|-----------|----------|
|
||||
| 2026-02-10 | Wave 0-2 complete | Interfaces defined, BKP + XPL normalizers work |
|
||||
| 2026-02-24 | Wave 3-5 complete | MIT, RCH, RTS normalizers work |
|
||||
| 2026-03-10 | Wave 6-7 complete | SRC normalizer + aggregator work |
|
||||
| 2026-03-24 | Wave 8-9 complete | Full DI integration, cross-module tests pass |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Normalizers are stateless | Thread-safe, testable, cacheable |
|
||||
| Configuration via options pattern | Hot-reload thresholds without restart |
|
||||
| Parallel normalization in aggregator | Performance for high-volume scoring |
|
||||
| Defaults for missing evidence | Graceful degradation with neutral scores |
|
||||
| Breakdown included in result | Enables UI explanation without recalculation |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Tier mapping disputes | Inaccurate BKP scores | Review with security team; iterate | Signals Guild |
|
||||
| EPSS percentile drift | Score instability | Use percentile bands, not raw values | Signals Guild |
|
||||
| Mitigation detection gaps | Under-counting mitigations | Extensible mitigation registry | Platform Guild |
|
||||
| Cross-module dependency breaks | Integration failures | Comprehensive integration tests | QA Guild |
|
||||
| Performance bottleneck in aggregator | Latency | Parallel fetch, caching, benchmarks | Platform Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created as second phase of evidence-weighted score implementation. | Project Mgmt |
|
||||
@@ -0,0 +1,446 @@
|
||||
# Sprint 8200.0012.0003 - Canonical Advisory Service
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement the **service layer** for canonical advisory management. This sprint delivers:
|
||||
|
||||
1. **CanonicalAdvisoryService**: Business logic for creating/retrieving canonical advisories
|
||||
2. **Deduplication Pipeline**: Ingest raw advisories, compute merge_hash, upsert canonical + edges
|
||||
3. **Query APIs**: Retrieve deduplicated advisories by CVE, PURL, or artifact
|
||||
4. **DSSE Integration**: Sign source edges during ingestion
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Core/`
|
||||
|
||||
**Evidence:** Ingesting same CVE from two sources produces single canonical with two source edges; query returns deduplicated results.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0001 (merge_hash), SPRINT_8200_0012_0002 (schema)
|
||||
- **Blocks:** Phase B sprints (learning cache)
|
||||
- **Safe to run in parallel with:** Nothing (foundational service)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- `src/Concelier/__Libraries/StellaOps.Concelier.Core/AGENTS.md`
|
||||
- `src/Attestor/StellaOps.Attestor.Envelope/DsseEnvelopeSerializer.cs`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Service Design** | | | | | |
|
||||
| 0 | CANSVC-8200-000 | TODO | Schema ready | Concelier Guild | Define `ICanonicalAdvisoryService` interface with all operations |
|
||||
| 1 | CANSVC-8200-001 | TODO | Task 0 | Concelier Guild | Define `CanonicalAdvisory` domain model (distinct from entity) |
|
||||
| 2 | CANSVC-8200-002 | TODO | Task 0 | Concelier Guild | Define `SourceEdge` domain model with DSSE envelope |
|
||||
| 3 | CANSVC-8200-003 | TODO | Task 0 | Concelier Guild | Define `IngestResult` result type with merge decision |
|
||||
| **Wave 1: Core Service Implementation** | | | | | |
|
||||
| 4 | CANSVC-8200-004 | TODO | Tasks 0-3 | Concelier Guild | Implement `CanonicalAdvisoryService` constructor with DI |
|
||||
| 5 | CANSVC-8200-005 | TODO | Task 4 | Concelier Guild | Implement `IngestAsync()` - raw advisory to canonical pipeline |
|
||||
| 6 | CANSVC-8200-006 | TODO | Task 5 | Concelier Guild | Implement merge_hash computation during ingest |
|
||||
| 7 | CANSVC-8200-007 | TODO | Task 6 | Concelier Guild | Implement canonical upsert with source edge creation |
|
||||
| 8 | CANSVC-8200-008 | TODO | Task 7 | Concelier Guild | Implement DSSE signing of source edge via Signer client |
|
||||
| 9 | CANSVC-8200-009 | TODO | Task 8 | QA Guild | Unit tests for ingest pipeline (new canonical, existing canonical) |
|
||||
| **Wave 2: Query Operations** | | | | | |
|
||||
| 10 | CANSVC-8200-010 | TODO | Task 4 | Concelier Guild | Implement `GetByIdAsync()` - fetch canonical with source edges |
|
||||
| 11 | CANSVC-8200-011 | TODO | Task 4 | Concelier Guild | Implement `GetByCveAsync()` - all canonicals for a CVE |
|
||||
| 12 | CANSVC-8200-012 | TODO | Task 4 | Concelier Guild | Implement `GetByArtifactAsync()` - canonicals affecting purl/cpe |
|
||||
| 13 | CANSVC-8200-013 | TODO | Task 4 | Concelier Guild | Implement `GetByMergeHashAsync()` - direct lookup |
|
||||
| 14 | CANSVC-8200-014 | TODO | Tasks 10-13 | Concelier Guild | Add caching layer for hot queries (in-memory, short TTL) |
|
||||
| 15 | CANSVC-8200-015 | TODO | Task 14 | QA Guild | Unit tests for all query operations |
|
||||
| **Wave 3: API Endpoints** | | | | | |
|
||||
| 16 | CANSVC-8200-016 | TODO | Task 15 | Concelier Guild | Create `GET /api/v1/canonical/{id}` endpoint |
|
||||
| 17 | CANSVC-8200-017 | TODO | Task 15 | Concelier Guild | Create `GET /api/v1/canonical?cve={cve}` endpoint |
|
||||
| 18 | CANSVC-8200-018 | TODO | Task 15 | Concelier Guild | Create `GET /api/v1/canonical?artifact={purl}` endpoint |
|
||||
| 19 | CANSVC-8200-019 | TODO | Task 15 | Concelier Guild | Create `POST /api/v1/ingest/{source}` endpoint |
|
||||
| 20 | CANSVC-8200-020 | TODO | Tasks 16-19 | QA Guild | Integration tests for all endpoints |
|
||||
| **Wave 4: Connector Integration** | | | | | |
|
||||
| 21 | CANSVC-8200-021 | TODO | Task 19 | Concelier Guild | Modify OSV connector to use canonical ingest pipeline |
|
||||
| 22 | CANSVC-8200-022 | TODO | Task 21 | Concelier Guild | Modify NVD connector to use canonical ingest pipeline |
|
||||
| 23 | CANSVC-8200-023 | TODO | Task 22 | Concelier Guild | Modify GHSA connector to use canonical ingest pipeline |
|
||||
| 24 | CANSVC-8200-024 | TODO | Task 23 | Concelier Guild | Modify distro connectors (Debian, RHEL, SUSE) to use canonical pipeline |
|
||||
| 25 | CANSVC-8200-025 | TODO | Task 24 | QA Guild | End-to-end test: ingest from multiple connectors, verify deduplication |
|
||||
| 26 | CANSVC-8200-026 | TODO | Task 25 | Docs Guild | Document canonical service in module README |
|
||||
|
||||
---
|
||||
|
||||
## Service Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Core.Canonical;
|
||||
|
||||
/// <summary>
|
||||
/// Service for managing canonical advisories with provenance-scoped deduplication.
|
||||
/// </summary>
|
||||
public interface ICanonicalAdvisoryService
|
||||
{
|
||||
// === Ingest Operations ===
|
||||
|
||||
/// <summary>
|
||||
/// Ingest raw advisory from source, creating or updating canonical record.
|
||||
/// </summary>
|
||||
/// <param name="source">Source identifier (osv, nvd, ghsa, redhat, debian, etc.)</param>
|
||||
/// <param name="rawAdvisory">Raw advisory document</param>
|
||||
/// <param name="ct">Cancellation token</param>
|
||||
/// <returns>Ingest result with canonical ID and merge decision</returns>
|
||||
Task<IngestResult> IngestAsync(
|
||||
string source,
|
||||
RawAdvisory rawAdvisory,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Batch ingest multiple advisories from same source.
|
||||
/// </summary>
|
||||
Task<IReadOnlyList<IngestResult>> IngestBatchAsync(
|
||||
string source,
|
||||
IEnumerable<RawAdvisory> advisories,
|
||||
CancellationToken ct = default);
|
||||
|
||||
// === Query Operations ===
|
||||
|
||||
/// <summary>
|
||||
/// Get canonical advisory by ID with all source edges.
|
||||
/// </summary>
|
||||
Task<CanonicalAdvisory?> GetByIdAsync(Guid id, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Get canonical advisory by merge hash.
|
||||
/// </summary>
|
||||
Task<CanonicalAdvisory?> GetByMergeHashAsync(string mergeHash, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Get all canonical advisories for a CVE.
|
||||
/// </summary>
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetByCveAsync(string cve, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Get canonical advisories affecting an artifact (PURL or CPE).
|
||||
/// </summary>
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetByArtifactAsync(
|
||||
string artifactKey,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Query canonical advisories with filters.
|
||||
/// </summary>
|
||||
Task<PagedResult<CanonicalAdvisory>> QueryAsync(
|
||||
CanonicalQueryOptions options,
|
||||
CancellationToken ct = default);
|
||||
|
||||
// === Status Operations ===
|
||||
|
||||
/// <summary>
|
||||
/// Update canonical status (active, stub, withdrawn).
|
||||
/// </summary>
|
||||
Task UpdateStatusAsync(Guid id, CanonicalStatus status, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Degrade low-interest canonicals to stub status.
|
||||
/// </summary>
|
||||
Task<int> DegradeToStubsAsync(double scoreThreshold, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Models
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Canonical advisory with all source edges.
|
||||
/// </summary>
|
||||
public sealed record CanonicalAdvisory
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public required string Cve { get; init; }
|
||||
public required string AffectsKey { get; init; }
|
||||
public VersionRange? VersionRange { get; init; }
|
||||
public IReadOnlyList<string> Weaknesses { get; init; } = [];
|
||||
public required string MergeHash { get; init; }
|
||||
public CanonicalStatus Status { get; init; } = CanonicalStatus.Active;
|
||||
public string? Severity { get; init; }
|
||||
public decimal? EpssScore { get; init; }
|
||||
public bool ExploitKnown { get; init; }
|
||||
public string? Title { get; init; }
|
||||
public string? Summary { get; init; }
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
public DateTimeOffset UpdatedAt { get; init; }
|
||||
|
||||
/// <summary>All source edges for this canonical, ordered by precedence.</summary>
|
||||
public IReadOnlyList<SourceEdge> SourceEdges { get; init; } = [];
|
||||
|
||||
/// <summary>Primary source edge (highest precedence).</summary>
|
||||
public SourceEdge? PrimarySource => SourceEdges.FirstOrDefault();
|
||||
}
|
||||
|
||||
public enum CanonicalStatus
|
||||
{
|
||||
Active,
|
||||
Stub,
|
||||
Withdrawn
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Link from canonical advisory to source document.
|
||||
/// </summary>
|
||||
public sealed record SourceEdge
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public required string SourceName { get; init; }
|
||||
public required string SourceAdvisoryId { get; init; }
|
||||
public required string SourceDocHash { get; init; }
|
||||
public VendorStatus? VendorStatus { get; init; }
|
||||
public int PrecedenceRank { get; init; }
|
||||
public DsseEnvelope? DsseEnvelope { get; init; }
|
||||
public DateTimeOffset FetchedAt { get; init; }
|
||||
}
|
||||
|
||||
public enum VendorStatus
|
||||
{
|
||||
Affected,
|
||||
NotAffected,
|
||||
Fixed,
|
||||
UnderInvestigation
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Result of ingesting a raw advisory.
|
||||
/// </summary>
|
||||
public sealed record IngestResult
|
||||
{
|
||||
public required Guid CanonicalId { get; init; }
|
||||
public required string MergeHash { get; init; }
|
||||
public required MergeDecision Decision { get; init; }
|
||||
public Guid? SignatureRef { get; init; }
|
||||
public string? ConflictReason { get; init; }
|
||||
}
|
||||
|
||||
public enum MergeDecision
|
||||
{
|
||||
Created, // New canonical created
|
||||
Merged, // Merged into existing canonical
|
||||
Duplicate, // Exact duplicate, no changes
|
||||
Conflict // Merge conflict detected
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ingest Pipeline
|
||||
|
||||
```csharp
|
||||
public async Task<IngestResult> IngestAsync(
|
||||
string source,
|
||||
RawAdvisory rawAdvisory,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
// 1. Normalize and extract merge hash components
|
||||
var cve = ExtractCve(rawAdvisory);
|
||||
var affectsKey = ExtractAffectsKey(rawAdvisory);
|
||||
var versionRange = ExtractVersionRange(rawAdvisory);
|
||||
var weaknesses = ExtractWeaknesses(rawAdvisory);
|
||||
var patchLineage = await ResolvePatchLineageAsync(rawAdvisory, ct);
|
||||
|
||||
// 2. Compute merge hash
|
||||
var mergeHashInput = new MergeHashInput
|
||||
{
|
||||
Cve = cve,
|
||||
AffectsKey = affectsKey,
|
||||
VersionRange = versionRange,
|
||||
Weaknesses = weaknesses,
|
||||
PatchLineage = patchLineage
|
||||
};
|
||||
var mergeHash = _mergeHashCalculator.ComputeMergeHash(mergeHashInput);
|
||||
|
||||
// 3. Check for existing canonical
|
||||
var existing = await _repository.GetByMergeHashAsync(mergeHash, ct);
|
||||
|
||||
MergeDecision decision;
|
||||
Guid canonicalId;
|
||||
|
||||
if (existing is null)
|
||||
{
|
||||
// 4a. Create new canonical
|
||||
var canonical = new AdvisoryCanonicalEntity
|
||||
{
|
||||
Cve = cve,
|
||||
AffectsKey = affectsKey,
|
||||
VersionRange = SerializeVersionRange(versionRange),
|
||||
Weakness = weaknesses.ToArray(),
|
||||
MergeHash = mergeHash,
|
||||
Severity = rawAdvisory.Severity,
|
||||
Title = rawAdvisory.Title,
|
||||
Summary = rawAdvisory.Summary
|
||||
};
|
||||
canonicalId = await _repository.UpsertAsync(canonical, ct);
|
||||
decision = MergeDecision.Created;
|
||||
}
|
||||
else
|
||||
{
|
||||
// 4b. Merge into existing
|
||||
canonicalId = existing.Id;
|
||||
decision = MergeDecision.Merged;
|
||||
|
||||
// Update metadata if newer/better
|
||||
await UpdateCanonicalMetadataAsync(existing, rawAdvisory, ct);
|
||||
}
|
||||
|
||||
// 5. Create source edge
|
||||
var sourceDocHash = ComputeDocumentHash(rawAdvisory);
|
||||
var sourceEdge = new AdvisorySourceEdgeEntity
|
||||
{
|
||||
CanonicalId = canonicalId,
|
||||
SourceId = await ResolveSourceIdAsync(source, ct),
|
||||
SourceAdvisoryId = rawAdvisory.AdvisoryId,
|
||||
SourceDocHash = sourceDocHash,
|
||||
VendorStatus = MapVendorStatus(rawAdvisory),
|
||||
PrecedenceRank = GetPrecedenceRank(source),
|
||||
RawPayload = JsonDocument.Parse(rawAdvisory.RawJson)
|
||||
};
|
||||
|
||||
// 6. Sign source edge
|
||||
Guid? signatureRef = null;
|
||||
if (_signingEnabled)
|
||||
{
|
||||
var envelope = await _signerClient.SignAsync(sourceDocHash, ct);
|
||||
sourceEdge = sourceEdge with { DsseEnvelope = envelope };
|
||||
signatureRef = envelope.SignatureId;
|
||||
}
|
||||
|
||||
// 7. Store source edge
|
||||
await _repository.AddSourceEdgeAsync(sourceEdge, ct);
|
||||
|
||||
// 8. Emit event
|
||||
await _eventBus.PublishAsync(new CanonicalAdvisoryIngested
|
||||
{
|
||||
CanonicalId = canonicalId,
|
||||
MergeHash = mergeHash,
|
||||
Source = source,
|
||||
Decision = decision
|
||||
}, ct);
|
||||
|
||||
return new IngestResult
|
||||
{
|
||||
CanonicalId = canonicalId,
|
||||
MergeHash = mergeHash,
|
||||
Decision = decision,
|
||||
SignatureRef = signatureRef
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
```csharp
|
||||
// GET /api/v1/canonical/{id}
|
||||
app.MapGet("/api/v1/canonical/{id:guid}", async (
|
||||
Guid id,
|
||||
ICanonicalAdvisoryService service,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var canonical = await service.GetByIdAsync(id, ct);
|
||||
return canonical is null
|
||||
? Results.NotFound()
|
||||
: Results.Ok(canonical);
|
||||
})
|
||||
.WithName("GetCanonicalById")
|
||||
.WithSummary("Get canonical advisory by ID")
|
||||
.Produces<CanonicalAdvisory>(200)
|
||||
.Produces(404);
|
||||
|
||||
// GET /api/v1/canonical?cve={cve}
|
||||
app.MapGet("/api/v1/canonical", async (
|
||||
[FromQuery] string? cve,
|
||||
[FromQuery] string? artifact,
|
||||
ICanonicalAdvisoryService service,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
if (!string.IsNullOrEmpty(cve))
|
||||
{
|
||||
return Results.Ok(await service.GetByCveAsync(cve, ct));
|
||||
}
|
||||
if (!string.IsNullOrEmpty(artifact))
|
||||
{
|
||||
return Results.Ok(await service.GetByArtifactAsync(artifact, ct));
|
||||
}
|
||||
return Results.BadRequest("Either 'cve' or 'artifact' query parameter required");
|
||||
})
|
||||
.WithName("QueryCanonical")
|
||||
.WithSummary("Query canonical advisories by CVE or artifact");
|
||||
|
||||
// POST /api/v1/ingest/{source}
|
||||
app.MapPost("/api/v1/ingest/{source}", async (
|
||||
string source,
|
||||
RawAdvisory advisory,
|
||||
ICanonicalAdvisoryService service,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var result = await service.IngestAsync(source, advisory, ct);
|
||||
return Results.Ok(result);
|
||||
})
|
||||
.WithName("IngestAdvisory")
|
||||
.WithSummary("Ingest raw advisory from source")
|
||||
.Produces<IngestResult>(200);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Precedence Configuration
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Source precedence ranks (lower = higher priority).
|
||||
/// </summary>
|
||||
public static class SourcePrecedence
|
||||
{
|
||||
public const int VendorPsirt = 10; // Vendor PSIRT (Cisco, Oracle, etc.)
|
||||
public const int VendorSbom = 15; // Vendor SBOM attestation
|
||||
public const int Distro = 20; // Linux distribution (Debian, RHEL, SUSE)
|
||||
public const int Osv = 30; // OSV database
|
||||
public const int Ghsa = 35; // GitHub Security Advisory
|
||||
public const int Nvd = 40; // NVD
|
||||
public const int Cert = 50; // CERT advisories
|
||||
public const int Community = 100; // Community sources
|
||||
|
||||
public static int GetRank(string source) => source.ToLowerInvariant() switch
|
||||
{
|
||||
"cisco" or "oracle" or "microsoft" or "adobe" => VendorPsirt,
|
||||
"redhat" or "debian" or "suse" or "ubuntu" or "alpine" => Distro,
|
||||
"osv" => Osv,
|
||||
"ghsa" => Ghsa,
|
||||
"nvd" => Nvd,
|
||||
"cert-cc" or "cert-bund" or "cert-fr" => Cert,
|
||||
_ => Community
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
| Scenario | Expected Behavior |
|
||||
|----------|-------------------|
|
||||
| Ingest new CVE from NVD | Creates canonical + source edge |
|
||||
| Ingest same CVE from RHEL | Adds source edge to existing canonical |
|
||||
| Ingest same CVE from GHSA | Adds source edge; GHSA higher precedence than NVD |
|
||||
| Ingest duplicate (same hash) | Returns Duplicate decision, no changes |
|
||||
| Query by CVE | Returns single canonical with multiple edges |
|
||||
| Query by PURL | Returns only canonicals affecting that package |
|
||||
| Degrade to stub | Low-interest canonicals become stubs |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
348
docs/implplan/SPRINT_8200_0012_0003_policy_engine_integration.md
Normal file
348
docs/implplan/SPRINT_8200_0012_0003_policy_engine_integration.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# Sprint 8200.0012.0003 · Policy Engine Integration
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Integrate the Evidence-Weighted Score into the **Policy Engine** pipeline so that findings receive unified scores during policy evaluation. This enables score-based policy rules, verdict enrichment, and attestation of scoring decisions.
|
||||
|
||||
This sprint delivers:
|
||||
|
||||
1. **Score Enrichment Pipeline**: Invoke EWS calculator during policy evaluation
|
||||
2. **Score-Based Policy Rules**: Enable rules like `when score < 40 then allow`
|
||||
3. **Verdict Enrichment**: Include EWS result in verdict artifacts
|
||||
4. **Score Attestation**: Sign scoring decisions with determinism proofs
|
||||
5. **Confidence→EWS Migration Path**: Gradual transition from existing confidence scoring
|
||||
6. **Policy DSL Extensions**: New DSL constructs for score-based conditions
|
||||
|
||||
**Working directory:** `src/Policy/StellaOps.Policy.Engine/Scoring/` (extend), `src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Scoring/` (tests)
|
||||
|
||||
**Evidence:** Policy engine emits EWS in verdicts; score-based rules evaluate correctly; attestations include scoring proofs.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0012.0001 (Core library), Sprint 8200.0012.0002 (Normalizers)
|
||||
- **Blocks:** Sprint 8200.0012.0004 (API — needs verdict enrichment)
|
||||
- **Safe to run in parallel with:** Sprint 8200.0012.0005 (Frontend — independent)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/signals/architecture.md` (from Sprint 0001)
|
||||
- `docs/modules/policy/architecture.md` (existing)
|
||||
- `docs/modules/policy/confidence-scoring.md` (existing — to be deprecated)
|
||||
- `docs/modules/policy/verdict-attestation.md` (existing)
|
||||
|
||||
---
|
||||
|
||||
## Integration Architecture
|
||||
|
||||
### Current Flow (Confidence-Based)
|
||||
|
||||
```
|
||||
Finding → ConfidenceCalculator → ConfidenceScore → Verdict → Attestation
|
||||
```
|
||||
|
||||
### Target Flow (EWS-Integrated)
|
||||
|
||||
```
|
||||
Finding → NormalizerAggregator → EvidenceWeightedScoreInput
|
||||
↓
|
||||
→ EvidenceWeightedScoreCalculator → EvidenceWeightedScoreResult
|
||||
↓
|
||||
→ PolicyEvaluator (with score-based rules)
|
||||
↓
|
||||
→ Verdict (enriched with EWS)
|
||||
↓
|
||||
→ VerdictAttestation (with EWS proof)
|
||||
```
|
||||
|
||||
### Coexistence Strategy
|
||||
|
||||
During migration, both scoring systems will run:
|
||||
|
||||
```csharp
|
||||
public sealed record EnrichedVerdict
|
||||
{
|
||||
// Legacy (deprecated, but maintained for compatibility)
|
||||
public ConfidenceScore? Confidence { get; init; }
|
||||
|
||||
// New unified score
|
||||
public EvidenceWeightedScoreResult? EvidenceWeightedScore { get; init; }
|
||||
|
||||
// Feature flag for gradual rollout
|
||||
public bool UseEvidenceWeightedScore { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Integration Setup)** | | | | | |
|
||||
| 0 | PINT-8200-000 | TODO | Sprint 0002 | Policy Guild | Add package reference from `StellaOps.Policy.Engine` to `StellaOps.Signals`. |
|
||||
| 1 | PINT-8200-001 | TODO | Task 0 | Policy Guild | Create `PolicyEvidenceWeightedScoreOptions` for integration configuration. |
|
||||
| 2 | PINT-8200-002 | TODO | Task 1 | Policy Guild | Add feature flag: `EnableEvidenceWeightedScore` (default: false for rollout). |
|
||||
| **Wave 1 (Score Enrichment Pipeline)** | | | | | |
|
||||
| 3 | PINT-8200-003 | TODO | Task 0 | Policy Guild | Create `IFindingScoreEnricher` interface for scoring during evaluation. |
|
||||
| 4 | PINT-8200-004 | TODO | Task 3 | Policy Guild | Implement `EvidenceWeightedScoreEnricher`: call aggregator + calculator. |
|
||||
| 5 | PINT-8200-005 | TODO | Task 4 | Policy Guild | Integrate enricher into `PolicyEvaluator` pipeline (after evidence collection). |
|
||||
| 6 | PINT-8200-006 | TODO | Task 5 | Policy Guild | Add score result to `EvaluationContext` for rule consumption. |
|
||||
| 7 | PINT-8200-007 | TODO | Task 5 | Policy Guild | Add caching: avoid recalculating score for same finding within evaluation. |
|
||||
| 8 | PINT-8200-008 | TODO | Tasks 3-7 | QA Guild | Add unit tests: enricher invocation, context population, caching. |
|
||||
| **Wave 2 (Score-Based Policy Rules)** | | | | | |
|
||||
| 9 | PINT-8200-009 | TODO | Task 6 | Policy Guild | Extend `PolicyRuleCondition` to support `score` field access. |
|
||||
| 10 | PINT-8200-010 | TODO | Task 9 | Policy Guild | Implement score comparison operators: `<`, `<=`, `>`, `>=`, `==`, `between`. |
|
||||
| 11 | PINT-8200-011 | TODO | Task 9 | Policy Guild | Implement score bucket matching: `when bucket == "ActNow" then ...`. |
|
||||
| 12 | PINT-8200-012 | TODO | Task 9 | Policy Guild | Implement score flag matching: `when flags contains "live-signal" then ...`. |
|
||||
| 13 | PINT-8200-013 | TODO | Task 9 | Policy Guild | Implement score dimension access: `when score.rch > 0.8 then ...`. |
|
||||
| 14 | PINT-8200-014 | TODO | Tasks 9-13 | QA Guild | Add unit tests: all score-based rule types, edge cases. |
|
||||
| 15 | PINT-8200-015 | TODO | Tasks 9-13 | QA Guild | Add property tests: rule monotonicity (higher score → stricter verdict if configured). |
|
||||
| **Wave 3 (Policy DSL Extensions)** | | | | | |
|
||||
| 16 | PINT-8200-016 | TODO | Task 9 | Policy Guild | Extend DSL grammar: `score`, `score.bucket`, `score.flags`, `score.<dimension>`. |
|
||||
| 17 | PINT-8200-017 | TODO | Task 16 | Policy Guild | Implement DSL parser for new score constructs. |
|
||||
| 18 | PINT-8200-018 | TODO | Task 16 | Policy Guild | Implement DSL validator for score field references. |
|
||||
| 19 | PINT-8200-019 | TODO | Task 16 | Policy Guild | Add DSL autocomplete hints for score fields. |
|
||||
| 20 | PINT-8200-020 | TODO | Tasks 16-19 | QA Guild | Add roundtrip tests for DSL score constructs. |
|
||||
| 21 | PINT-8200-021 | TODO | Tasks 16-19 | QA Guild | Add golden tests for invalid score DSL patterns. |
|
||||
| **Wave 4 (Verdict Enrichment)** | | | | | |
|
||||
| 22 | PINT-8200-022 | TODO | Task 5 | Policy Guild | Extend `Verdict` record with `EvidenceWeightedScoreResult?` field. |
|
||||
| 23 | PINT-8200-023 | TODO | Task 22 | Policy Guild | Populate EWS in verdict during policy evaluation completion. |
|
||||
| 24 | PINT-8200-024 | TODO | Task 22 | Policy Guild | Add `VerdictSummary` extension: include score bucket and top factors. |
|
||||
| 25 | PINT-8200-025 | TODO | Task 22 | Policy Guild | Ensure verdict serialization includes full EWS decomposition. |
|
||||
| 26 | PINT-8200-026 | TODO | Tasks 22-25 | QA Guild | Add snapshot tests for enriched verdict JSON structure. |
|
||||
| **Wave 5 (Score Attestation)** | | | | | |
|
||||
| 27 | PINT-8200-027 | TODO | Task 22 | Policy Guild | Extend `VerdictPredicate` to include EWS in attestation subject. |
|
||||
| 28 | PINT-8200-028 | TODO | Task 27 | Policy Guild | Add `ScoringProof` to attestation: inputs, policy digest, calculation timestamp. |
|
||||
| 29 | PINT-8200-029 | TODO | Task 27 | Policy Guild | Implement scoring determinism verification in attestation verification. |
|
||||
| 30 | PINT-8200-030 | TODO | Task 27 | Policy Guild | Add score provenance chain: finding → evidence → score → verdict. |
|
||||
| 31 | PINT-8200-031 | TODO | Tasks 27-30 | QA Guild | Add attestation verification tests with scoring proofs. |
|
||||
| **Wave 6 (Migration Support)** | | | | | |
|
||||
| 32 | PINT-8200-032 | TODO | Task 22 | Policy Guild | Implement `ConfidenceToEwsAdapter`: translate legacy scores for comparison. |
|
||||
| 33 | PINT-8200-033 | TODO | Task 32 | Policy Guild | Add dual-emit mode: both Confidence and EWS in verdicts (for A/B). |
|
||||
| 34 | PINT-8200-034 | TODO | Task 32 | Policy Guild | Add migration telemetry: compare Confidence vs EWS rankings. |
|
||||
| 35 | PINT-8200-035 | TODO | Task 32 | Policy Guild | Document migration path: feature flag → dual-emit → EWS-only. |
|
||||
| 36 | PINT-8200-036 | TODO | Tasks 32-35 | QA Guild | Add comparison tests: verify EWS produces reasonable rankings vs Confidence. |
|
||||
| **Wave 7 (DI & Configuration)** | | | | | |
|
||||
| 37 | PINT-8200-037 | TODO | All above | Policy Guild | Extend `AddPolicyEngine()` to include EWS services when enabled. |
|
||||
| 38 | PINT-8200-038 | TODO | Task 37 | Policy Guild | Add conditional wiring based on feature flag. |
|
||||
| 39 | PINT-8200-039 | TODO | Task 37 | Policy Guild | Add telemetry: score calculation duration, cache hit rate. |
|
||||
| 40 | PINT-8200-040 | TODO | Tasks 37-39 | QA Guild | Add integration tests for full policy→EWS pipeline. |
|
||||
| **Wave 8 (Determinism & Quality Gates)** | | | | | |
|
||||
| 41 | PINT-8200-041 | TODO | All above | QA Guild | Add determinism test: same finding + policy → same EWS in verdict. |
|
||||
| 42 | PINT-8200-042 | TODO | All above | QA Guild | Add concurrent evaluation test: thread-safe EWS in policy pipeline. |
|
||||
| 43 | PINT-8200-043 | TODO | All above | QA Guild | Add attestation reproducibility test: verify EWS proofs validate. |
|
||||
| 44 | PINT-8200-044 | TODO | All above | Platform Guild | Add benchmark: policy evaluation with EWS < 50ms per finding. |
|
||||
|
||||
---
|
||||
|
||||
## Policy DSL Examples
|
||||
|
||||
### Score Threshold Rules
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- name: block-high-evidence-risk
|
||||
when: score >= 90
|
||||
then: block
|
||||
message: "High evidence of exploitability (score: {score})"
|
||||
|
||||
- name: allow-low-evidence
|
||||
when: score < 40
|
||||
then: allow
|
||||
message: "Insufficient evidence of risk (score: {score})"
|
||||
|
||||
- name: require-review-medium
|
||||
when: score between 40 and 89
|
||||
then: review
|
||||
message: "Requires manual review (score: {score})"
|
||||
```
|
||||
|
||||
### Bucket-Based Rules
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- name: block-act-now
|
||||
when: score.bucket == "ActNow"
|
||||
then: block
|
||||
|
||||
- name: warn-schedule-next
|
||||
when: score.bucket == "ScheduleNext"
|
||||
then: warn
|
||||
```
|
||||
|
||||
### Flag-Based Rules
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- name: block-live-signal
|
||||
when: score.flags contains "live-signal"
|
||||
then: block
|
||||
message: "Runtime evidence detected"
|
||||
|
||||
- name: allow-vendor-na
|
||||
when: score.flags contains "vendor-na"
|
||||
then: allow
|
||||
message: "Vendor confirms not affected"
|
||||
```
|
||||
|
||||
### Dimension Access Rules
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- name: require-reachability-proof
|
||||
when:
|
||||
- score >= 70
|
||||
- score.rch < 0.3 # Low reachability evidence
|
||||
then: review
|
||||
message: "High score but low reachability proof"
|
||||
|
||||
- name: trust-vendor-vex
|
||||
when:
|
||||
- score.src >= 0.9 # High source trust
|
||||
- score.bkp >= 0.8 # Strong backport evidence
|
||||
then: allow
|
||||
message: "Trusted vendor VEX with backport proof"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Additions
|
||||
|
||||
### EnrichedVerdict
|
||||
|
||||
```csharp
|
||||
public sealed record EnrichedVerdict
|
||||
{
|
||||
public required string VerdictId { get; init; }
|
||||
public required string FindingId { get; init; }
|
||||
public required VerdictStatus Status { get; init; }
|
||||
public required DateTimeOffset EvaluatedAt { get; init; }
|
||||
|
||||
// Legacy (maintained for compatibility)
|
||||
[Obsolete("Use EvidenceWeightedScore. Will be removed in v3.0.")]
|
||||
public ConfidenceScore? Confidence { get; init; }
|
||||
|
||||
// New unified score
|
||||
public EvidenceWeightedScoreResult? EvidenceWeightedScore { get; init; }
|
||||
|
||||
// Policy evaluation details
|
||||
public required IReadOnlyList<RuleEvaluation> RuleEvaluations { get; init; }
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
// Attestation
|
||||
public string? AttestationDigest { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### ScoringProof
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Proof of scoring calculation for attestation.
|
||||
/// </summary>
|
||||
public sealed record ScoringProof
|
||||
{
|
||||
/// <summary>Normalized inputs used.</summary>
|
||||
public required EvidenceInputs Inputs { get; init; }
|
||||
|
||||
/// <summary>Policy digest used for calculation.</summary>
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
/// <summary>Calculator version.</summary>
|
||||
public required string CalculatorVersion { get; init; }
|
||||
|
||||
/// <summary>Calculation timestamp (UTC).</summary>
|
||||
public required DateTimeOffset CalculatedAt { get; init; }
|
||||
|
||||
/// <summary>Applied guardrails.</summary>
|
||||
public required AppliedGuardrails Guardrails { get; init; }
|
||||
|
||||
/// <summary>Final score.</summary>
|
||||
public required int Score { get; init; }
|
||||
|
||||
/// <summary>Proof verification: recalculate and compare.</summary>
|
||||
public bool Verify(IEvidenceWeightedScoreCalculator calculator)
|
||||
{
|
||||
var recalculated = calculator.Calculate(
|
||||
Inputs.ToInput(),
|
||||
PolicyDigest);
|
||||
return recalculated.Score == Score;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-2 | Setup | Package refs, feature flag defined |
|
||||
| **Wave 1** | 3-8 | Enrichment pipeline | EWS calculated during evaluation |
|
||||
| **Wave 2** | 9-15 | Score-based rules | All rule types work |
|
||||
| **Wave 3** | 16-21 | DSL extensions | DSL parses score constructs |
|
||||
| **Wave 4** | 22-26 | Verdict enrichment | EWS in verdict JSON |
|
||||
| **Wave 5** | 27-31 | Attestation | Scoring proofs in attestations |
|
||||
| **Wave 6** | 32-36 | Migration | Dual-emit, comparison telemetry |
|
||||
| **Wave 7** | 37-40 | DI integration | Full pipeline via DI |
|
||||
| **Wave 8** | 41-44 | Quality gates | Determinism, performance |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint/Module |
|
||||
|-----------|-------------|----------------------|
|
||||
| EWS calculator | Uses calculator from Sprint 0001 | 8200.0012.0001 |
|
||||
| Normalizer aggregator | Uses aggregator from Sprint 0002 | 8200.0012.0002 |
|
||||
| Existing confidence | Must coexist during migration | Policy/ConfidenceCalculator |
|
||||
| Verdict structure | Changes must be backward compatible | Policy/Verdict |
|
||||
| Attestation format | Scoring proofs must validate | Attestor/VerdictPredicate |
|
||||
| DSL grammar | Score extensions must be additive | Policy/DSL |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming Checkpoints
|
||||
|
||||
| Date (UTC) | Milestone | Evidence |
|
||||
|------------|-----------|----------|
|
||||
| 2026-03-24 | Wave 0-2 complete | EWS in evaluation context, basic rules work |
|
||||
| 2026-04-07 | Wave 3-4 complete | DSL extensions, verdict enrichment |
|
||||
| 2026-04-21 | Wave 5-6 complete | Attestation, migration support |
|
||||
| 2026-05-05 | Wave 7-8 complete | Full integration, quality gates pass |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Feature flag for rollout | Safe gradual adoption |
|
||||
| Dual-emit during migration | A/B comparison, no breaking changes |
|
||||
| Score in DSL via property access | Consistent with existing DSL patterns |
|
||||
| Scoring proof in attestation | Audit trail, reproducibility |
|
||||
| Deprecate Confidence gradually | Give consumers time to migrate |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Rule migration complexity | Existing rules break | Compatibility layer, docs | Policy Guild |
|
||||
| Performance regression | Slower evaluation | Caching, benchmarks | Platform Guild |
|
||||
| Attestation size increase | Storage cost | Compact proof format | Policy Guild |
|
||||
| Migration confusion | User errors | Clear docs, warnings | Product Guild |
|
||||
| DSL backward compatibility | Parse failures | Additive-only grammar changes | Policy Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created for Policy engine integration. | Project Mgmt |
|
||||
458
docs/implplan/SPRINT_8200_0012_0004_api_endpoints.md
Normal file
458
docs/implplan/SPRINT_8200_0012_0004_api_endpoints.md
Normal file
@@ -0,0 +1,458 @@
|
||||
# Sprint 8200.0012.0004 · API Endpoints & Contracts
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Expose the Evidence-Weighted Score through **REST API endpoints** with proper OpenAPI documentation, authentication, rate limiting, and observability. This enables UI consumption, external integrations, and programmatic access to scoring.
|
||||
|
||||
This sprint delivers:
|
||||
|
||||
1. **Score Calculation Endpoint**: `POST /api/v1/findings/{id}/score` — calculate score for a finding
|
||||
2. **Bulk Score Endpoint**: `POST /api/v1/findings/scores` — calculate scores for multiple findings
|
||||
3. **Score History Endpoint**: `GET /api/v1/findings/{id}/score-history` — retrieve historical scores
|
||||
4. **Policy Config Endpoint**: `GET /api/v1/scoring/policy` — retrieve active weight policy
|
||||
5. **OpenAPI Documentation**: Full schema with examples for all score types
|
||||
6. **Webhook Integration**: Score change notifications
|
||||
|
||||
**Working directory:** `src/Findings/StellaOps.Findings.Ledger.WebService/Endpoints/` (extend), `src/Findings/__Tests/StellaOps.Findings.Ledger.WebService.Tests/` (tests)
|
||||
|
||||
**Evidence:** All endpoints return correct EWS JSON; OpenAPI spec validates; auth enforced; rate limits work.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0012.0001 (Core library), Sprint 8200.0012.0002 (Normalizers), Sprint 8200.0012.0003 (Policy Integration — for verdict enrichment)
|
||||
- **Blocks:** Sprint 8200.0012.0005 (Frontend — needs API)
|
||||
- **Safe to run in parallel with:** None (depends on core sprints)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/signals/architecture.md` (from Sprint 0001)
|
||||
- `docs/api/findings-api.md` (existing)
|
||||
- `docs/api/openapi-conventions.md` (existing)
|
||||
- `docs/modules/gateway/rate-limiting.md` (existing)
|
||||
|
||||
---
|
||||
|
||||
## API Specification
|
||||
|
||||
### Endpoint Summary
|
||||
|
||||
| Method | Path | Description | Auth | Rate Limit |
|
||||
|--------|------|-------------|------|------------|
|
||||
| `POST` | `/api/v1/findings/{findingId}/score` | Calculate score for single finding | Required | 100/min |
|
||||
| `POST` | `/api/v1/findings/scores` | Calculate scores for batch (max 100) | Required | 10/min |
|
||||
| `GET` | `/api/v1/findings/{findingId}/score` | Get cached/latest score | Required | 1000/min |
|
||||
| `GET` | `/api/v1/findings/{findingId}/score-history` | Get score history | Required | 100/min |
|
||||
| `GET` | `/api/v1/scoring/policy` | Get active weight policy | Required | 100/min |
|
||||
| `GET` | `/api/v1/scoring/policy/{version}` | Get specific policy version | Required | 100/min |
|
||||
| `POST` | `/api/v1/scoring/webhooks` | Register score change webhook | Admin | 10/min |
|
||||
|
||||
### Request/Response Schemas
|
||||
|
||||
#### Calculate Score (Single)
|
||||
|
||||
**Request:**
|
||||
```http
|
||||
POST /api/v1/findings/{findingId}/score
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer {token}
|
||||
|
||||
{
|
||||
"forceRecalculate": false,
|
||||
"includeBreakdown": true,
|
||||
"policyVersion": null // null = use latest
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"findingId": "CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4",
|
||||
"score": 78,
|
||||
"bucket": "ScheduleNext",
|
||||
"inputs": {
|
||||
"rch": 0.85,
|
||||
"rts": 0.40,
|
||||
"bkp": 0.00,
|
||||
"xpl": 0.70,
|
||||
"src": 0.80,
|
||||
"mit": 0.10
|
||||
},
|
||||
"weights": {
|
||||
"rch": 0.30,
|
||||
"rts": 0.25,
|
||||
"bkp": 0.15,
|
||||
"xpl": 0.15,
|
||||
"src": 0.10,
|
||||
"mit": 0.10
|
||||
},
|
||||
"flags": ["live-signal", "proven-path"],
|
||||
"explanations": [
|
||||
"Static reachability: path to vulnerable sink (confidence: 85%)",
|
||||
"Runtime: 3 observations in last 24 hours",
|
||||
"EPSS: 0.8% probability (High band)",
|
||||
"Source: Distro VEX signed (trust: 80%)",
|
||||
"Mitigations: seccomp profile active"
|
||||
],
|
||||
"caps": {
|
||||
"speculativeCap": false,
|
||||
"notAffectedCap": false,
|
||||
"runtimeFloor": false
|
||||
},
|
||||
"policyDigest": "sha256:abc123...",
|
||||
"calculatedAt": "2026-01-15T14:30:00Z",
|
||||
"cachedUntil": "2026-01-15T15:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### Calculate Scores (Batch)
|
||||
|
||||
**Request:**
|
||||
```http
|
||||
POST /api/v1/findings/scores
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer {token}
|
||||
|
||||
{
|
||||
"findingIds": [
|
||||
"CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4",
|
||||
"CVE-2024-5678@pkg:npm/lodash@4.17.20",
|
||||
"GHSA-abc123@pkg:pypi/requests@2.25.0"
|
||||
],
|
||||
"forceRecalculate": false,
|
||||
"includeBreakdown": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{ "findingId": "...", "score": 78, "bucket": "ScheduleNext", ... },
|
||||
{ "findingId": "...", "score": 45, "bucket": "Investigate", ... },
|
||||
{ "findingId": "...", "score": 92, "bucket": "ActNow", ... }
|
||||
],
|
||||
"summary": {
|
||||
"total": 3,
|
||||
"byBucket": {
|
||||
"ActNow": 1,
|
||||
"ScheduleNext": 1,
|
||||
"Investigate": 1,
|
||||
"Watchlist": 0
|
||||
},
|
||||
"averageScore": 71.7,
|
||||
"calculationTimeMs": 45
|
||||
},
|
||||
"policyDigest": "sha256:abc123...",
|
||||
"calculatedAt": "2026-01-15T14:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Score History
|
||||
|
||||
**Request:**
|
||||
```http
|
||||
GET /api/v1/findings/{findingId}/score-history?from=2026-01-01&to=2026-01-15&limit=50
|
||||
Authorization: Bearer {token}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"findingId": "CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4",
|
||||
"history": [
|
||||
{
|
||||
"score": 78,
|
||||
"bucket": "ScheduleNext",
|
||||
"policyDigest": "sha256:abc123...",
|
||||
"calculatedAt": "2026-01-15T14:30:00Z",
|
||||
"trigger": "evidence_update",
|
||||
"changedFactors": ["rts", "xpl"]
|
||||
},
|
||||
{
|
||||
"score": 65,
|
||||
"bucket": "Investigate",
|
||||
"policyDigest": "sha256:abc123...",
|
||||
"calculatedAt": "2026-01-10T09:15:00Z",
|
||||
"trigger": "scheduled",
|
||||
"changedFactors": []
|
||||
}
|
||||
],
|
||||
"pagination": {
|
||||
"hasMore": true,
|
||||
"nextCursor": "eyJvZmZzZXQiOjUwfQ=="
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Scoring Policy
|
||||
|
||||
**Request:**
|
||||
```http
|
||||
GET /api/v1/scoring/policy
|
||||
Authorization: Bearer {token}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"version": "ews.v1.2",
|
||||
"digest": "sha256:abc123...",
|
||||
"activeSince": "2026-01-01T00:00:00Z",
|
||||
"environment": "production",
|
||||
"weights": {
|
||||
"rch": 0.30,
|
||||
"rts": 0.25,
|
||||
"bkp": 0.15,
|
||||
"xpl": 0.15,
|
||||
"src": 0.10,
|
||||
"mit": 0.10
|
||||
},
|
||||
"guardrails": {
|
||||
"notAffectedCap": { "enabled": true, "maxScore": 15 },
|
||||
"runtimeFloor": { "enabled": true, "minScore": 60 },
|
||||
"speculativeCap": { "enabled": true, "maxScore": 45 }
|
||||
},
|
||||
"buckets": {
|
||||
"actNowMin": 90,
|
||||
"scheduleNextMin": 70,
|
||||
"investigateMin": 40
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (API Design)** | | | | | |
|
||||
| 0 | API-8200-000 | TODO | Sprint 0001 | API Guild | Finalize OpenAPI spec for all EWS endpoints. |
|
||||
| 1 | API-8200-001 | TODO | Task 0 | API Guild | Define request/response DTOs in `StellaOps.Findings.Contracts`. |
|
||||
| 2 | API-8200-002 | TODO | Task 0 | API Guild | Define error response format for scoring failures. |
|
||||
| **Wave 1 (Single Score Endpoint)** | | | | | |
|
||||
| 3 | API-8200-003 | TODO | Task 1 | API Guild | Implement `POST /api/v1/findings/{findingId}/score` endpoint. |
|
||||
| 4 | API-8200-004 | TODO | Task 3 | API Guild | Wire endpoint to `NormalizerAggregator` + `EvidenceWeightedScoreCalculator`. |
|
||||
| 5 | API-8200-005 | TODO | Task 3 | API Guild | Implement `forceRecalculate` parameter (bypass cache). |
|
||||
| 6 | API-8200-006 | TODO | Task 3 | API Guild | Implement `includeBreakdown` parameter (control response verbosity). |
|
||||
| 7 | API-8200-007 | TODO | Task 3 | API Guild | Add response caching with configurable TTL. |
|
||||
| 8 | API-8200-008 | TODO | Tasks 3-7 | QA Guild | Add endpoint tests: success, validation, errors, caching. |
|
||||
| **Wave 2 (Get Cached Score)** | | | | | |
|
||||
| 9 | API-8200-009 | TODO | Task 7 | API Guild | Implement `GET /api/v1/findings/{findingId}/score` endpoint. |
|
||||
| 10 | API-8200-010 | TODO | Task 9 | API Guild | Return cached score if available, 404 if not calculated. |
|
||||
| 11 | API-8200-011 | TODO | Task 9 | API Guild | Add `cachedUntil` field for cache freshness indication. |
|
||||
| 12 | API-8200-012 | TODO | Tasks 9-11 | QA Guild | Add endpoint tests: cache hit, cache miss, stale cache. |
|
||||
| **Wave 3 (Batch Score Endpoint)** | | | | | |
|
||||
| 13 | API-8200-013 | TODO | Task 3 | API Guild | Implement `POST /api/v1/findings/scores` batch endpoint. |
|
||||
| 14 | API-8200-014 | TODO | Task 13 | API Guild | Implement batch size limit (max 100 findings). |
|
||||
| 15 | API-8200-015 | TODO | Task 13 | API Guild | Implement parallel calculation with configurable concurrency. |
|
||||
| 16 | API-8200-016 | TODO | Task 13 | API Guild | Add summary statistics (byBucket, averageScore, calculationTimeMs). |
|
||||
| 17 | API-8200-017 | TODO | Task 13 | API Guild | Handle partial failures: return results + errors for failed items. |
|
||||
| 18 | API-8200-018 | TODO | Tasks 13-17 | QA Guild | Add endpoint tests: batch success, partial failure, size limits. |
|
||||
| **Wave 4 (Score History)** | | | | | |
|
||||
| 19 | API-8200-019 | TODO | Task 3 | API Guild | Implement score history storage (append-only log). |
|
||||
| 20 | API-8200-020 | TODO | Task 19 | API Guild | Implement `GET /api/v1/findings/{findingId}/score-history` endpoint. |
|
||||
| 21 | API-8200-021 | TODO | Task 20 | API Guild | Add date range filtering (`from`, `to` parameters). |
|
||||
| 22 | API-8200-022 | TODO | Task 20 | API Guild | Add pagination with cursor-based navigation. |
|
||||
| 23 | API-8200-023 | TODO | Task 20 | API Guild | Track score change triggers (evidence_update, policy_change, scheduled). |
|
||||
| 24 | API-8200-024 | TODO | Task 20 | API Guild | Track changed factors between score versions. |
|
||||
| 25 | API-8200-025 | TODO | Tasks 19-24 | QA Guild | Add endpoint tests: history retrieval, pagination, filtering. |
|
||||
| **Wave 5 (Policy Endpoints)** | | | | | |
|
||||
| 26 | API-8200-026 | TODO | Sprint 0001 | API Guild | Implement `GET /api/v1/scoring/policy` endpoint. |
|
||||
| 27 | API-8200-027 | TODO | Task 26 | API Guild | Return active policy with full configuration. |
|
||||
| 28 | API-8200-028 | TODO | Task 26 | API Guild | Implement `GET /api/v1/scoring/policy/{version}` for specific versions. |
|
||||
| 29 | API-8200-029 | TODO | Task 26 | API Guild | Add policy version history listing. |
|
||||
| 30 | API-8200-030 | TODO | Tasks 26-29 | QA Guild | Add endpoint tests: policy retrieval, version history. |
|
||||
| **Wave 6 (Webhooks)** | | | | | |
|
||||
| 31 | API-8200-031 | TODO | Task 19 | API Guild | Define webhook payload schema for score changes. |
|
||||
| 32 | API-8200-032 | TODO | Task 31 | API Guild | Implement `POST /api/v1/scoring/webhooks` registration endpoint. |
|
||||
| 33 | API-8200-033 | TODO | Task 32 | API Guild | Implement webhook delivery with retry logic. |
|
||||
| 34 | API-8200-034 | TODO | Task 32 | API Guild | Add webhook signature verification (HMAC-SHA256). |
|
||||
| 35 | API-8200-035 | TODO | Task 32 | API Guild | Add webhook management: list, update, delete. |
|
||||
| 36 | API-8200-036 | TODO | Tasks 31-35 | QA Guild | Add webhook tests: registration, delivery, retries, signatures. |
|
||||
| **Wave 7 (Auth & Rate Limiting)** | | | | | |
|
||||
| 37 | API-8200-037 | TODO | All endpoints | API Guild | Add authentication requirement to all endpoints. |
|
||||
| 38 | API-8200-038 | TODO | Task 37 | API Guild | Add scope-based authorization (read:scores, write:scores, admin:scoring). |
|
||||
| 39 | API-8200-039 | TODO | Task 37 | API Guild | Implement rate limiting per endpoint (see spec). |
|
||||
| 40 | API-8200-040 | TODO | Task 37 | API Guild | Add rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining). |
|
||||
| 41 | API-8200-041 | TODO | Tasks 37-40 | QA Guild | Add auth/rate limit tests: unauthorized, forbidden, rate exceeded. |
|
||||
| **Wave 8 (OpenAPI & Documentation)** | | | | | |
|
||||
| 42 | API-8200-042 | TODO | All endpoints | API Guild | Generate OpenAPI 3.1 spec with all endpoints. |
|
||||
| 43 | API-8200-043 | TODO | Task 42 | API Guild | Add request/response examples for all operations. |
|
||||
| 44 | API-8200-044 | TODO | Task 42 | API Guild | Add schema descriptions and validation constraints. |
|
||||
| 45 | API-8200-045 | TODO | Task 42 | Docs Guild | Update `docs/api/findings-api.md` with EWS section. |
|
||||
| 46 | API-8200-046 | TODO | Tasks 42-45 | QA Guild | Validate OpenAPI spec with spectral linter. |
|
||||
| **Wave 9 (Observability)** | | | | | |
|
||||
| 47 | API-8200-047 | TODO | All endpoints | API Guild | Add OpenTelemetry traces for all endpoints. |
|
||||
| 48 | API-8200-048 | TODO | Task 47 | API Guild | Add span attributes: finding_id, score, bucket, calculation_time_ms. |
|
||||
| 49 | API-8200-049 | TODO | Task 47 | API Guild | Add metrics: ews_calculations_total, ews_calculation_duration_seconds. |
|
||||
| 50 | API-8200-050 | TODO | Task 47 | API Guild | Add logging: score changes, policy updates, webhook deliveries. |
|
||||
| 51 | API-8200-051 | TODO | Tasks 47-50 | QA Guild | Verify OTel traces in integration tests. |
|
||||
|
||||
---
|
||||
|
||||
## OpenAPI Excerpt
|
||||
|
||||
```yaml
|
||||
openapi: 3.1.0
|
||||
info:
|
||||
title: StellaOps Findings API - Evidence-Weighted Score
|
||||
version: 1.0.0
|
||||
|
||||
paths:
|
||||
/api/v1/findings/{findingId}/score:
|
||||
post:
|
||||
operationId: calculateFindingScore
|
||||
summary: Calculate evidence-weighted score for a finding
|
||||
tags: [Scoring]
|
||||
security:
|
||||
- BearerAuth: [write:scores]
|
||||
parameters:
|
||||
- name: findingId
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
pattern: "^[A-Z]+-\\d+@pkg:.+$"
|
||||
example: "CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4"
|
||||
requestBody:
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/CalculateScoreRequest'
|
||||
responses:
|
||||
'200':
|
||||
description: Score calculated successfully
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/EvidenceWeightedScoreResult'
|
||||
'404':
|
||||
description: Finding not found
|
||||
'429':
|
||||
description: Rate limit exceeded
|
||||
|
||||
components:
|
||||
schemas:
|
||||
EvidenceWeightedScoreResult:
|
||||
type: object
|
||||
required:
|
||||
- findingId
|
||||
- score
|
||||
- bucket
|
||||
- inputs
|
||||
- weights
|
||||
- flags
|
||||
- explanations
|
||||
- caps
|
||||
- policyDigest
|
||||
- calculatedAt
|
||||
properties:
|
||||
findingId:
|
||||
type: string
|
||||
score:
|
||||
type: integer
|
||||
minimum: 0
|
||||
maximum: 100
|
||||
bucket:
|
||||
type: string
|
||||
enum: [ActNow, ScheduleNext, Investigate, Watchlist]
|
||||
inputs:
|
||||
$ref: '#/components/schemas/EvidenceInputs'
|
||||
weights:
|
||||
$ref: '#/components/schemas/EvidenceWeights'
|
||||
flags:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
enum: [live-signal, proven-path, vendor-na, speculative]
|
||||
explanations:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
caps:
|
||||
$ref: '#/components/schemas/AppliedGuardrails'
|
||||
policyDigest:
|
||||
type: string
|
||||
pattern: "^sha256:[a-f0-9]{64}$"
|
||||
calculatedAt:
|
||||
type: string
|
||||
format: date-time
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-2 | API design | OpenAPI spec, DTOs defined |
|
||||
| **Wave 1** | 3-8 | Single score | POST endpoint works |
|
||||
| **Wave 2** | 9-12 | Get cached | GET endpoint works |
|
||||
| **Wave 3** | 13-18 | Batch | Batch endpoint works |
|
||||
| **Wave 4** | 19-25 | History | History endpoint works |
|
||||
| **Wave 5** | 26-30 | Policy | Policy endpoints work |
|
||||
| **Wave 6** | 31-36 | Webhooks | Webhook system works |
|
||||
| **Wave 7** | 37-41 | Auth/Rate | Security enforced |
|
||||
| **Wave 8** | 42-46 | OpenAPI | Spec validated |
|
||||
| **Wave 9** | 47-51 | Observability | Traces, metrics work |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint/Module |
|
||||
|-----------|-------------|----------------------|
|
||||
| Core calculator | Endpoints call calculator from Sprint 0001 | 8200.0012.0001 |
|
||||
| Aggregator | Endpoints call aggregator from Sprint 0002 | 8200.0012.0002 |
|
||||
| Verdict enrichment | History may come from verdicts | 8200.0012.0003 |
|
||||
| Frontend consumption | UI calls these endpoints | 8200.0012.0005 |
|
||||
| Gateway routing | Endpoints registered via Router | Gateway/Router |
|
||||
| Auth integration | Uses Authority tokens | Authority |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming Checkpoints
|
||||
|
||||
| Date (UTC) | Milestone | Evidence |
|
||||
|------------|-----------|----------|
|
||||
| 2026-04-07 | Wave 0-2 complete | Single + cached score endpoints work |
|
||||
| 2026-04-21 | Wave 3-4 complete | Batch + history endpoints work |
|
||||
| 2026-05-05 | Wave 5-6 complete | Policy + webhooks work |
|
||||
| 2026-05-19 | Wave 7-9 complete | Auth, rate limits, observability, OpenAPI |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Separate calculate (POST) and get (GET) | Calculate is expensive; GET is cheap cache lookup |
|
||||
| Max 100 findings per batch | Balance between utility and resource consumption |
|
||||
| Cursor-based pagination for history | Better for append-only logs than offset |
|
||||
| Webhook with HMAC signature | Standard pattern for webhook security |
|
||||
| Score history retention 90 days | Balance storage vs auditability |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| High batch calculation load | Resource exhaustion | Rate limits, queue processing | Platform Guild |
|
||||
| Cache invalidation complexity | Stale scores | Event-driven invalidation | API Guild |
|
||||
| Webhook delivery failures | Missed notifications | Retry with exponential backoff | API Guild |
|
||||
| OpenAPI spec drift | Integration breaks | Spec-first, contract tests | API Guild |
|
||||
| Rate limit tuning | User frustration or abuse | Monitor, adjust thresholds | Platform Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created for API endpoints. | Project Mgmt |
|
||||
371
docs/implplan/SPRINT_8200_0012_0005_frontend_ui.md
Normal file
371
docs/implplan/SPRINT_8200_0012_0005_frontend_ui.md
Normal file
@@ -0,0 +1,371 @@
|
||||
# Sprint 8200.0012.0005 · Frontend UI Components
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Build **Angular UI components** for displaying and interacting with Evidence-Weighted Scores. This enables users to visually triage findings, understand score breakdowns, and take action based on evidence strength.
|
||||
|
||||
This sprint delivers:
|
||||
|
||||
1. **Score Pill Component**: Compact 0-100 score display with color coding
|
||||
2. **Score Breakdown Popover**: Hover/click breakdown of all six dimensions
|
||||
3. **Score Badge Components**: Live, Proven Path, Vendor-N/A badges
|
||||
4. **Findings List Sorting**: Sort by score, filter by bucket
|
||||
5. **Score History Chart**: Timeline visualization of score changes
|
||||
6. **Bulk Triage View**: Multi-select findings by score bucket
|
||||
|
||||
**Working directory:** `src/Web/StellaOps.Web/src/app/features/findings/` (extend), `src/Web/StellaOps.Web/src/app/shared/components/score/` (new)
|
||||
|
||||
**Evidence:** All components render correctly; accessibility passes; responsive design works; storybook documentation complete.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 8200.0012.0004 (API Endpoints — needs data source)
|
||||
- **Blocks:** None (final sprint in chain)
|
||||
- **Safe to run in parallel with:** Sprints 0001-0003 (backend independent of UI)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/signals/architecture.md` (from Sprint 0001)
|
||||
- `docs/ui/design-system.md` (existing)
|
||||
- `docs/ui/component-guidelines.md` (existing)
|
||||
- `src/Web/StellaOps.Web/.storybook/` (existing storybook setup)
|
||||
|
||||
---
|
||||
|
||||
## Design Specifications
|
||||
|
||||
### Score Pill Component
|
||||
|
||||
```
|
||||
┌───────┐
|
||||
│ 78 │ ← Score value (bold, white text)
|
||||
└───────┘ ← Background color based on bucket
|
||||
```
|
||||
|
||||
**Color Mapping:**
|
||||
| Bucket | Score Range | Background | Text |
|
||||
|--------|-------------|------------|------|
|
||||
| ActNow | 90-100 | `#DC2626` (red-600) | white |
|
||||
| ScheduleNext | 70-89 | `#F59E0B` (amber-500) | black |
|
||||
| Investigate | 40-69 | `#3B82F6` (blue-500) | white |
|
||||
| Watchlist | 0-39 | `#6B7280` (gray-500) | white |
|
||||
|
||||
**Size Variants:**
|
||||
- `sm`: 24x20px, 12px font
|
||||
- `md`: 32x24px, 14px font (default)
|
||||
- `lg`: 40x28px, 16px font
|
||||
|
||||
### Score Breakdown Popover
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Evidence Score: 78/100 │
|
||||
│ Bucket: Schedule Next Sprint │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Reachability ████████▒▒ 0.85 │
|
||||
│ Runtime ████▒▒▒▒▒▒ 0.40 │
|
||||
│ Backport ▒▒▒▒▒▒▒▒▒▒ 0.00 │
|
||||
│ Exploit ███████▒▒▒ 0.70 │
|
||||
│ Source Trust ████████▒▒ 0.80 │
|
||||
│ Mitigations -█▒▒▒▒▒▒▒▒▒ 0.10 │
|
||||
├─────────────────────────────────────────┤
|
||||
│ 🟢 Live signal detected │
|
||||
│ ✓ Proven reachability path │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Top factors: │
|
||||
│ • Static path to vulnerable sink │
|
||||
│ • EPSS: 0.8% (High band) │
|
||||
│ • Distro VEX signed │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Score Badges
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌─────────────┐ ┌────────────┐
|
||||
│ 🟢 Live │ │ ✓ Proven │ │ ⊘ Vendor │
|
||||
│ Signal │ │ Path │ │ N/A │
|
||||
└──────────────┘ └─────────────┘ └────────────┘
|
||||
(green bg) (blue bg) (gray bg)
|
||||
```
|
||||
|
||||
### Findings List with Scores
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Findings Sort: [Score ▼] │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Filter: [All Buckets ▼] [Has Live Signal ☑] [Has Backport ☐] │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ ☑ │ 92 │ CVE-2024-1234 │ curl 7.64.0-4 │ 🟢 Live │ Critical │
|
||||
│ ☐ │ 78 │ CVE-2024-5678 │ lodash 4.17 │ ✓ Path │ High │
|
||||
│ ☐ │ 45 │ GHSA-abc123 │ requests 2.25 │ │ Medium │
|
||||
│ ☐ │ 23 │ CVE-2023-9999 │ openssl 1.1.1 │ ⊘ N/A │ Low │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Score History Chart
|
||||
|
||||
```
|
||||
Score
|
||||
100 ┤
|
||||
80 ┤ ●━━━━━━●━━━━━●
|
||||
60 ┤ ●━━━●
|
||||
40 ┤●━━━●
|
||||
20 ┤
|
||||
0 ┼────────────────────────→ Time
|
||||
Jan 1 Jan 5 Jan 10 Jan 15
|
||||
|
||||
Legend: ● Evidence update ○ Policy change
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Project Setup)** | | | | | |
|
||||
| 0 | FE-8200-000 | TODO | Sprint 0004 | FE Guild | Create `src/app/shared/components/score/` module. |
|
||||
| 1 | FE-8200-001 | TODO | Task 0 | FE Guild | Add EWS API service in `src/app/core/services/scoring.service.ts`. |
|
||||
| 2 | FE-8200-002 | TODO | Task 1 | FE Guild | Define TypeScript interfaces for EWS response types. |
|
||||
| 3 | FE-8200-003 | TODO | Task 0 | FE Guild | Set up Storybook stories directory for score components. |
|
||||
| **Wave 1 (Score Pill Component)** | | | | | |
|
||||
| 4 | FE-8200-004 | TODO | Task 0 | FE Guild | Create `ScorePillComponent` with score input. |
|
||||
| 5 | FE-8200-005 | TODO | Task 4 | FE Guild | Implement bucket-based color mapping. |
|
||||
| 6 | FE-8200-006 | TODO | Task 4 | FE Guild | Add size variants (sm, md, lg). |
|
||||
| 7 | FE-8200-007 | TODO | Task 4 | FE Guild | Add ARIA attributes for accessibility. |
|
||||
| 8 | FE-8200-008 | TODO | Task 4 | FE Guild | Add click handler for breakdown popover trigger. |
|
||||
| 9 | FE-8200-009 | TODO | Tasks 4-8 | QA Guild | Add unit tests for all variants and states. |
|
||||
| 10 | FE-8200-010 | TODO | Tasks 4-8 | FE Guild | Add Storybook stories with all variants. |
|
||||
| **Wave 2 (Score Breakdown Popover)** | | | | | |
|
||||
| 11 | FE-8200-011 | TODO | Task 4 | FE Guild | Create `ScoreBreakdownPopoverComponent`. |
|
||||
| 12 | FE-8200-012 | TODO | Task 11 | FE Guild | Implement dimension bar chart (6 horizontal bars). |
|
||||
| 13 | FE-8200-013 | TODO | Task 11 | FE Guild | Add mitigation bar with negative styling. |
|
||||
| 14 | FE-8200-014 | TODO | Task 11 | FE Guild | Implement flags section with icons. |
|
||||
| 15 | FE-8200-015 | TODO | Task 11 | FE Guild | Implement explanations list. |
|
||||
| 16 | FE-8200-016 | TODO | Task 11 | FE Guild | Add guardrails indication (caps/floors applied). |
|
||||
| 17 | FE-8200-017 | TODO | Task 11 | FE Guild | Implement hover positioning (smart placement). |
|
||||
| 18 | FE-8200-018 | TODO | Task 11 | FE Guild | Add keyboard navigation (Escape to close). |
|
||||
| 19 | FE-8200-019 | TODO | Tasks 11-18 | QA Guild | Add unit tests for popover logic. |
|
||||
| 20 | FE-8200-020 | TODO | Tasks 11-18 | FE Guild | Add Storybook stories. |
|
||||
| **Wave 3 (Score Badges)** | | | | | |
|
||||
| 21 | FE-8200-021 | TODO | Task 0 | FE Guild | Create `ScoreBadgeComponent` with type input. |
|
||||
| 22 | FE-8200-022 | TODO | Task 21 | FE Guild | Implement "Live Signal" badge (green, pulse animation). |
|
||||
| 23 | FE-8200-023 | TODO | Task 21 | FE Guild | Implement "Proven Path" badge (blue, checkmark). |
|
||||
| 24 | FE-8200-024 | TODO | Task 21 | FE Guild | Implement "Vendor N/A" badge (gray, strikethrough). |
|
||||
| 25 | FE-8200-025 | TODO | Task 21 | FE Guild | Implement "Speculative" badge (orange, question mark). |
|
||||
| 26 | FE-8200-026 | TODO | Task 21 | FE Guild | Add tooltip with badge explanation. |
|
||||
| 27 | FE-8200-027 | TODO | Tasks 21-26 | QA Guild | Add unit tests for all badge types. |
|
||||
| 28 | FE-8200-028 | TODO | Tasks 21-26 | FE Guild | Add Storybook stories. |
|
||||
| **Wave 4 (Findings List Integration)** | | | | | |
|
||||
| 29 | FE-8200-029 | TODO | Wave 1-3 | FE Guild | Integrate ScorePillComponent into findings list. |
|
||||
| 30 | FE-8200-030 | TODO | Task 29 | FE Guild | Add score column to findings table. |
|
||||
| 31 | FE-8200-031 | TODO | Task 29 | FE Guild | Implement sort by score (ascending/descending). |
|
||||
| 32 | FE-8200-032 | TODO | Task 29 | FE Guild | Implement filter by bucket dropdown. |
|
||||
| 33 | FE-8200-033 | TODO | Task 29 | FE Guild | Implement filter by flags (checkboxes). |
|
||||
| 34 | FE-8200-034 | TODO | Task 29 | FE Guild | Add badges column showing active flags. |
|
||||
| 35 | FE-8200-035 | TODO | Task 29 | FE Guild | Integrate breakdown popover on pill click. |
|
||||
| 36 | FE-8200-036 | TODO | Tasks 29-35 | QA Guild | Add integration tests for list with scores. |
|
||||
| **Wave 5 (Score History)** | | | | | |
|
||||
| 37 | FE-8200-037 | TODO | Task 1 | FE Guild | Create `ScoreHistoryChartComponent`. |
|
||||
| 38 | FE-8200-038 | TODO | Task 37 | FE Guild | Implement line chart with ngx-charts or similar. |
|
||||
| 39 | FE-8200-039 | TODO | Task 37 | FE Guild | Add data points for each score change. |
|
||||
| 40 | FE-8200-040 | TODO | Task 37 | FE Guild | Implement hover tooltip with change details. |
|
||||
| 41 | FE-8200-041 | TODO | Task 37 | FE Guild | Add change type indicators (evidence update vs policy change). |
|
||||
| 42 | FE-8200-042 | TODO | Task 37 | FE Guild | Implement date range selector. |
|
||||
| 43 | FE-8200-043 | TODO | Task 37 | FE Guild | Add bucket band overlays (colored horizontal regions). |
|
||||
| 44 | FE-8200-044 | TODO | Tasks 37-43 | QA Guild | Add unit tests for chart component. |
|
||||
| 45 | FE-8200-045 | TODO | Tasks 37-43 | FE Guild | Add Storybook stories. |
|
||||
| **Wave 6 (Bulk Triage View)** | | | | | |
|
||||
| 46 | FE-8200-046 | TODO | Wave 4 | FE Guild | Create `BulkTriageViewComponent`. |
|
||||
| 47 | FE-8200-047 | TODO | Task 46 | FE Guild | Implement bucket summary cards (ActNow: N, ScheduleNext: M, etc.). |
|
||||
| 48 | FE-8200-048 | TODO | Task 46 | FE Guild | Implement "Select All in Bucket" action. |
|
||||
| 49 | FE-8200-049 | TODO | Task 46 | FE Guild | Implement bulk actions (Acknowledge, Suppress, Assign). |
|
||||
| 50 | FE-8200-050 | TODO | Task 46 | FE Guild | Add progress indicator for bulk operations. |
|
||||
| 51 | FE-8200-051 | TODO | Task 46 | FE Guild | Add undo capability for bulk actions. |
|
||||
| 52 | FE-8200-052 | TODO | Tasks 46-51 | QA Guild | Add integration tests for bulk triage. |
|
||||
| **Wave 7 (Accessibility & Polish)** | | | | | |
|
||||
| 53 | FE-8200-053 | TODO | All above | FE Guild | Audit all components with axe-core. |
|
||||
| 54 | FE-8200-054 | TODO | Task 53 | FE Guild | Add ARIA labels and roles. |
|
||||
| 55 | FE-8200-055 | TODO | Task 53 | FE Guild | Ensure keyboard navigation works throughout. |
|
||||
| 56 | FE-8200-056 | TODO | Task 53 | FE Guild | Add high contrast mode support. |
|
||||
| 57 | FE-8200-057 | TODO | Task 53 | FE Guild | Add screen reader announcements for score changes. |
|
||||
| 58 | FE-8200-058 | TODO | Tasks 53-57 | QA Guild | Run automated accessibility tests. |
|
||||
| **Wave 8 (Responsive Design)** | | | | | |
|
||||
| 59 | FE-8200-059 | TODO | All above | FE Guild | Test all components on mobile viewports. |
|
||||
| 60 | FE-8200-060 | TODO | Task 59 | FE Guild | Implement mobile-friendly popover (bottom sheet). |
|
||||
| 61 | FE-8200-061 | TODO | Task 59 | FE Guild | Implement compact table mode for mobile. |
|
||||
| 62 | FE-8200-062 | TODO | Task 59 | FE Guild | Add touch-friendly interactions. |
|
||||
| 63 | FE-8200-063 | TODO | Tasks 59-62 | QA Guild | Add visual regression tests for mobile. |
|
||||
| **Wave 9 (Documentation & Release)** | | | | | |
|
||||
| 64 | FE-8200-064 | TODO | All above | FE Guild | Complete Storybook documentation for all components. |
|
||||
| 65 | FE-8200-065 | TODO | Task 64 | FE Guild | Add usage examples and code snippets. |
|
||||
| 66 | FE-8200-066 | TODO | Task 64 | Docs Guild | Update `docs/ui/components/` with EWS components. |
|
||||
| 67 | FE-8200-067 | TODO | Task 64 | FE Guild | Create design tokens for score colors. |
|
||||
| 68 | FE-8200-068 | TODO | All above | QA Guild | Final E2E test suite for score features. |
|
||||
|
||||
---
|
||||
|
||||
## Component API Reference
|
||||
|
||||
### ScorePillComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stella-score-pill',
|
||||
template: `...`
|
||||
})
|
||||
export class ScorePillComponent {
|
||||
/** Score value (0-100) */
|
||||
@Input() score: number;
|
||||
|
||||
/** Size variant */
|
||||
@Input() size: 'sm' | 'md' | 'lg' = 'md';
|
||||
|
||||
/** Whether to show bucket tooltip on hover */
|
||||
@Input() showTooltip: boolean = true;
|
||||
|
||||
/** Emits when pill is clicked */
|
||||
@Output() pillClick = new EventEmitter<number>();
|
||||
}
|
||||
```
|
||||
|
||||
### ScoreBreakdownPopoverComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stella-score-breakdown-popover',
|
||||
template: `...`
|
||||
})
|
||||
export class ScoreBreakdownPopoverComponent {
|
||||
/** Full score result from API */
|
||||
@Input() scoreResult: EvidenceWeightedScoreResult;
|
||||
|
||||
/** Anchor element for positioning */
|
||||
@Input() anchorElement: HTMLElement;
|
||||
|
||||
/** Emits when popover should close */
|
||||
@Output() close = new EventEmitter<void>();
|
||||
}
|
||||
```
|
||||
|
||||
### ScoreBadgeComponent
|
||||
|
||||
```typescript
|
||||
@Component({
|
||||
selector: 'stella-score-badge',
|
||||
template: `...`
|
||||
})
|
||||
export class ScoreBadgeComponent {
|
||||
/** Badge type based on score flags */
|
||||
@Input() type: 'live-signal' | 'proven-path' | 'vendor-na' | 'speculative';
|
||||
|
||||
/** Size variant */
|
||||
@Input() size: 'sm' | 'md' = 'md';
|
||||
|
||||
/** Whether to show tooltip */
|
||||
@Input() showTooltip: boolean = true;
|
||||
}
|
||||
```
|
||||
|
||||
### ScoringService
|
||||
|
||||
```typescript
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class ScoringService {
|
||||
/** Calculate score for a single finding */
|
||||
calculateScore(findingId: string, options?: CalculateScoreOptions)
|
||||
: Observable<EvidenceWeightedScoreResult>;
|
||||
|
||||
/** Calculate scores for multiple findings */
|
||||
calculateScores(findingIds: string[], options?: CalculateScoreOptions)
|
||||
: Observable<BatchScoreResult>;
|
||||
|
||||
/** Get cached score */
|
||||
getScore(findingId: string): Observable<EvidenceWeightedScoreResult>;
|
||||
|
||||
/** Get score history */
|
||||
getScoreHistory(findingId: string, options?: HistoryOptions)
|
||||
: Observable<ScoreHistoryResult>;
|
||||
|
||||
/** Get current scoring policy */
|
||||
getScoringPolicy(): Observable<ScoringPolicy>;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 0-3 | Setup | Module created, service defined |
|
||||
| **Wave 1** | 4-10 | Score pill | Pill component with colors |
|
||||
| **Wave 2** | 11-20 | Breakdown popover | Full breakdown on hover |
|
||||
| **Wave 3** | 21-28 | Badges | All badge types |
|
||||
| **Wave 4** | 29-36 | List integration | Scores in findings list |
|
||||
| **Wave 5** | 37-45 | History chart | Timeline visualization |
|
||||
| **Wave 6** | 46-52 | Bulk triage | Multi-select by bucket |
|
||||
| **Wave 7** | 53-58 | Accessibility | WCAG 2.1 AA compliance |
|
||||
| **Wave 8** | 59-63 | Responsive | Mobile support |
|
||||
| **Wave 9** | 64-68 | Documentation | Storybook, docs complete |
|
||||
|
||||
---
|
||||
|
||||
## Interlocks
|
||||
|
||||
| Interlock | Description | Related Sprint/Module |
|
||||
|-----------|-------------|----------------------|
|
||||
| API endpoints | UI calls API from Sprint 0004 | 8200.0012.0004 |
|
||||
| Design system | Uses existing design tokens | UI/Design System |
|
||||
| Findings feature | Integrates with existing findings list | Findings/UI |
|
||||
| Storybook | Uses existing Storybook setup | UI/Storybook |
|
||||
| ngx-charts | May use for history chart | Third-party lib |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming Checkpoints
|
||||
|
||||
| Date (UTC) | Milestone | Evidence |
|
||||
|------------|-----------|----------|
|
||||
| 2026-05-19 | Wave 0-2 complete | Pill + breakdown popover work |
|
||||
| 2026-06-02 | Wave 3-4 complete | Badges + list integration |
|
||||
| 2026-06-16 | Wave 5-6 complete | History chart + bulk triage |
|
||||
| 2026-06-30 | Wave 7-9 complete | Accessibility, responsive, docs |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Bucket-based coloring | Matches advisory recommendation; clear visual hierarchy |
|
||||
| Popover for breakdown | Reduces visual clutter; progressive disclosure |
|
||||
| Bar chart for dimensions | Intuitive relative comparison |
|
||||
| Negative styling for mitigations | Visually indicates subtractive effect |
|
||||
| Smart popover positioning | Prevents viewport overflow |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Performance with many scores | Slow rendering | Virtual scrolling, lazy calculation | FE Guild |
|
||||
| Color contrast issues | Accessibility failure | Use design system colors, test contrast | FE Guild |
|
||||
| Popover z-index conflicts | Visual bugs | Use portal rendering | FE Guild |
|
||||
| Chart library compatibility | Angular version issues | Evaluate libraries early | FE Guild |
|
||||
| Mobile usability | Poor touch experience | Dedicated mobile testing | FE Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created for Frontend UI components. | Project Mgmt |
|
||||
321
docs/implplan/SPRINT_8200_0013_0001_GW_valkey_advisory_cache.md
Normal file
321
docs/implplan/SPRINT_8200_0013_0001_GW_valkey_advisory_cache.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# Sprint 8200.0013.0001 - Valkey Advisory Cache
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **Valkey-based caching** for canonical advisories to achieve p99 < 20ms read latency. This sprint delivers:
|
||||
|
||||
1. **Advisory Cache Keys**: `advisory:{merge_hash}` with TTL based on interest score
|
||||
2. **Hot Set Index**: `rank:hot` sorted set for top advisories
|
||||
3. **PURL Index**: `by:purl:{purl}` sets for fast artifact lookups
|
||||
4. **Cache Service**: Read-through cache with automatic population and invalidation
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Cache.Valkey/` (new)
|
||||
|
||||
**Evidence:** Advisory lookups return in < 20ms from Valkey; cache hit rate > 80% for repeated queries.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0003 (canonical service), existing Gateway Valkey infrastructure
|
||||
- **Blocks:** SPRINT_8200_0013_0002 (interest scoring - needs cache to store scores)
|
||||
- **Safe to run in parallel with:** SPRINT_8200_0013_0003 (SBOM scoring)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- `src/Gateway/StellaOps.Gateway.WebService/Configuration/GatewayOptions.cs` (Valkey config)
|
||||
- `docs/modules/router/messaging-valkey-transport.md`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Project Setup** | | | | | |
|
||||
| 0 | VCACHE-8200-000 | TODO | Gateway Valkey | Platform Guild | Review existing Gateway Valkey configuration and connection handling |
|
||||
| 1 | VCACHE-8200-001 | TODO | Task 0 | Concelier Guild | Create `StellaOps.Concelier.Cache.Valkey` project with StackExchange.Redis dependency |
|
||||
| 2 | VCACHE-8200-002 | TODO | Task 1 | Concelier Guild | Define `ConcelierCacheOptions` with connection string, database, TTL settings |
|
||||
| 3 | VCACHE-8200-003 | TODO | Task 2 | Concelier Guild | Implement `IConnectionMultiplexerFactory` for Valkey connection management |
|
||||
| **Wave 1: Key Schema Implementation** | | | | | |
|
||||
| 4 | VCACHE-8200-004 | TODO | Task 3 | Concelier Guild | Define `AdvisoryCacheKeys` static class with key patterns |
|
||||
| 5 | VCACHE-8200-005 | TODO | Task 4 | Concelier Guild | Implement `advisory:{merge_hash}` key serialization (JSON canonical advisory) |
|
||||
| 6 | VCACHE-8200-006 | TODO | Task 4 | Concelier Guild | Implement `rank:hot` sorted set operations (ZADD, ZRANGE, ZREM) |
|
||||
| 7 | VCACHE-8200-007 | TODO | Task 4 | Concelier Guild | Implement `by:purl:{purl}` set operations (SADD, SMEMBERS, SREM) |
|
||||
| 8 | VCACHE-8200-008 | TODO | Task 4 | Concelier Guild | Implement `by:cve:{cve}` mapping key |
|
||||
| 9 | VCACHE-8200-009 | TODO | Tasks 5-8 | QA Guild | Unit tests for key generation and serialization |
|
||||
| **Wave 2: Cache Service** | | | | | |
|
||||
| 10 | VCACHE-8200-010 | TODO | Task 9 | Concelier Guild | Define `IAdvisoryCacheService` interface |
|
||||
| 11 | VCACHE-8200-011 | TODO | Task 10 | Concelier Guild | Implement `ValkeyAdvisoryCacheService` with connection pooling |
|
||||
| 12 | VCACHE-8200-012 | TODO | Task 11 | Concelier Guild | Implement `GetAsync()` - read-through cache with Postgres fallback |
|
||||
| 13 | VCACHE-8200-013 | TODO | Task 12 | Concelier Guild | Implement `SetAsync()` - write with TTL based on interest score |
|
||||
| 14 | VCACHE-8200-014 | TODO | Task 13 | Concelier Guild | Implement `InvalidateAsync()` - remove from cache on update |
|
||||
| 15 | VCACHE-8200-015 | TODO | Task 14 | Concelier Guild | Implement `GetByPurlAsync()` - use PURL index for fast lookup |
|
||||
| 16 | VCACHE-8200-016 | TODO | Tasks 11-15 | QA Guild | Integration tests with Testcontainers (Valkey) |
|
||||
| **Wave 3: TTL Policy** | | | | | |
|
||||
| 17 | VCACHE-8200-017 | TODO | Task 16 | Concelier Guild | Define `CacheTtlPolicy` with score-based TTL tiers |
|
||||
| 18 | VCACHE-8200-018 | TODO | Task 17 | Concelier Guild | Implement TTL tier calculation: high (24h), medium (4h), low (1h) |
|
||||
| 19 | VCACHE-8200-019 | TODO | Task 18 | Concelier Guild | Implement background TTL refresh for hot advisories |
|
||||
| 20 | VCACHE-8200-020 | TODO | Task 19 | QA Guild | Test TTL expiration and refresh behavior |
|
||||
| **Wave 4: Index Management** | | | | | |
|
||||
| 21 | VCACHE-8200-021 | TODO | Task 16 | Concelier Guild | Implement hot set maintenance (add/remove on score change) |
|
||||
| 22 | VCACHE-8200-022 | TODO | Task 21 | Concelier Guild | Implement PURL index maintenance (add on ingest, remove on withdrawn) |
|
||||
| 23 | VCACHE-8200-023 | TODO | Task 22 | Concelier Guild | Implement `GetHotAdvisories()` - top N by interest score |
|
||||
| 24 | VCACHE-8200-024 | TODO | Task 23 | Concelier Guild | Implement cache warmup job for CI builds (preload hot set) |
|
||||
| 25 | VCACHE-8200-025 | TODO | Task 24 | QA Guild | Test index consistency under concurrent writes |
|
||||
| **Wave 5: Integration & Metrics** | | | | | |
|
||||
| 26 | VCACHE-8200-026 | TODO | Task 25 | Concelier Guild | Wire cache service into `CanonicalAdvisoryService` |
|
||||
| 27 | VCACHE-8200-027 | TODO | Task 26 | Concelier Guild | Add cache metrics: hit rate, latency, evictions |
|
||||
| 28 | VCACHE-8200-028 | TODO | Task 27 | Concelier Guild | Add OpenTelemetry spans for cache operations |
|
||||
| 29 | VCACHE-8200-029 | TODO | Task 28 | Concelier Guild | Implement fallback mode when Valkey unavailable |
|
||||
| 30 | VCACHE-8200-030 | TODO | Task 29 | QA Guild | Performance benchmark: verify p99 < 20ms |
|
||||
| 31 | VCACHE-8200-031 | TODO | Task 30 | Docs Guild | Document cache configuration and operations |
|
||||
|
||||
---
|
||||
|
||||
## Key Schema
|
||||
|
||||
```
|
||||
# Canonical advisory (JSON)
|
||||
advisory:{merge_hash} -> JSON(CanonicalAdvisory)
|
||||
TTL: Based on interest_score tier
|
||||
|
||||
# Hot advisory set (sorted by interest score)
|
||||
rank:hot -> ZSET { merge_hash: interest_score }
|
||||
Max size: 10,000 entries
|
||||
|
||||
# PURL index (set of merge_hashes affecting this PURL)
|
||||
by:purl:{normalized_purl} -> SET { merge_hash, ... }
|
||||
TTL: 24h (refreshed on access)
|
||||
|
||||
# CVE mapping (single merge_hash for primary CVE canonical)
|
||||
by:cve:{cve_id} -> STRING merge_hash
|
||||
TTL: 24h
|
||||
|
||||
# Cache metadata
|
||||
cache:stats:hits -> INCR counter
|
||||
cache:stats:misses -> INCR counter
|
||||
cache:warmup:last -> STRING ISO8601 timestamp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Cache.Valkey;
|
||||
|
||||
/// <summary>
|
||||
/// Valkey-based cache for canonical advisories.
|
||||
/// </summary>
|
||||
public interface IAdvisoryCacheService
|
||||
{
|
||||
// === Read Operations ===
|
||||
|
||||
/// <summary>Get canonical by merge hash (cache-first).</summary>
|
||||
Task<CanonicalAdvisory?> GetAsync(string mergeHash, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get canonicals by PURL (uses index).</summary>
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetByPurlAsync(string purl, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get canonical by CVE (uses mapping).</summary>
|
||||
Task<CanonicalAdvisory?> GetByCveAsync(string cve, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get hot advisories (top N by interest score).</summary>
|
||||
Task<IReadOnlyList<CanonicalAdvisory>> GetHotAsync(int limit = 100, CancellationToken ct = default);
|
||||
|
||||
// === Write Operations ===
|
||||
|
||||
/// <summary>Cache canonical with TTL based on interest score.</summary>
|
||||
Task SetAsync(CanonicalAdvisory advisory, double? interestScore = null, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Invalidate cached advisory.</summary>
|
||||
Task InvalidateAsync(string mergeHash, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Update interest score (affects TTL and hot set).</summary>
|
||||
Task UpdateScoreAsync(string mergeHash, double score, CancellationToken ct = default);
|
||||
|
||||
// === Index Operations ===
|
||||
|
||||
/// <summary>Add merge hash to PURL index.</summary>
|
||||
Task IndexPurlAsync(string purl, string mergeHash, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Remove merge hash from PURL index.</summary>
|
||||
Task UnindexPurlAsync(string purl, string mergeHash, CancellationToken ct = default);
|
||||
|
||||
// === Maintenance ===
|
||||
|
||||
/// <summary>Warm cache with hot advisories from database.</summary>
|
||||
Task WarmupAsync(int limit = 1000, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get cache statistics.</summary>
|
||||
Task<CacheStatistics> GetStatisticsAsync(CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed record CacheStatistics
|
||||
{
|
||||
public long Hits { get; init; }
|
||||
public long Misses { get; init; }
|
||||
public double HitRate => Hits + Misses > 0 ? (double)Hits / (Hits + Misses) : 0;
|
||||
public long HotSetSize { get; init; }
|
||||
public long TotalCachedAdvisories { get; init; }
|
||||
public DateTimeOffset? LastWarmup { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TTL Policy
|
||||
|
||||
```csharp
|
||||
public sealed class CacheTtlPolicy
|
||||
{
|
||||
public TimeSpan HighScoreTtl { get; init; } = TimeSpan.FromHours(24);
|
||||
public TimeSpan MediumScoreTtl { get; init; } = TimeSpan.FromHours(4);
|
||||
public TimeSpan LowScoreTtl { get; init; } = TimeSpan.FromHours(1);
|
||||
public double HighScoreThreshold { get; init; } = 0.7;
|
||||
public double MediumScoreThreshold { get; init; } = 0.4;
|
||||
|
||||
public TimeSpan GetTtl(double? score)
|
||||
{
|
||||
if (!score.HasValue) return LowScoreTtl;
|
||||
|
||||
return score.Value switch
|
||||
{
|
||||
>= 0.7 => HighScoreTtl, // High interest: 24h
|
||||
>= 0.4 => MediumScoreTtl, // Medium interest: 4h
|
||||
_ => LowScoreTtl // Low interest: 1h
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
```csharp
|
||||
public sealed class ConcelierCacheOptions
|
||||
{
|
||||
public const string SectionName = "Concelier:Cache";
|
||||
|
||||
/// <summary>Whether Valkey caching is enabled.</summary>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>Valkey connection string.</summary>
|
||||
public string ConnectionString { get; set; } = "localhost:6379";
|
||||
|
||||
/// <summary>Valkey database number (0-15).</summary>
|
||||
public int Database { get; set; } = 1;
|
||||
|
||||
/// <summary>Key prefix for all cache keys.</summary>
|
||||
public string KeyPrefix { get; set; } = "concelier:";
|
||||
|
||||
/// <summary>Maximum hot set size.</summary>
|
||||
public int MaxHotSetSize { get; set; } = 10_000;
|
||||
|
||||
/// <summary>Connection timeout.</summary>
|
||||
public TimeSpan ConnectTimeout { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Operation timeout.</summary>
|
||||
public TimeSpan OperationTimeout { get; set; } = TimeSpan.FromMilliseconds(100);
|
||||
|
||||
/// <summary>TTL policy configuration.</summary>
|
||||
public CacheTtlPolicy TtlPolicy { get; set; } = new();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Read-Through Pattern
|
||||
|
||||
```csharp
|
||||
public async Task<CanonicalAdvisory?> GetAsync(string mergeHash, CancellationToken ct)
|
||||
{
|
||||
var key = AdvisoryCacheKeys.Advisory(mergeHash);
|
||||
|
||||
// Try cache first
|
||||
var cached = await _redis.StringGetAsync(key);
|
||||
if (cached.HasValue)
|
||||
{
|
||||
await _redis.StringIncrementAsync(AdvisoryCacheKeys.StatsHits);
|
||||
return JsonSerializer.Deserialize<CanonicalAdvisory>(cached!);
|
||||
}
|
||||
|
||||
// Cache miss - load from database
|
||||
await _redis.StringIncrementAsync(AdvisoryCacheKeys.StatsMisses);
|
||||
var advisory = await _repository.GetByMergeHashAsync(mergeHash, ct);
|
||||
|
||||
if (advisory is not null)
|
||||
{
|
||||
// Populate cache
|
||||
var score = await GetInterestScoreAsync(advisory.Id, ct);
|
||||
await SetAsync(advisory, score, ct);
|
||||
}
|
||||
|
||||
return advisory;
|
||||
}
|
||||
```
|
||||
|
||||
### Hot Set Maintenance
|
||||
|
||||
```csharp
|
||||
public async Task UpdateScoreAsync(string mergeHash, double score, CancellationToken ct)
|
||||
{
|
||||
// Update hot set
|
||||
var hotKey = AdvisoryCacheKeys.HotSet;
|
||||
await _redis.SortedSetAddAsync(hotKey, mergeHash, score);
|
||||
|
||||
// Trim to max size
|
||||
var currentSize = await _redis.SortedSetLengthAsync(hotKey);
|
||||
if (currentSize > _options.MaxHotSetSize)
|
||||
{
|
||||
await _redis.SortedSetRemoveRangeByRankAsync(
|
||||
hotKey, 0, currentSize - _options.MaxHotSetSize - 1);
|
||||
}
|
||||
|
||||
// Update advisory TTL
|
||||
var advisoryKey = AdvisoryCacheKeys.Advisory(mergeHash);
|
||||
var ttl = _options.TtlPolicy.GetTtl(score);
|
||||
await _redis.KeyExpireAsync(advisoryKey, ttl);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| `concelier_cache_hits_total` | Counter | - | Total cache hits |
|
||||
| `concelier_cache_misses_total` | Counter | - | Total cache misses |
|
||||
| `concelier_cache_latency_ms` | Histogram | operation | Cache operation latency |
|
||||
| `concelier_cache_hot_set_size` | Gauge | - | Current hot set size |
|
||||
| `concelier_cache_evictions_total` | Counter | reason | Cache evictions (ttl, manual, trim) |
|
||||
|
||||
---
|
||||
|
||||
## Test Evidence Requirements
|
||||
|
||||
| Test | Evidence |
|
||||
|------|----------|
|
||||
| Cache hit | Repeated query returns cached value without DB call |
|
||||
| Cache miss | First query loads from DB, populates cache |
|
||||
| TTL expiration | Entry expires after TTL, next query reloads |
|
||||
| Hot set ordering | `GetHotAsync()` returns by descending score |
|
||||
| PURL index | `GetByPurlAsync()` returns all canonicals for PURL |
|
||||
| Fallback mode | Service works when Valkey unavailable (degraded) |
|
||||
| Performance | p99 latency < 20ms with 100K entries |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
429
docs/implplan/SPRINT_8200_0013_0002_CONCEL_interest_scoring.md
Normal file
429
docs/implplan/SPRINT_8200_0013_0002_CONCEL_interest_scoring.md
Normal file
@@ -0,0 +1,429 @@
|
||||
# Sprint 8200.0013.0002 - Interest Scoring Service
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **interest scoring** that learns which advisories matter to your organization. This sprint delivers:
|
||||
|
||||
1. **interest_score table**: Store per-canonical scores with reasons
|
||||
2. **InterestScoringService**: Compute scores from SBOM/VEX/runtime signals
|
||||
3. **Scoring Job**: Periodic batch recalculation of scores
|
||||
4. **Stub Degradation**: Demote low-interest advisories to lightweight stubs
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Interest/` (new)
|
||||
|
||||
**Evidence:** Advisories intersecting org SBOMs receive high scores; unused advisories degrade to stubs.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0003 (canonical service), SPRINT_8200_0013_0001 (Valkey cache)
|
||||
- **Blocks:** Nothing (feature complete for Phase B)
|
||||
- **Safe to run in parallel with:** SPRINT_8200_0013_0003 (SBOM scoring integration)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- `src/Excititor/__Libraries/StellaOps.Excititor.Core/TrustVector/` (existing scoring reference)
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Schema & Project Setup** | | | | | |
|
||||
| 0 | ISCORE-8200-000 | TODO | Canonical service | Platform Guild | Create migration `20250201000001_CreateInterestScore.sql` |
|
||||
| 1 | ISCORE-8200-001 | TODO | Task 0 | Concelier Guild | Create `StellaOps.Concelier.Interest` project |
|
||||
| 2 | ISCORE-8200-002 | TODO | Task 1 | Concelier Guild | Define `InterestScoreEntity` and repository interface |
|
||||
| 3 | ISCORE-8200-003 | TODO | Task 2 | Concelier Guild | Implement `PostgresInterestScoreRepository` |
|
||||
| 4 | ISCORE-8200-004 | TODO | Task 3 | QA Guild | Unit tests for repository CRUD |
|
||||
| **Wave 1: Scoring Algorithm** | | | | | |
|
||||
| 5 | ISCORE-8200-005 | TODO | Task 4 | Concelier Guild | Define `IInterestScoringService` interface |
|
||||
| 6 | ISCORE-8200-006 | TODO | Task 5 | Concelier Guild | Define `InterestScoreInput` with all signal types |
|
||||
| 7 | ISCORE-8200-007 | TODO | Task 6 | Concelier Guild | Implement `InterestScoreCalculator` with weighted factors |
|
||||
| 8 | ISCORE-8200-008 | TODO | Task 7 | Concelier Guild | Implement SBOM intersection factor (`in_sbom`) |
|
||||
| 9 | ISCORE-8200-009 | TODO | Task 8 | Concelier Guild | Implement reachability factor (`reachable`) |
|
||||
| 10 | ISCORE-8200-010 | TODO | Task 9 | Concelier Guild | Implement deployment factor (`deployed`) |
|
||||
| 11 | ISCORE-8200-011 | TODO | Task 10 | Concelier Guild | Implement VEX factor (`no_vex_na`) |
|
||||
| 12 | ISCORE-8200-012 | TODO | Task 11 | Concelier Guild | Implement age decay factor (`recent`) |
|
||||
| 13 | ISCORE-8200-013 | TODO | Tasks 8-12 | QA Guild | Unit tests for score calculation with various inputs |
|
||||
| **Wave 2: Scoring Service** | | | | | |
|
||||
| 14 | ISCORE-8200-014 | TODO | Task 13 | Concelier Guild | Implement `InterestScoringService.ComputeScoreAsync()` |
|
||||
| 15 | ISCORE-8200-015 | TODO | Task 14 | Concelier Guild | Implement `UpdateScoreAsync()` - persist + update cache |
|
||||
| 16 | ISCORE-8200-016 | TODO | Task 15 | Concelier Guild | Implement `GetScoreAsync()` - cached score retrieval |
|
||||
| 17 | ISCORE-8200-017 | TODO | Task 16 | Concelier Guild | Implement `BatchUpdateAsync()` - bulk score updates |
|
||||
| 18 | ISCORE-8200-018 | TODO | Task 17 | QA Guild | Integration tests with Postgres + Valkey |
|
||||
| **Wave 3: Scoring Job** | | | | | |
|
||||
| 19 | ISCORE-8200-019 | TODO | Task 18 | Concelier Guild | Create `InterestScoreRecalculationJob` hosted service |
|
||||
| 20 | ISCORE-8200-020 | TODO | Task 19 | Concelier Guild | Implement incremental scoring (only changed advisories) |
|
||||
| 21 | ISCORE-8200-021 | TODO | Task 20 | Concelier Guild | Implement full recalculation mode (nightly) |
|
||||
| 22 | ISCORE-8200-022 | TODO | Task 21 | Concelier Guild | Add job metrics and OpenTelemetry tracing |
|
||||
| 23 | ISCORE-8200-023 | TODO | Task 22 | QA Guild | Test job execution and score consistency |
|
||||
| **Wave 4: Stub Degradation** | | | | | |
|
||||
| 24 | ISCORE-8200-024 | TODO | Task 18 | Concelier Guild | Define stub degradation policy (score threshold, retention) |
|
||||
| 25 | ISCORE-8200-025 | TODO | Task 24 | Concelier Guild | Implement `DegradeToStubAsync()` - convert full to stub |
|
||||
| 26 | ISCORE-8200-026 | TODO | Task 25 | Concelier Guild | Implement `RestoreFromStubAsync()` - promote on score increase |
|
||||
| 27 | ISCORE-8200-027 | TODO | Task 26 | Concelier Guild | Create `StubDegradationJob` for periodic cleanup |
|
||||
| 28 | ISCORE-8200-028 | TODO | Task 27 | QA Guild | Test degradation/restoration cycle |
|
||||
| **Wave 5: API & Integration** | | | | | |
|
||||
| 29 | ISCORE-8200-029 | TODO | Task 28 | Concelier Guild | Create `GET /api/v1/canonical/{id}/score` endpoint |
|
||||
| 30 | ISCORE-8200-030 | TODO | Task 29 | Concelier Guild | Add score to canonical advisory response |
|
||||
| 31 | ISCORE-8200-031 | TODO | Task 30 | Concelier Guild | Create `POST /api/v1/scores/recalculate` admin endpoint |
|
||||
| 32 | ISCORE-8200-032 | TODO | Task 31 | QA Guild | End-to-end test: ingest advisory, update SBOM, verify score change |
|
||||
| 33 | ISCORE-8200-033 | TODO | Task 32 | Docs Guild | Document interest scoring in module README |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
-- Migration: 20250201000001_CreateInterestScore.sql
|
||||
|
||||
CREATE TABLE vuln.interest_score (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
score NUMERIC(3,2) NOT NULL CHECK (score >= 0 AND score <= 1),
|
||||
reasons JSONB NOT NULL DEFAULT '[]',
|
||||
last_seen_in_build UUID,
|
||||
computed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT uq_interest_score_canonical UNIQUE (canonical_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_interest_score_score ON vuln.interest_score(score DESC);
|
||||
CREATE INDEX idx_interest_score_computed ON vuln.interest_score(computed_at DESC);
|
||||
|
||||
-- Partial index for high-interest advisories
|
||||
CREATE INDEX idx_interest_score_high ON vuln.interest_score(canonical_id)
|
||||
WHERE score >= 0.7;
|
||||
|
||||
COMMENT ON TABLE vuln.interest_score IS 'Per-canonical interest scores based on org signals';
|
||||
COMMENT ON COLUMN vuln.interest_score.reasons IS 'Array of reason codes: in_sbom, reachable, deployed, no_vex_na, recent';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scoring Algorithm
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Interest;
|
||||
|
||||
public sealed class InterestScoreCalculator
|
||||
{
|
||||
private readonly InterestScoreWeights _weights;
|
||||
|
||||
public InterestScoreCalculator(InterestScoreWeights weights)
|
||||
{
|
||||
_weights = weights;
|
||||
}
|
||||
|
||||
public InterestScore Calculate(InterestScoreInput input)
|
||||
{
|
||||
var reasons = new List<string>();
|
||||
double score = 0.0;
|
||||
|
||||
// Factor 1: In SBOM (30%)
|
||||
if (input.SbomMatches.Count > 0)
|
||||
{
|
||||
score += _weights.InSbom;
|
||||
reasons.Add("in_sbom");
|
||||
}
|
||||
|
||||
// Factor 2: Reachable from entrypoint (25%)
|
||||
if (input.SbomMatches.Any(m => m.IsReachable))
|
||||
{
|
||||
score += _weights.Reachable;
|
||||
reasons.Add("reachable");
|
||||
}
|
||||
|
||||
// Factor 3: Deployed in production (20%)
|
||||
if (input.SbomMatches.Any(m => m.IsDeployed))
|
||||
{
|
||||
score += _weights.Deployed;
|
||||
reasons.Add("deployed");
|
||||
}
|
||||
|
||||
// Factor 4: No VEX Not-Affected (15%)
|
||||
if (!input.VexStatements.Any(v => v.Status == VexStatus.NotAffected))
|
||||
{
|
||||
score += _weights.NoVexNotAffected;
|
||||
reasons.Add("no_vex_na");
|
||||
}
|
||||
|
||||
// Factor 5: Age decay (10%) - newer builds = higher score
|
||||
if (input.LastSeenInBuild.HasValue)
|
||||
{
|
||||
var age = DateTimeOffset.UtcNow - input.LastSeenInBuild.Value;
|
||||
var decayFactor = Math.Max(0, 1 - (age.TotalDays / 365));
|
||||
var ageScore = _weights.Recent * decayFactor;
|
||||
score += ageScore;
|
||||
if (decayFactor > 0.5)
|
||||
{
|
||||
reasons.Add("recent");
|
||||
}
|
||||
}
|
||||
|
||||
return new InterestScore
|
||||
{
|
||||
CanonicalId = input.CanonicalId,
|
||||
Score = Math.Round(Math.Min(score, 1.0), 2),
|
||||
Reasons = reasons.ToArray(),
|
||||
ComputedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record InterestScoreWeights
|
||||
{
|
||||
public double InSbom { get; init; } = 0.30;
|
||||
public double Reachable { get; init; } = 0.25;
|
||||
public double Deployed { get; init; } = 0.20;
|
||||
public double NoVexNotAffected { get; init; } = 0.15;
|
||||
public double Recent { get; init; } = 0.10;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Models
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Interest score for a canonical advisory.
|
||||
/// </summary>
|
||||
public sealed record InterestScore
|
||||
{
|
||||
public Guid CanonicalId { get; init; }
|
||||
public double Score { get; init; }
|
||||
public IReadOnlyList<string> Reasons { get; init; } = [];
|
||||
public Guid? LastSeenInBuild { get; init; }
|
||||
public DateTimeOffset ComputedAt { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Input signals for interest score calculation.
|
||||
/// </summary>
|
||||
public sealed record InterestScoreInput
|
||||
{
|
||||
public required Guid CanonicalId { get; init; }
|
||||
public IReadOnlyList<SbomMatch> SbomMatches { get; init; } = [];
|
||||
public IReadOnlyList<VexStatement> VexStatements { get; init; } = [];
|
||||
public IReadOnlyList<RuntimeSignal> RuntimeSignals { get; init; } = [];
|
||||
public DateTimeOffset? LastSeenInBuild { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SBOM match indicating canonical affects a package in an org's SBOM.
|
||||
/// </summary>
|
||||
public sealed record SbomMatch
|
||||
{
|
||||
public required string SbomDigest { get; init; }
|
||||
public required string Purl { get; init; }
|
||||
public bool IsReachable { get; init; }
|
||||
public bool IsDeployed { get; init; }
|
||||
public DateTimeOffset ScannedAt { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// VEX statement affecting the canonical.
|
||||
/// </summary>
|
||||
public sealed record VexStatement
|
||||
{
|
||||
public required string StatementId { get; init; }
|
||||
public required VexStatus Status { get; init; }
|
||||
public string? Justification { get; init; }
|
||||
}
|
||||
|
||||
public enum VexStatus
|
||||
{
|
||||
Affected,
|
||||
NotAffected,
|
||||
Fixed,
|
||||
UnderInvestigation
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service Interface
|
||||
|
||||
```csharp
|
||||
public interface IInterestScoringService
|
||||
{
|
||||
/// <summary>Compute interest score for a canonical advisory.</summary>
|
||||
Task<InterestScore> ComputeScoreAsync(Guid canonicalId, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get current interest score (cached).</summary>
|
||||
Task<InterestScore?> GetScoreAsync(Guid canonicalId, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Update interest score and persist.</summary>
|
||||
Task UpdateScoreAsync(InterestScore score, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Batch update scores for multiple canonicals.</summary>
|
||||
Task BatchUpdateAsync(IEnumerable<Guid> canonicalIds, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Trigger full recalculation for all active canonicals.</summary>
|
||||
Task RecalculateAllAsync(CancellationToken ct = default);
|
||||
|
||||
/// <summary>Degrade low-interest canonicals to stub status.</summary>
|
||||
Task<int> DegradeToStubsAsync(double threshold, CancellationToken ct = default);
|
||||
|
||||
/// <summary>Restore stubs to active when score increases.</summary>
|
||||
Task<int> RestoreFromStubsAsync(double threshold, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stub Degradation Policy
|
||||
|
||||
```csharp
|
||||
public sealed class StubDegradationPolicy
|
||||
{
|
||||
/// <summary>Score below which canonicals become stubs.</summary>
|
||||
public double DegradationThreshold { get; init; } = 0.2;
|
||||
|
||||
/// <summary>Score above which stubs are restored to active.</summary>
|
||||
public double RestorationThreshold { get; init; } = 0.4;
|
||||
|
||||
/// <summary>Minimum age before degradation (days).</summary>
|
||||
public int MinAgeDays { get; init; } = 30;
|
||||
|
||||
/// <summary>Maximum stubs to process per job run.</summary>
|
||||
public int BatchSize { get; init; } = 1000;
|
||||
}
|
||||
```
|
||||
|
||||
### Stub Content
|
||||
|
||||
When an advisory is degraded to stub, only these fields are retained:
|
||||
|
||||
| Field | Retained | Reason |
|
||||
|-------|----------|--------|
|
||||
| `id`, `merge_hash` | Yes | Identity |
|
||||
| `cve`, `affects_key` | Yes | Lookup keys |
|
||||
| `severity`, `exploit_known` | Yes | Quick triage |
|
||||
| `title` | Yes | Human reference |
|
||||
| `summary`, `version_range` | No | Space savings |
|
||||
| Source edges | First only | Reduces storage |
|
||||
|
||||
---
|
||||
|
||||
## Scoring Job
|
||||
|
||||
```csharp
|
||||
public sealed class InterestScoreRecalculationJob : BackgroundService
|
||||
{
|
||||
private readonly IServiceProvider _services;
|
||||
private readonly ILogger<InterestScoreRecalculationJob> _logger;
|
||||
private readonly InterestScoreJobOptions _options;
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var scope = _services.CreateAsyncScope();
|
||||
var scoringService = scope.ServiceProvider
|
||||
.GetRequiredService<IInterestScoringService>();
|
||||
|
||||
if (IsFullRecalculationTime())
|
||||
{
|
||||
_logger.LogInformation("Starting full interest score recalculation");
|
||||
await scoringService.RecalculateAllAsync(stoppingToken);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation("Starting incremental interest score update");
|
||||
var changedIds = await GetChangedCanonicalIdsAsync(stoppingToken);
|
||||
await scoringService.BatchUpdateAsync(changedIds, stoppingToken);
|
||||
}
|
||||
|
||||
// Run stub degradation
|
||||
var degraded = await scoringService.DegradeToStubsAsync(
|
||||
_options.DegradationThreshold, stoppingToken);
|
||||
_logger.LogInformation("Degraded {Count} advisories to stubs", degraded);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Interest score job failed");
|
||||
}
|
||||
|
||||
await Task.Delay(_options.Interval, stoppingToken);
|
||||
}
|
||||
}
|
||||
|
||||
private bool IsFullRecalculationTime()
|
||||
{
|
||||
// Full recalculation at 3 AM UTC daily
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
return now.Hour == 3 && now.Minute < _options.Interval.TotalMinutes;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
```csharp
|
||||
// GET /api/v1/canonical/{id}/score
|
||||
app.MapGet("/api/v1/canonical/{id:guid}/score", async (
|
||||
Guid id,
|
||||
IInterestScoringService scoringService,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var score = await scoringService.GetScoreAsync(id, ct);
|
||||
return score is null ? Results.NotFound() : Results.Ok(score);
|
||||
})
|
||||
.WithName("GetInterestScore")
|
||||
.Produces<InterestScore>(200);
|
||||
|
||||
// POST /api/v1/scores/recalculate (admin)
|
||||
app.MapPost("/api/v1/scores/recalculate", async (
|
||||
IInterestScoringService scoringService,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
await scoringService.RecalculateAllAsync(ct);
|
||||
return Results.Accepted();
|
||||
})
|
||||
.WithName("RecalculateScores")
|
||||
.RequireAuthorization("admin")
|
||||
.Produces(202);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| `concelier_interest_score_computed_total` | Counter | - | Total scores computed |
|
||||
| `concelier_interest_score_distribution` | Histogram | - | Score value distribution |
|
||||
| `concelier_stub_degradations_total` | Counter | - | Total stub degradations |
|
||||
| `concelier_stub_restorations_total` | Counter | - | Total stub restorations |
|
||||
| `concelier_scoring_job_duration_seconds` | Histogram | mode | Job execution time |
|
||||
|
||||
---
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
| Scenario | Expected Score | Reasons |
|
||||
|----------|---------------|---------|
|
||||
| Advisory in SBOM, reachable, deployed | 0.75+ | in_sbom, reachable, deployed |
|
||||
| Advisory in SBOM only | 0.30 | in_sbom |
|
||||
| Advisory with VEX not_affected | 0.00 | (none - excluded by VEX) |
|
||||
| Advisory not in any SBOM | 0.00 | (none) |
|
||||
| Stale advisory (> 1 year) | ~0.00-0.10 | age decay |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,474 @@
|
||||
# Sprint 8200.0013.0003 - SBOM Intersection Scoring
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **SBOM-based interest scoring integration** that connects Scanner SBOMs to Concelier interest scores. This sprint delivers:
|
||||
|
||||
1. **Learn SBOM Endpoint**: `POST /api/v1/learn/sbom` to register org SBOMs
|
||||
2. **SBOM Matching Service**: Find canonical advisories affecting SBOM components
|
||||
3. **Score Updates**: Trigger interest score recalculation on SBOM changes
|
||||
4. **BOM Index Integration**: Use existing BOM Index for fast PURL lookups
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.SbomIntegration/` (new)
|
||||
|
||||
**Evidence:** Registering an SBOM updates interest scores for all affected advisories within 5 minutes.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0013_0002 (interest scoring), Scanner BOM Index
|
||||
- **Blocks:** Nothing
|
||||
- **Safe to run in parallel with:** SPRINT_8200_0013_0001 (Valkey cache)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- Scanner BOM Index documentation
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Emit/Index/BomIndexBuilder.cs`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Project Setup** | | | | | |
|
||||
| 0 | SBOM-8200-000 | TODO | Interest scoring | Concelier Guild | Create `StellaOps.Concelier.SbomIntegration` project |
|
||||
| 1 | SBOM-8200-001 | TODO | Task 0 | Concelier Guild | Define `ISbomRegistryService` interface |
|
||||
| 2 | SBOM-8200-002 | TODO | Task 1 | Platform Guild | Create `vuln.sbom_registry` table for tracking registered SBOMs |
|
||||
| 3 | SBOM-8200-003 | TODO | Task 2 | Concelier Guild | Implement `PostgresSbomRegistryRepository` |
|
||||
| **Wave 1: SBOM Registration** | | | | | |
|
||||
| 4 | SBOM-8200-004 | TODO | Task 3 | Concelier Guild | Implement `RegisterSbomAsync()` - store SBOM reference |
|
||||
| 5 | SBOM-8200-005 | TODO | Task 4 | Concelier Guild | Implement PURL extraction from SBOM (CycloneDX/SPDX) |
|
||||
| 6 | SBOM-8200-006 | TODO | Task 5 | Concelier Guild | Create PURL→canonical mapping cache |
|
||||
| 7 | SBOM-8200-007 | TODO | Task 6 | QA Guild | Unit tests for SBOM registration and PURL extraction |
|
||||
| **Wave 2: Advisory Matching** | | | | | |
|
||||
| 8 | SBOM-8200-008 | TODO | Task 7 | Concelier Guild | Define `ISbomAdvisoryMatcher` interface |
|
||||
| 9 | SBOM-8200-009 | TODO | Task 8 | Concelier Guild | Implement PURL-based matching (exact + version range) |
|
||||
| 10 | SBOM-8200-010 | TODO | Task 9 | Concelier Guild | Implement CPE-based matching for OS packages |
|
||||
| 11 | SBOM-8200-011 | TODO | Task 10 | Concelier Guild | Integrate with Valkey PURL index for fast lookups |
|
||||
| 12 | SBOM-8200-012 | TODO | Task 11 | QA Guild | Matching tests with various package ecosystems |
|
||||
| **Wave 3: Score Integration** | | | | | |
|
||||
| 13 | SBOM-8200-013 | TODO | Task 12 | Concelier Guild | Implement `LearnSbomAsync()` - orchestrates full flow |
|
||||
| 14 | SBOM-8200-014 | TODO | Task 13 | Concelier Guild | Create `SbomMatch` records linking SBOM to canonicals |
|
||||
| 15 | SBOM-8200-015 | TODO | Task 14 | Concelier Guild | Trigger interest score updates for matched canonicals |
|
||||
| 16 | SBOM-8200-016 | TODO | Task 15 | Concelier Guild | Implement incremental matching (delta SBOMs) |
|
||||
| 17 | SBOM-8200-017 | TODO | Task 16 | QA Guild | Integration tests: register SBOM → score updates |
|
||||
| **Wave 4: Reachability Integration** | | | | | |
|
||||
| 18 | SBOM-8200-018 | TODO | Task 17 | Concelier Guild | Query Scanner reachability data for matched components |
|
||||
| 19 | SBOM-8200-019 | TODO | Task 18 | Concelier Guild | Include reachability in SbomMatch (IsReachable flag) |
|
||||
| 20 | SBOM-8200-020 | TODO | Task 19 | Concelier Guild | Update interest scores with reachability factor |
|
||||
| 21 | SBOM-8200-021 | TODO | Task 20 | QA Guild | Test reachability-aware scoring |
|
||||
| **Wave 5: API & Events** | | | | | |
|
||||
| 22 | SBOM-8200-022 | TODO | Task 21 | Concelier Guild | Create `POST /api/v1/learn/sbom` endpoint |
|
||||
| 23 | SBOM-8200-023 | TODO | Task 22 | Concelier Guild | Create `GET /api/v1/sboms/{digest}/affected` endpoint |
|
||||
| 24 | SBOM-8200-024 | TODO | Task 23 | Concelier Guild | Emit `SbomLearned` event for downstream consumers |
|
||||
| 25 | SBOM-8200-025 | TODO | Task 24 | Concelier Guild | Subscribe to Scanner `ScanCompleted` events for auto-learning |
|
||||
| 26 | SBOM-8200-026 | TODO | Task 25 | QA Guild | End-to-end test: scan image → SBOM registered → scores updated |
|
||||
| 27 | SBOM-8200-027 | TODO | Task 26 | Docs Guild | Document SBOM learning API and integration |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
-- Migration: 20250301000001_CreateSbomRegistry.sql
|
||||
|
||||
CREATE TABLE vuln.sbom_registry (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
artifact_id TEXT NOT NULL, -- Image digest or artifact identifier
|
||||
sbom_digest TEXT NOT NULL, -- SHA256 of SBOM content
|
||||
sbom_format TEXT NOT NULL, -- cyclonedx, spdx
|
||||
component_count INT NOT NULL DEFAULT 0,
|
||||
registered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_matched_at TIMESTAMPTZ,
|
||||
|
||||
CONSTRAINT uq_sbom_registry_digest UNIQUE (tenant_id, sbom_digest)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sbom_registry_tenant ON vuln.sbom_registry(tenant_id);
|
||||
CREATE INDEX idx_sbom_registry_artifact ON vuln.sbom_registry(artifact_id);
|
||||
|
||||
-- Junction table for SBOM component matches
|
||||
CREATE TABLE vuln.sbom_canonical_match (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
sbom_id UUID NOT NULL REFERENCES vuln.sbom_registry(id) ON DELETE CASCADE,
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
purl TEXT NOT NULL,
|
||||
is_reachable BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
is_deployed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
matched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT uq_sbom_canonical_match UNIQUE (sbom_id, canonical_id, purl)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sbom_match_canonical ON vuln.sbom_canonical_match(canonical_id);
|
||||
CREATE INDEX idx_sbom_match_sbom ON vuln.sbom_canonical_match(sbom_id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service Interfaces
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.SbomIntegration;
|
||||
|
||||
/// <summary>
|
||||
/// Service for registering and querying org SBOMs.
|
||||
/// </summary>
|
||||
public interface ISbomRegistryService
|
||||
{
|
||||
/// <summary>Register an SBOM for interest tracking.</summary>
|
||||
Task<SbomRegistration> RegisterAsync(
|
||||
Guid tenantId,
|
||||
string artifactId,
|
||||
string sbomDigest,
|
||||
Stream sbomContent,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get registration by SBOM digest.</summary>
|
||||
Task<SbomRegistration?> GetByDigestAsync(
|
||||
Guid tenantId,
|
||||
string sbomDigest,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>List all SBOMs for a tenant.</summary>
|
||||
Task<IReadOnlyList<SbomRegistration>> ListAsync(
|
||||
Guid tenantId,
|
||||
int limit = 100,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Unregister an SBOM.</summary>
|
||||
Task UnregisterAsync(
|
||||
Guid tenantId,
|
||||
string sbomDigest,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Service for matching SBOMs to canonical advisories.
|
||||
/// </summary>
|
||||
public interface ISbomAdvisoryMatcher
|
||||
{
|
||||
/// <summary>Find canonical advisories affecting SBOM components.</summary>
|
||||
Task<IReadOnlyList<SbomCanonicalMatch>> MatchAsync(
|
||||
SbomRegistration sbom,
|
||||
IReadOnlyList<string> purls,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get all matches for a canonical advisory.</summary>
|
||||
Task<IReadOnlyList<SbomCanonicalMatch>> GetMatchesForCanonicalAsync(
|
||||
Guid canonicalId,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Orchestrates SBOM learning and score updates.
|
||||
/// </summary>
|
||||
public interface ISbomLearningService
|
||||
{
|
||||
/// <summary>Learn from SBOM and update interest scores.</summary>
|
||||
Task<SbomLearningResult> LearnAsync(
|
||||
Guid tenantId,
|
||||
string artifactId,
|
||||
string sbomDigest,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Learn from runtime signals (deployment, reachability).</summary>
|
||||
Task<RuntimeLearningResult> LearnRuntimeAsync(
|
||||
Guid tenantId,
|
||||
string artifactId,
|
||||
IReadOnlyList<RuntimeSignal> signals,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Models
|
||||
|
||||
```csharp
|
||||
public sealed record SbomRegistration
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public Guid TenantId { get; init; }
|
||||
public required string ArtifactId { get; init; }
|
||||
public required string SbomDigest { get; init; }
|
||||
public required string SbomFormat { get; init; }
|
||||
public int ComponentCount { get; init; }
|
||||
public DateTimeOffset RegisteredAt { get; init; }
|
||||
public DateTimeOffset? LastMatchedAt { get; init; }
|
||||
}
|
||||
|
||||
public sealed record SbomCanonicalMatch
|
||||
{
|
||||
public Guid SbomId { get; init; }
|
||||
public Guid CanonicalId { get; init; }
|
||||
public required string Purl { get; init; }
|
||||
public bool IsReachable { get; init; }
|
||||
public bool IsDeployed { get; init; }
|
||||
public DateTimeOffset MatchedAt { get; init; }
|
||||
}
|
||||
|
||||
public sealed record SbomLearningResult
|
||||
{
|
||||
public required string SbomDigest { get; init; }
|
||||
public int ComponentsProcessed { get; init; }
|
||||
public int AdvisoriesMatched { get; init; }
|
||||
public int ScoresUpdated { get; init; }
|
||||
public TimeSpan Duration { get; init; }
|
||||
}
|
||||
|
||||
public sealed record RuntimeSignal
|
||||
{
|
||||
public required string Purl { get; init; }
|
||||
public required RuntimeSignalType Type { get; init; }
|
||||
public DateTimeOffset ObservedAt { get; init; }
|
||||
public Dictionary<string, string> Metadata { get; init; } = new();
|
||||
}
|
||||
|
||||
public enum RuntimeSignalType
|
||||
{
|
||||
Deployed,
|
||||
Reachable,
|
||||
Executed,
|
||||
NetworkActive
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SBOM Parsing
|
||||
|
||||
```csharp
|
||||
public sealed class SbomParser
|
||||
{
|
||||
public IReadOnlyList<string> ExtractPurls(Stream sbomContent, string format)
|
||||
{
|
||||
return format.ToLowerInvariant() switch
|
||||
{
|
||||
"cyclonedx" => ParseCycloneDx(sbomContent),
|
||||
"spdx" => ParseSpdx(sbomContent),
|
||||
_ => throw new NotSupportedException($"SBOM format '{format}' not supported")
|
||||
};
|
||||
}
|
||||
|
||||
private IReadOnlyList<string> ParseCycloneDx(Stream content)
|
||||
{
|
||||
using var doc = JsonDocument.Parse(content);
|
||||
var purls = new List<string>();
|
||||
|
||||
if (doc.RootElement.TryGetProperty("components", out var components))
|
||||
{
|
||||
foreach (var component in components.EnumerateArray())
|
||||
{
|
||||
if (component.TryGetProperty("purl", out var purl))
|
||||
{
|
||||
purls.Add(purl.GetString()!);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return purls;
|
||||
}
|
||||
|
||||
private IReadOnlyList<string> ParseSpdx(Stream content)
|
||||
{
|
||||
using var doc = JsonDocument.Parse(content);
|
||||
var purls = new List<string>();
|
||||
|
||||
if (doc.RootElement.TryGetProperty("packages", out var packages))
|
||||
{
|
||||
foreach (var package in packages.EnumerateArray())
|
||||
{
|
||||
if (package.TryGetProperty("externalRefs", out var refs))
|
||||
{
|
||||
foreach (var extRef in refs.EnumerateArray())
|
||||
{
|
||||
if (extRef.TryGetProperty("referenceType", out var refType) &&
|
||||
refType.GetString() == "purl" &&
|
||||
extRef.TryGetProperty("referenceLocator", out var locator))
|
||||
{
|
||||
purls.Add(locator.GetString()!);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return purls;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Learning Flow
|
||||
|
||||
```csharp
|
||||
public async Task<SbomLearningResult> LearnAsync(
|
||||
Guid tenantId,
|
||||
string artifactId,
|
||||
string sbomDigest,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
|
||||
// 1. Register SBOM if not already registered
|
||||
var registration = await _registryService.GetByDigestAsync(tenantId, sbomDigest, ct);
|
||||
if (registration is null)
|
||||
{
|
||||
var sbomContent = await _sbomStore.GetAsync(sbomDigest, ct);
|
||||
registration = await _registryService.RegisterAsync(
|
||||
tenantId, artifactId, sbomDigest, sbomContent, ct);
|
||||
}
|
||||
|
||||
// 2. Extract PURLs from SBOM
|
||||
var sbomContent = await _sbomStore.GetAsync(sbomDigest, ct);
|
||||
var purls = _sbomParser.ExtractPurls(sbomContent, registration.SbomFormat);
|
||||
|
||||
// 3. Match PURLs to canonical advisories
|
||||
var matches = await _matcher.MatchAsync(registration, purls, ct);
|
||||
|
||||
// 4. Fetch reachability data from Scanner
|
||||
var reachabilityData = await _scannerClient.GetReachabilityAsync(sbomDigest, ct);
|
||||
matches = EnrichWithReachability(matches, reachabilityData);
|
||||
|
||||
// 5. Persist matches
|
||||
await _matchRepository.UpsertBatchAsync(matches, ct);
|
||||
|
||||
// 6. Update interest scores for matched canonicals
|
||||
var canonicalIds = matches.Select(m => m.CanonicalId).Distinct().ToList();
|
||||
await _scoringService.BatchUpdateAsync(canonicalIds, ct);
|
||||
|
||||
// 7. Emit event
|
||||
await _eventBus.PublishAsync(new SbomLearned
|
||||
{
|
||||
TenantId = tenantId,
|
||||
SbomDigest = sbomDigest,
|
||||
CanonicalIdsAffected = canonicalIds
|
||||
}, ct);
|
||||
|
||||
return new SbomLearningResult
|
||||
{
|
||||
SbomDigest = sbomDigest,
|
||||
ComponentsProcessed = purls.Count,
|
||||
AdvisoriesMatched = matches.Count,
|
||||
ScoresUpdated = canonicalIds.Count,
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
```csharp
|
||||
// POST /api/v1/learn/sbom
|
||||
app.MapPost("/api/v1/learn/sbom", async (
|
||||
LearnSbomRequest request,
|
||||
ISbomLearningService learningService,
|
||||
ClaimsPrincipal user,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var tenantId = user.GetTenantId();
|
||||
var result = await learningService.LearnAsync(
|
||||
tenantId, request.ArtifactId, request.SbomDigest, ct);
|
||||
return Results.Ok(result);
|
||||
})
|
||||
.WithName("LearnSbom")
|
||||
.WithSummary("Register SBOM and update interest scores")
|
||||
.Produces<SbomLearningResult>(200);
|
||||
|
||||
// GET /api/v1/sboms/{digest}/affected
|
||||
app.MapGet("/api/v1/sboms/{digest}/affected", async (
|
||||
string digest,
|
||||
ISbomAdvisoryMatcher matcher,
|
||||
ISbomRegistryService registry,
|
||||
ClaimsPrincipal user,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var tenantId = user.GetTenantId();
|
||||
var registration = await registry.GetByDigestAsync(tenantId, digest, ct);
|
||||
if (registration is null) return Results.NotFound();
|
||||
|
||||
var purls = await GetPurlsFromSbom(digest, ct);
|
||||
var matches = await matcher.MatchAsync(registration, purls, ct);
|
||||
|
||||
return Results.Ok(matches);
|
||||
})
|
||||
.WithName("GetSbomAffectedAdvisories")
|
||||
.Produces<IReadOnlyList<SbomCanonicalMatch>>(200);
|
||||
|
||||
// POST /api/v1/learn/runtime
|
||||
app.MapPost("/api/v1/learn/runtime", async (
|
||||
LearnRuntimeRequest request,
|
||||
ISbomLearningService learningService,
|
||||
ClaimsPrincipal user,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var tenantId = user.GetTenantId();
|
||||
var result = await learningService.LearnRuntimeAsync(
|
||||
tenantId, request.ArtifactId, request.Signals, ct);
|
||||
return Results.Ok(result);
|
||||
})
|
||||
.WithName("LearnRuntime")
|
||||
.WithSummary("Learn from runtime signals");
|
||||
|
||||
public sealed record LearnSbomRequest
|
||||
{
|
||||
public required string ArtifactId { get; init; }
|
||||
public required string SbomDigest { get; init; }
|
||||
}
|
||||
|
||||
public sealed record LearnRuntimeRequest
|
||||
{
|
||||
public required string ArtifactId { get; init; }
|
||||
public required IReadOnlyList<RuntimeSignal> Signals { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with Scanner Events
|
||||
|
||||
```csharp
|
||||
public sealed class ScanCompletedEventHandler : IEventHandler<ScanCompleted>
|
||||
{
|
||||
private readonly ISbomLearningService _learningService;
|
||||
|
||||
public async Task HandleAsync(ScanCompleted @event, CancellationToken ct)
|
||||
{
|
||||
// Auto-learn when a scan completes
|
||||
await _learningService.LearnAsync(
|
||||
@event.TenantId,
|
||||
@event.ImageDigest,
|
||||
@event.SbomDigest,
|
||||
ct);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| `concelier_sbom_learned_total` | Counter | format | SBOMs processed |
|
||||
| `concelier_sbom_components_total` | Counter | - | Components extracted |
|
||||
| `concelier_sbom_matches_total` | Counter | - | Advisory matches found |
|
||||
| `concelier_sbom_learning_duration_seconds` | Histogram | - | Learning operation time |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
220
docs/implplan/SPRINT_8200_0014_0001_DB_sync_ledger_schema.md
Normal file
220
docs/implplan/SPRINT_8200_0014_0001_DB_sync_ledger_schema.md
Normal file
@@ -0,0 +1,220 @@
|
||||
# Sprint 8200.0014.0001 - Sync Ledger Schema
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement the **sync_ledger** database schema for federation cursor tracking. This sprint delivers:
|
||||
|
||||
1. **sync_ledger table**: Track site_id, cursor position, bundle hashes
|
||||
2. **site_policy table**: Per-site allow/deny lists and size budgets
|
||||
3. **Migration scripts**: Create tables with indexes
|
||||
4. **Repository layer**: CRUD operations for ledger entries
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres/`
|
||||
|
||||
**Evidence:** Sites can track sync cursors; duplicate bundle import is rejected.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0002 (canonical schema)
|
||||
- **Blocks:** SPRINT_8200_0014_0002 (export), SPRINT_8200_0014_0003 (import)
|
||||
- **Safe to run in parallel with:** Phase B sprints
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Schema Design** | | | | | |
|
||||
| 0 | SYNC-8200-000 | TODO | Canonical schema | Platform Guild | Design `sync_ledger` table with cursor semantics |
|
||||
| 1 | SYNC-8200-001 | TODO | Task 0 | Platform Guild | Design `site_policy` table for federation governance |
|
||||
| 2 | SYNC-8200-002 | TODO | Task 1 | Platform Guild | Create migration `20250401000001_CreateSyncLedger.sql` |
|
||||
| 3 | SYNC-8200-003 | TODO | Task 2 | QA Guild | Validate migration (up/down/up) |
|
||||
| **Wave 1: Entity & Repository** | | | | | |
|
||||
| 4 | SYNC-8200-004 | TODO | Task 3 | Concelier Guild | Create `SyncLedgerEntity` record |
|
||||
| 5 | SYNC-8200-005 | TODO | Task 4 | Concelier Guild | Create `SitePolicyEntity` record |
|
||||
| 6 | SYNC-8200-006 | TODO | Task 5 | Concelier Guild | Define `ISyncLedgerRepository` interface |
|
||||
| 7 | SYNC-8200-007 | TODO | Task 6 | Concelier Guild | Implement `PostgresSyncLedgerRepository` |
|
||||
| 8 | SYNC-8200-008 | TODO | Task 7 | QA Guild | Unit tests for repository operations |
|
||||
| **Wave 2: Cursor Management** | | | | | |
|
||||
| 9 | SYNC-8200-009 | TODO | Task 8 | Concelier Guild | Implement `GetLatestCursorAsync(siteId)` |
|
||||
| 10 | SYNC-8200-010 | TODO | Task 9 | Concelier Guild | Implement `AdvanceCursorAsync(siteId, newCursor, bundleHash)` |
|
||||
| 11 | SYNC-8200-011 | TODO | Task 10 | Concelier Guild | Implement cursor conflict detection (out-of-order import) |
|
||||
| 12 | SYNC-8200-012 | TODO | Task 11 | QA Guild | Test cursor advancement and conflict handling |
|
||||
| **Wave 3: Site Policy** | | | | | |
|
||||
| 13 | SYNC-8200-013 | TODO | Task 8 | Concelier Guild | Implement `GetSitePolicyAsync(siteId)` |
|
||||
| 14 | SYNC-8200-014 | TODO | Task 13 | Concelier Guild | Implement source allow/deny list enforcement |
|
||||
| 15 | SYNC-8200-015 | TODO | Task 14 | Concelier Guild | Implement size budget tracking |
|
||||
| 16 | SYNC-8200-016 | TODO | Task 15 | QA Guild | Test policy enforcement |
|
||||
| 17 | SYNC-8200-017 | TODO | Task 16 | Docs Guild | Document sync_ledger schema and usage |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
-- Migration: 20250401000001_CreateSyncLedger.sql
|
||||
|
||||
-- Track federation sync state per remote site
|
||||
CREATE TABLE vuln.sync_ledger (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
site_id TEXT NOT NULL, -- Remote site identifier (e.g., "site-us-west", "airgap-dc2")
|
||||
cursor TEXT NOT NULL, -- Opaque cursor (usually ISO8601 timestamp or sequence)
|
||||
bundle_hash TEXT NOT NULL, -- SHA256 of imported bundle
|
||||
items_count INT NOT NULL DEFAULT 0, -- Number of items in bundle
|
||||
signed_at TIMESTAMPTZ NOT NULL, -- When bundle was signed by remote
|
||||
imported_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT uq_sync_ledger_site_cursor UNIQUE (site_id, cursor),
|
||||
CONSTRAINT uq_sync_ledger_bundle UNIQUE (bundle_hash)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sync_ledger_site ON vuln.sync_ledger(site_id);
|
||||
CREATE INDEX idx_sync_ledger_site_time ON vuln.sync_ledger(site_id, signed_at DESC);
|
||||
|
||||
COMMENT ON TABLE vuln.sync_ledger IS 'Federation sync cursor tracking per remote site';
|
||||
COMMENT ON COLUMN vuln.sync_ledger.cursor IS 'Position marker for incremental sync (monotonically increasing)';
|
||||
|
||||
-- Site federation policies
|
||||
CREATE TABLE vuln.site_policy (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
site_id TEXT NOT NULL UNIQUE,
|
||||
display_name TEXT,
|
||||
allowed_sources TEXT[] DEFAULT '{}', -- Empty = allow all
|
||||
denied_sources TEXT[] DEFAULT '{}',
|
||||
max_bundle_size_mb INT DEFAULT 100,
|
||||
max_items_per_bundle INT DEFAULT 10000,
|
||||
require_signature BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
allowed_signers TEXT[] DEFAULT '{}', -- Key IDs or issuers
|
||||
enabled BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_site_policy_enabled ON vuln.site_policy(enabled) WHERE enabled = TRUE;
|
||||
|
||||
COMMENT ON TABLE vuln.site_policy IS 'Per-site federation governance policies';
|
||||
|
||||
-- Trigger for updated_at
|
||||
CREATE TRIGGER trg_site_policy_updated
|
||||
BEFORE UPDATE ON vuln.site_policy
|
||||
FOR EACH ROW EXECUTE FUNCTION vuln.update_timestamp();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entity Models
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Models;
|
||||
|
||||
public sealed record SyncLedgerEntity
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public required string SiteId { get; init; }
|
||||
public required string Cursor { get; init; }
|
||||
public required string BundleHash { get; init; }
|
||||
public int ItemsCount { get; init; }
|
||||
public DateTimeOffset SignedAt { get; init; }
|
||||
public DateTimeOffset ImportedAt { get; init; }
|
||||
}
|
||||
|
||||
public sealed record SitePolicyEntity
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public required string SiteId { get; init; }
|
||||
public string? DisplayName { get; init; }
|
||||
public string[] AllowedSources { get; init; } = [];
|
||||
public string[] DeniedSources { get; init; } = [];
|
||||
public int MaxBundleSizeMb { get; init; } = 100;
|
||||
public int MaxItemsPerBundle { get; init; } = 10000;
|
||||
public bool RequireSignature { get; init; } = true;
|
||||
public string[] AllowedSigners { get; init; } = [];
|
||||
public bool Enabled { get; init; } = true;
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
public DateTimeOffset UpdatedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repository Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Storage.Sync;
|
||||
|
||||
public interface ISyncLedgerRepository
|
||||
{
|
||||
// Ledger operations
|
||||
Task<SyncLedgerEntity?> GetLatestAsync(string siteId, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<SyncLedgerEntity>> GetHistoryAsync(string siteId, int limit = 10, CancellationToken ct = default);
|
||||
Task<SyncLedgerEntity?> GetByBundleHashAsync(string bundleHash, CancellationToken ct = default);
|
||||
Task<Guid> InsertAsync(SyncLedgerEntity entry, CancellationToken ct = default);
|
||||
|
||||
// Cursor operations
|
||||
Task<string?> GetCursorAsync(string siteId, CancellationToken ct = default);
|
||||
Task AdvanceCursorAsync(string siteId, string newCursor, string bundleHash, int itemsCount, DateTimeOffset signedAt, CancellationToken ct = default);
|
||||
|
||||
// Site policy operations
|
||||
Task<SitePolicyEntity?> GetPolicyAsync(string siteId, CancellationToken ct = default);
|
||||
Task UpsertPolicyAsync(SitePolicyEntity policy, CancellationToken ct = default);
|
||||
Task<IReadOnlyList<SitePolicyEntity>> GetAllPoliciesAsync(bool enabledOnly = true, CancellationToken ct = default);
|
||||
|
||||
// Statistics
|
||||
Task<SyncStatistics> GetStatisticsAsync(CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed record SyncStatistics
|
||||
{
|
||||
public int TotalSites { get; init; }
|
||||
public int EnabledSites { get; init; }
|
||||
public long TotalBundlesImported { get; init; }
|
||||
public long TotalItemsImported { get; init; }
|
||||
public DateTimeOffset? LastImportAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cursor Semantics
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Cursor format: ISO8601 timestamp with sequence suffix.
|
||||
/// Example: "2025-01-15T10:30:00.000Z#0042"
|
||||
/// </summary>
|
||||
public static class CursorFormat
|
||||
{
|
||||
public static string Create(DateTimeOffset timestamp, int sequence = 0)
|
||||
{
|
||||
return $"{timestamp:O}#{sequence:D4}";
|
||||
}
|
||||
|
||||
public static (DateTimeOffset Timestamp, int Sequence) Parse(string cursor)
|
||||
{
|
||||
var parts = cursor.Split('#');
|
||||
var timestamp = DateTimeOffset.Parse(parts[0]);
|
||||
var sequence = parts.Length > 1 ? int.Parse(parts[1]) : 0;
|
||||
return (timestamp, sequence);
|
||||
}
|
||||
|
||||
public static bool IsAfter(string cursor1, string cursor2)
|
||||
{
|
||||
var (ts1, seq1) = Parse(cursor1);
|
||||
var (ts2, seq2) = Parse(cursor2);
|
||||
|
||||
if (ts1 != ts2) return ts1 > ts2;
|
||||
return seq1 > seq2;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,387 @@
|
||||
# Sprint 8200.0014.0002 - Delta Bundle Export
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **cursor-based delta bundle export** for federation sync. This sprint delivers:
|
||||
|
||||
1. **Bundle Format**: ZST-compressed NDJSON with manifest and DSSE signature
|
||||
2. **Delta Export**: Only canonicals changed since cursor position
|
||||
3. **Export Endpoint**: `GET /api/v1/federation/export?since_cursor={cursor}`
|
||||
4. **CLI Command**: `feedser bundle export` for air-gap workflows
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Federation/` (new)
|
||||
|
||||
**Evidence:** Export produces deterministic bundles; importing same bundle twice yields identical state.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0014_0001 (sync_ledger), SPRINT_8200_0012_0003 (canonical service)
|
||||
- **Blocks:** SPRINT_8200_0014_0003 (import)
|
||||
- **Safe to run in parallel with:** Nothing
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Project Setup** | | | | | |
|
||||
| 0 | EXPORT-8200-000 | TODO | Sync ledger | Concelier Guild | Create `StellaOps.Concelier.Federation` project |
|
||||
| 1 | EXPORT-8200-001 | TODO | Task 0 | Concelier Guild | Add ZstdSharp dependency for compression |
|
||||
| 2 | EXPORT-8200-002 | TODO | Task 1 | Concelier Guild | Define `FederationBundle` record with manifest structure |
|
||||
| **Wave 1: Bundle Format** | | | | | |
|
||||
| 3 | EXPORT-8200-003 | TODO | Task 2 | Concelier Guild | Define bundle manifest schema (version, site_id, cursor, items) |
|
||||
| 4 | EXPORT-8200-004 | TODO | Task 3 | Concelier Guild | Implement `BundleManifestWriter` |
|
||||
| 5 | EXPORT-8200-005 | TODO | Task 4 | Concelier Guild | Implement canonical advisory NDJSON serialization |
|
||||
| 6 | EXPORT-8200-006 | TODO | Task 5 | Concelier Guild | Implement source edge NDJSON serialization |
|
||||
| 7 | EXPORT-8200-007 | TODO | Task 6 | Concelier Guild | Implement ZST compression with configurable level |
|
||||
| 8 | EXPORT-8200-008 | TODO | Task 7 | QA Guild | Unit tests for serialization and compression |
|
||||
| **Wave 2: Delta Query** | | | | | |
|
||||
| 9 | EXPORT-8200-009 | TODO | Task 8 | Concelier Guild | Implement `GetChangedSinceAsync(cursor)` query |
|
||||
| 10 | EXPORT-8200-010 | TODO | Task 9 | Concelier Guild | Include source edges for changed canonicals |
|
||||
| 11 | EXPORT-8200-011 | TODO | Task 10 | Concelier Guild | Handle deleted/withdrawn advisories in delta |
|
||||
| 12 | EXPORT-8200-012 | TODO | Task 11 | Concelier Guild | Implement pagination for large deltas |
|
||||
| 13 | EXPORT-8200-013 | TODO | Task 12 | QA Guild | Test delta correctness across various change patterns |
|
||||
| **Wave 3: Export Service** | | | | | |
|
||||
| 14 | EXPORT-8200-014 | TODO | Task 13 | Concelier Guild | Define `IBundleExportService` interface |
|
||||
| 15 | EXPORT-8200-015 | TODO | Task 14 | Concelier Guild | Implement `ExportAsync(sinceCursor)` method |
|
||||
| 16 | EXPORT-8200-016 | TODO | Task 15 | Concelier Guild | Compute bundle hash (SHA256 of compressed content) |
|
||||
| 17 | EXPORT-8200-017 | TODO | Task 16 | Concelier Guild | Generate new cursor for export |
|
||||
| 18 | EXPORT-8200-018 | TODO | Task 17 | QA Guild | Test export determinism (same inputs = same hash) |
|
||||
| **Wave 4: DSSE Signing** | | | | | |
|
||||
| 19 | EXPORT-8200-019 | TODO | Task 18 | Concelier Guild | Integrate with Signer service for bundle signing |
|
||||
| 20 | EXPORT-8200-020 | TODO | Task 19 | Concelier Guild | Create DSSE envelope over bundle hash |
|
||||
| 21 | EXPORT-8200-021 | TODO | Task 20 | Concelier Guild | Include certificate chain in manifest |
|
||||
| 22 | EXPORT-8200-022 | TODO | Task 21 | QA Guild | Test signature verification |
|
||||
| **Wave 5: API & CLI** | | | | | |
|
||||
| 23 | EXPORT-8200-023 | TODO | Task 22 | Concelier Guild | Create `GET /api/v1/federation/export` endpoint |
|
||||
| 24 | EXPORT-8200-024 | TODO | Task 23 | Concelier Guild | Support streaming response for large bundles |
|
||||
| 25 | EXPORT-8200-025 | TODO | Task 24 | Concelier Guild | Add `feedser bundle export` CLI command |
|
||||
| 26 | EXPORT-8200-026 | TODO | Task 25 | Concelier Guild | Support output to file or stdout |
|
||||
| 27 | EXPORT-8200-027 | TODO | Task 26 | QA Guild | End-to-end test: export bundle, verify contents |
|
||||
| 28 | EXPORT-8200-028 | TODO | Task 27 | Docs Guild | Document bundle format and export API |
|
||||
|
||||
---
|
||||
|
||||
## Bundle Format
|
||||
|
||||
```
|
||||
feedser-bundle-v1.zst
|
||||
├── MANIFEST.json # Bundle metadata
|
||||
├── canonicals.ndjson # Canonical advisories (one per line)
|
||||
├── edges.ndjson # Source edges (one per line)
|
||||
├── deletions.ndjson # Withdrawn/deleted canonical IDs
|
||||
└── SIGNATURE.json # DSSE envelope
|
||||
```
|
||||
|
||||
### Manifest Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "feedser-bundle/1.0",
|
||||
"site_id": "site-us-west-1",
|
||||
"export_cursor": "2025-01-15T10:30:00.000Z#0042",
|
||||
"since_cursor": "2025-01-14T00:00:00.000Z#0000",
|
||||
"exported_at": "2025-01-15T10:30:15.123Z",
|
||||
"counts": {
|
||||
"canonicals": 1234,
|
||||
"edges": 3456,
|
||||
"deletions": 12
|
||||
},
|
||||
"bundle_hash": "sha256:a1b2c3d4...",
|
||||
"signature": {
|
||||
"key_id": "sha256:xyz...",
|
||||
"algorithm": "ES256",
|
||||
"issuer": "https://authority.stellaops.example.com"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Canonical NDJSON Line
|
||||
|
||||
```json
|
||||
{"id":"uuid","cve":"CVE-2024-1234","affects_key":"pkg:npm/express@4.0.0","merge_hash":"a1b2c3...","status":"active","severity":"high","title":"...","source_edges":["edge-uuid-1","edge-uuid-2"]}
|
||||
```
|
||||
|
||||
### Source Edge NDJSON Line
|
||||
|
||||
```json
|
||||
{"id":"uuid","canonical_id":"uuid","source":"nvd","source_advisory_id":"CVE-2024-1234","vendor_status":"affected","dsse_envelope":{...}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service Interface
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Federation;
|
||||
|
||||
public interface IBundleExportService
|
||||
{
|
||||
/// <summary>Export delta bundle since cursor.</summary>
|
||||
Task<BundleExportResult> ExportAsync(
|
||||
string? sinceCursor = null,
|
||||
BundleExportOptions? options = null,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Export delta bundle to stream.</summary>
|
||||
Task ExportToStreamAsync(
|
||||
Stream output,
|
||||
string? sinceCursor = null,
|
||||
BundleExportOptions? options = null,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Get export statistics without creating bundle.</summary>
|
||||
Task<BundleExportPreview> PreviewAsync(
|
||||
string? sinceCursor = null,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed record BundleExportOptions
|
||||
{
|
||||
public int CompressionLevel { get; init; } = 3; // ZST 1-19
|
||||
public bool Sign { get; init; } = true;
|
||||
public int MaxItems { get; init; } = 10_000;
|
||||
public string[]? IncludeSources { get; init; }
|
||||
public string[]? ExcludeSources { get; init; }
|
||||
}
|
||||
|
||||
public sealed record BundleExportResult
|
||||
{
|
||||
public required string BundleHash { get; init; }
|
||||
public required string ExportCursor { get; init; }
|
||||
public string? SinceCursor { get; init; }
|
||||
public required BundleCounts Counts { get; init; }
|
||||
public long CompressedSizeBytes { get; init; }
|
||||
public DsseEnvelope? Signature { get; init; }
|
||||
public TimeSpan Duration { get; init; }
|
||||
}
|
||||
|
||||
public sealed record BundleCounts
|
||||
{
|
||||
public int Canonicals { get; init; }
|
||||
public int Edges { get; init; }
|
||||
public int Deletions { get; init; }
|
||||
public int Total => Canonicals + Edges + Deletions;
|
||||
}
|
||||
|
||||
public sealed record BundleExportPreview
|
||||
{
|
||||
public int EstimatedCanonicals { get; init; }
|
||||
public int EstimatedEdges { get; init; }
|
||||
public int EstimatedDeletions { get; init; }
|
||||
public long EstimatedSizeBytes { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Export Implementation
|
||||
|
||||
```csharp
|
||||
public async Task<BundleExportResult> ExportAsync(
|
||||
string? sinceCursor,
|
||||
BundleExportOptions? options,
|
||||
CancellationToken ct)
|
||||
{
|
||||
options ??= new BundleExportOptions();
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
|
||||
// 1. Query changed canonicals since cursor
|
||||
var changes = await _repository.GetChangedSinceAsync(sinceCursor, options.MaxItems, ct);
|
||||
|
||||
// 2. Create temporary file for bundle
|
||||
var tempPath = Path.GetTempFileName();
|
||||
try
|
||||
{
|
||||
await using var fileStream = File.Create(tempPath);
|
||||
await using var zstStream = new ZstdSharp.CompressionStream(
|
||||
fileStream, options.CompressionLevel);
|
||||
await using var tarWriter = new TarWriter(zstStream);
|
||||
|
||||
// 3. Write manifest placeholder (update later)
|
||||
var manifestPlaceholder = new byte[4096];
|
||||
await WriteEntryAsync(tarWriter, "MANIFEST.json", manifestPlaceholder, ct);
|
||||
|
||||
// 4. Write canonicals NDJSON
|
||||
var canonicalCount = 0;
|
||||
await using var canonicalStream = new MemoryStream();
|
||||
await foreach (var canonical in changes.Canonicals.WithCancellation(ct))
|
||||
{
|
||||
await WriteNdjsonLineAsync(canonicalStream, canonical, ct);
|
||||
canonicalCount++;
|
||||
}
|
||||
canonicalStream.Position = 0;
|
||||
await WriteEntryAsync(tarWriter, "canonicals.ndjson", canonicalStream, ct);
|
||||
|
||||
// 5. Write edges NDJSON
|
||||
var edgeCount = 0;
|
||||
await using var edgeStream = new MemoryStream();
|
||||
await foreach (var edge in changes.Edges.WithCancellation(ct))
|
||||
{
|
||||
await WriteNdjsonLineAsync(edgeStream, edge, ct);
|
||||
edgeCount++;
|
||||
}
|
||||
edgeStream.Position = 0;
|
||||
await WriteEntryAsync(tarWriter, "edges.ndjson", edgeStream, ct);
|
||||
|
||||
// 6. Write deletions NDJSON
|
||||
var deletionCount = 0;
|
||||
await using var deletionStream = new MemoryStream();
|
||||
await foreach (var deletion in changes.Deletions.WithCancellation(ct))
|
||||
{
|
||||
await WriteNdjsonLineAsync(deletionStream, deletion, ct);
|
||||
deletionCount++;
|
||||
}
|
||||
deletionStream.Position = 0;
|
||||
await WriteEntryAsync(tarWriter, "deletions.ndjson", deletionStream, ct);
|
||||
}
|
||||
|
||||
// 7. Compute bundle hash
|
||||
var bundleHash = await ComputeHashAsync(tempPath, ct);
|
||||
|
||||
// 8. Sign bundle if requested
|
||||
DsseEnvelope? signature = null;
|
||||
if (options.Sign)
|
||||
{
|
||||
signature = await _signerClient.SignBundleAsync(bundleHash, ct);
|
||||
}
|
||||
|
||||
// 9. Generate new cursor
|
||||
var exportCursor = CursorFormat.Create(DateTimeOffset.UtcNow);
|
||||
|
||||
// 10. Update manifest and rewrite
|
||||
var manifest = new BundleManifest
|
||||
{
|
||||
Version = "feedser-bundle/1.0",
|
||||
SiteId = _siteId,
|
||||
ExportCursor = exportCursor,
|
||||
SinceCursor = sinceCursor,
|
||||
ExportedAt = DateTimeOffset.UtcNow,
|
||||
Counts = new BundleCounts
|
||||
{
|
||||
Canonicals = canonicalCount,
|
||||
Edges = edgeCount,
|
||||
Deletions = deletionCount
|
||||
},
|
||||
BundleHash = bundleHash
|
||||
};
|
||||
|
||||
// ... finalize bundle with updated manifest ...
|
||||
|
||||
return new BundleExportResult
|
||||
{
|
||||
BundleHash = bundleHash,
|
||||
ExportCursor = exportCursor,
|
||||
SinceCursor = sinceCursor,
|
||||
Counts = manifest.Counts,
|
||||
CompressedSizeBytes = new FileInfo(tempPath).Length,
|
||||
Signature = signature,
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoint
|
||||
|
||||
```csharp
|
||||
// GET /api/v1/federation/export
|
||||
app.MapGet("/api/v1/federation/export", async (
|
||||
[FromQuery] string? since_cursor,
|
||||
[FromQuery] bool sign = true,
|
||||
[FromQuery] int max_items = 10000,
|
||||
IBundleExportService exportService,
|
||||
HttpResponse response,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var options = new BundleExportOptions
|
||||
{
|
||||
Sign = sign,
|
||||
MaxItems = max_items
|
||||
};
|
||||
|
||||
response.ContentType = "application/zstd";
|
||||
response.Headers.ContentDisposition = $"attachment; filename=\"feedser-bundle-{DateTime.UtcNow:yyyyMMdd-HHmmss}.zst\"";
|
||||
|
||||
await exportService.ExportToStreamAsync(response.Body, since_cursor, options, ct);
|
||||
})
|
||||
.WithName("ExportBundle")
|
||||
.WithSummary("Export delta bundle for federation sync")
|
||||
.Produces(200, contentType: "application/zstd");
|
||||
|
||||
// GET /api/v1/federation/export/preview
|
||||
app.MapGet("/api/v1/federation/export/preview", async (
|
||||
[FromQuery] string? since_cursor,
|
||||
IBundleExportService exportService,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var preview = await exportService.PreviewAsync(since_cursor, ct);
|
||||
return Results.Ok(preview);
|
||||
})
|
||||
.WithName("PreviewExport")
|
||||
.Produces<BundleExportPreview>(200);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Command
|
||||
|
||||
```csharp
|
||||
// feedser bundle export --since-cursor <cursor> --output <path> [--sign] [--compress-level 3]
|
||||
[Command("bundle export", Description = "Export federation bundle")]
|
||||
public class BundleExportCommand : ICommand
|
||||
{
|
||||
[Option('c', "since-cursor", Description = "Export changes since cursor")]
|
||||
public string? SinceCursor { get; set; }
|
||||
|
||||
[Option('o', "output", Description = "Output file path (default: stdout)")]
|
||||
public string? Output { get; set; }
|
||||
|
||||
[Option('s', "sign", Description = "Sign bundle with Authority key")]
|
||||
public bool Sign { get; set; } = true;
|
||||
|
||||
[Option('l', "compress-level", Description = "ZST compression level (1-19)")]
|
||||
public int CompressLevel { get; set; } = 3;
|
||||
|
||||
public async ValueTask ExecuteAsync(IConsole console)
|
||||
{
|
||||
var options = new BundleExportOptions
|
||||
{
|
||||
Sign = Sign,
|
||||
CompressionLevel = CompressLevel
|
||||
};
|
||||
|
||||
Stream output = string.IsNullOrEmpty(Output)
|
||||
? Console.OpenStandardOutput()
|
||||
: File.Create(Output);
|
||||
|
||||
try
|
||||
{
|
||||
await _exportService.ExportToStreamAsync(output, SinceCursor, options);
|
||||
if (!string.IsNullOrEmpty(Output))
|
||||
{
|
||||
console.Output.WriteLine($"Bundle exported to {Output}");
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (!string.IsNullOrEmpty(Output))
|
||||
{
|
||||
await output.DisposeAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,456 @@
|
||||
# Sprint 8200.0014.0003 - Bundle Import & Merge
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **bundle import with verification and merge** for federation sync. This sprint delivers:
|
||||
|
||||
1. **Bundle Verification**: Validate signature, hash, format, and policy compliance
|
||||
2. **Merge Logic**: Apply canonicals/edges with conflict detection
|
||||
3. **Import Endpoint**: `POST /api/v1/federation/import`
|
||||
4. **CLI Command**: `feedser bundle import` for air-gap workflows
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Federation/`
|
||||
|
||||
**Evidence:** Importing bundle from Site A to Site B produces identical canonical state; conflicts are logged and handled.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0014_0001 (sync_ledger), SPRINT_8200_0014_0002 (export)
|
||||
- **Blocks:** Nothing (completes Phase C)
|
||||
- **Safe to run in parallel with:** Nothing
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Bundle Parsing** | | | | | |
|
||||
| 0 | IMPORT-8200-000 | TODO | Export format | Concelier Guild | Implement `BundleReader` for ZST decompression |
|
||||
| 1 | IMPORT-8200-001 | TODO | Task 0 | Concelier Guild | Parse and validate MANIFEST.json |
|
||||
| 2 | IMPORT-8200-002 | TODO | Task 1 | Concelier Guild | Stream-parse canonicals.ndjson |
|
||||
| 3 | IMPORT-8200-003 | TODO | Task 2 | Concelier Guild | Stream-parse edges.ndjson |
|
||||
| 4 | IMPORT-8200-004 | TODO | Task 3 | Concelier Guild | Parse deletions.ndjson |
|
||||
| 5 | IMPORT-8200-005 | TODO | Task 4 | QA Guild | Unit tests for bundle parsing |
|
||||
| **Wave 1: Verification** | | | | | |
|
||||
| 6 | IMPORT-8200-006 | TODO | Task 5 | Concelier Guild | Define `IBundleVerifier` interface |
|
||||
| 7 | IMPORT-8200-007 | TODO | Task 6 | Concelier Guild | Implement hash verification (bundle hash matches content) |
|
||||
| 8 | IMPORT-8200-008 | TODO | Task 7 | Concelier Guild | Implement DSSE signature verification |
|
||||
| 9 | IMPORT-8200-009 | TODO | Task 8 | Concelier Guild | Implement site policy enforcement (allowed sources, size limits) |
|
||||
| 10 | IMPORT-8200-010 | TODO | Task 9 | Concelier Guild | Implement cursor validation (must be after current cursor) |
|
||||
| 11 | IMPORT-8200-011 | TODO | Task 10 | QA Guild | Test verification failures (bad hash, invalid sig, policy violation) |
|
||||
| **Wave 2: Merge Logic** | | | | | |
|
||||
| 12 | IMPORT-8200-012 | TODO | Task 11 | Concelier Guild | Define `IBundleMergeService` interface |
|
||||
| 13 | IMPORT-8200-013 | TODO | Task 12 | Concelier Guild | Implement canonical upsert (ON CONFLICT by merge_hash) |
|
||||
| 14 | IMPORT-8200-014 | TODO | Task 13 | Concelier Guild | Implement source edge merge (add if not exists) |
|
||||
| 15 | IMPORT-8200-015 | TODO | Task 14 | Concelier Guild | Implement deletion handling (mark as withdrawn) |
|
||||
| 16 | IMPORT-8200-016 | TODO | Task 15 | Concelier Guild | Implement conflict detection and logging |
|
||||
| 17 | IMPORT-8200-017 | TODO | Task 16 | Concelier Guild | Implement transactional import (all or nothing) |
|
||||
| 18 | IMPORT-8200-018 | TODO | Task 17 | QA Guild | Test merge scenarios (new, update, conflict, deletion) |
|
||||
| **Wave 3: Import Service** | | | | | |
|
||||
| 19 | IMPORT-8200-019 | TODO | Task 18 | Concelier Guild | Define `IBundleImportService` interface |
|
||||
| 20 | IMPORT-8200-020 | TODO | Task 19 | Concelier Guild | Implement `ImportAsync()` orchestration |
|
||||
| 21 | IMPORT-8200-021 | TODO | Task 20 | Concelier Guild | Update sync_ledger with new cursor |
|
||||
| 22 | IMPORT-8200-022 | TODO | Task 21 | Concelier Guild | Emit import events for downstream consumers |
|
||||
| 23 | IMPORT-8200-023 | TODO | Task 22 | Concelier Guild | Update Valkey cache for imported canonicals |
|
||||
| 24 | IMPORT-8200-024 | TODO | Task 23 | QA Guild | Integration test: export from A, import to B, verify state |
|
||||
| **Wave 4: API & CLI** | | | | | |
|
||||
| 25 | IMPORT-8200-025 | TODO | Task 24 | Concelier Guild | Create `POST /api/v1/federation/import` endpoint |
|
||||
| 26 | IMPORT-8200-026 | TODO | Task 25 | Concelier Guild | Support streaming upload for large bundles |
|
||||
| 27 | IMPORT-8200-027 | TODO | Task 26 | Concelier Guild | Add `feedser bundle import` CLI command |
|
||||
| 28 | IMPORT-8200-028 | TODO | Task 27 | Concelier Guild | Support input from file or stdin |
|
||||
| 29 | IMPORT-8200-029 | TODO | Task 28 | QA Guild | End-to-end air-gap test (export to file, transfer, import) |
|
||||
| **Wave 5: Site Management** | | | | | |
|
||||
| 30 | IMPORT-8200-030 | TODO | Task 29 | Concelier Guild | Create `GET /api/v1/federation/sites` endpoint |
|
||||
| 31 | IMPORT-8200-031 | TODO | Task 30 | Concelier Guild | Create `PUT /api/v1/federation/sites/{id}/policy` endpoint |
|
||||
| 32 | IMPORT-8200-032 | TODO | Task 31 | Concelier Guild | Add `feedser sites list` CLI command |
|
||||
| 33 | IMPORT-8200-033 | TODO | Task 32 | QA Guild | Test multi-site federation scenario |
|
||||
| 34 | IMPORT-8200-034 | TODO | Task 33 | Docs Guild | Document federation setup and operations |
|
||||
|
||||
---
|
||||
|
||||
## Service Interfaces
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Federation;
|
||||
|
||||
public interface IBundleImportService
|
||||
{
|
||||
/// <summary>Import bundle from stream.</summary>
|
||||
Task<BundleImportResult> ImportAsync(
|
||||
Stream bundleStream,
|
||||
BundleImportOptions? options = null,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Import bundle from file path.</summary>
|
||||
Task<BundleImportResult> ImportFromFileAsync(
|
||||
string filePath,
|
||||
BundleImportOptions? options = null,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Validate bundle without importing.</summary>
|
||||
Task<BundleValidationResult> ValidateAsync(
|
||||
Stream bundleStream,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed record BundleImportOptions
|
||||
{
|
||||
public bool SkipSignatureVerification { get; init; } = false;
|
||||
public bool DryRun { get; init; } = false;
|
||||
public ConflictResolution OnConflict { get; init; } = ConflictResolution.PreferRemote;
|
||||
}
|
||||
|
||||
public enum ConflictResolution
|
||||
{
|
||||
PreferRemote, // Remote wins (default for federation)
|
||||
PreferLocal, // Local wins
|
||||
Fail // Abort import on conflict
|
||||
}
|
||||
|
||||
public sealed record BundleImportResult
|
||||
{
|
||||
public required string BundleHash { get; init; }
|
||||
public required string ImportedCursor { get; init; }
|
||||
public required ImportCounts Counts { get; init; }
|
||||
public IReadOnlyList<ImportConflict> Conflicts { get; init; } = [];
|
||||
public bool Success { get; init; }
|
||||
public string? FailureReason { get; init; }
|
||||
public TimeSpan Duration { get; init; }
|
||||
}
|
||||
|
||||
public sealed record ImportCounts
|
||||
{
|
||||
public int CanonicalCreated { get; init; }
|
||||
public int CanonicalUpdated { get; init; }
|
||||
public int CanonicalSkipped { get; init; }
|
||||
public int EdgesAdded { get; init; }
|
||||
public int DeletionsProcessed { get; init; }
|
||||
}
|
||||
|
||||
public sealed record ImportConflict
|
||||
{
|
||||
public required string MergeHash { get; init; }
|
||||
public required string Field { get; init; }
|
||||
public string? LocalValue { get; init; }
|
||||
public string? RemoteValue { get; init; }
|
||||
public required ConflictResolution Resolution { get; init; }
|
||||
}
|
||||
|
||||
public sealed record BundleValidationResult
|
||||
{
|
||||
public bool IsValid { get; init; }
|
||||
public IReadOnlyList<string> Errors { get; init; } = [];
|
||||
public IReadOnlyList<string> Warnings { get; init; } = [];
|
||||
public BundleManifest? Manifest { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Import Flow
|
||||
|
||||
```csharp
|
||||
public async Task<BundleImportResult> ImportAsync(
|
||||
Stream bundleStream,
|
||||
BundleImportOptions? options,
|
||||
CancellationToken ct)
|
||||
{
|
||||
options ??= new BundleImportOptions();
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
var conflicts = new List<ImportConflict>();
|
||||
|
||||
// 1. Parse and validate bundle
|
||||
using var bundle = await _bundleReader.ReadAsync(bundleStream, ct);
|
||||
var validation = await _verifier.VerifyAsync(bundle, options.SkipSignatureVerification, ct);
|
||||
|
||||
if (!validation.IsValid)
|
||||
{
|
||||
return new BundleImportResult
|
||||
{
|
||||
BundleHash = bundle.Manifest?.BundleHash ?? "unknown",
|
||||
ImportedCursor = "",
|
||||
Counts = new ImportCounts(),
|
||||
Success = false,
|
||||
FailureReason = string.Join("; ", validation.Errors),
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
|
||||
// 2. Check cursor (must be after current)
|
||||
var currentCursor = await _ledgerRepository.GetCursorAsync(bundle.Manifest.SiteId, ct);
|
||||
if (currentCursor != null && !CursorFormat.IsAfter(bundle.Manifest.ExportCursor, currentCursor))
|
||||
{
|
||||
return new BundleImportResult
|
||||
{
|
||||
BundleHash = bundle.Manifest.BundleHash,
|
||||
ImportedCursor = "",
|
||||
Counts = new ImportCounts(),
|
||||
Success = false,
|
||||
FailureReason = $"Bundle cursor {bundle.Manifest.ExportCursor} is not after current cursor {currentCursor}",
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
|
||||
// 3. Check for duplicate bundle
|
||||
var existingBundle = await _ledgerRepository.GetByBundleHashAsync(bundle.Manifest.BundleHash, ct);
|
||||
if (existingBundle != null)
|
||||
{
|
||||
return new BundleImportResult
|
||||
{
|
||||
BundleHash = bundle.Manifest.BundleHash,
|
||||
ImportedCursor = existingBundle.Cursor,
|
||||
Counts = new ImportCounts { CanonicalSkipped = bundle.Manifest.Counts.Canonicals },
|
||||
Success = true,
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
|
||||
if (options.DryRun)
|
||||
{
|
||||
return new BundleImportResult
|
||||
{
|
||||
BundleHash = bundle.Manifest.BundleHash,
|
||||
ImportedCursor = bundle.Manifest.ExportCursor,
|
||||
Counts = new ImportCounts
|
||||
{
|
||||
CanonicalCreated = bundle.Manifest.Counts.Canonicals,
|
||||
EdgesAdded = bundle.Manifest.Counts.Edges,
|
||||
DeletionsProcessed = bundle.Manifest.Counts.Deletions
|
||||
},
|
||||
Success = true,
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
|
||||
// 4. Begin transaction
|
||||
await using var transaction = await _dataSource.BeginTransactionAsync(ct);
|
||||
var counts = new ImportCounts();
|
||||
|
||||
try
|
||||
{
|
||||
// 5. Import canonicals
|
||||
await foreach (var canonical in bundle.StreamCanonicalsAsync(ct))
|
||||
{
|
||||
var result = await _mergeService.MergeCanonicalAsync(canonical, options.OnConflict, ct);
|
||||
counts = counts with
|
||||
{
|
||||
CanonicalCreated = counts.CanonicalCreated + (result.Action == MergeAction.Created ? 1 : 0),
|
||||
CanonicalUpdated = counts.CanonicalUpdated + (result.Action == MergeAction.Updated ? 1 : 0),
|
||||
CanonicalSkipped = counts.CanonicalSkipped + (result.Action == MergeAction.Skipped ? 1 : 0)
|
||||
};
|
||||
|
||||
if (result.Conflict != null)
|
||||
{
|
||||
conflicts.Add(result.Conflict);
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Import source edges
|
||||
await foreach (var edge in bundle.StreamEdgesAsync(ct))
|
||||
{
|
||||
var added = await _mergeService.MergeEdgeAsync(edge, ct);
|
||||
if (added)
|
||||
{
|
||||
counts = counts with { EdgesAdded = counts.EdgesAdded + 1 };
|
||||
}
|
||||
}
|
||||
|
||||
// 7. Process deletions
|
||||
await foreach (var deletion in bundle.StreamDeletionsAsync(ct))
|
||||
{
|
||||
await _canonicalRepository.UpdateStatusAsync(deletion.CanonicalId, "withdrawn", ct);
|
||||
counts = counts with { DeletionsProcessed = counts.DeletionsProcessed + 1 };
|
||||
}
|
||||
|
||||
// 8. Update sync ledger
|
||||
await _ledgerRepository.AdvanceCursorAsync(
|
||||
bundle.Manifest.SiteId,
|
||||
bundle.Manifest.ExportCursor,
|
||||
bundle.Manifest.BundleHash,
|
||||
bundle.Manifest.Counts.Total,
|
||||
bundle.Manifest.ExportedAt,
|
||||
ct);
|
||||
|
||||
// 9. Commit transaction
|
||||
await transaction.CommitAsync(ct);
|
||||
}
|
||||
catch
|
||||
{
|
||||
await transaction.RollbackAsync(ct);
|
||||
throw;
|
||||
}
|
||||
|
||||
// 10. Update cache
|
||||
await _cacheService.InvalidateManyAsync(
|
||||
bundle.StreamCanonicalsAsync(ct).Select(c => c.MergeHash),
|
||||
ct);
|
||||
|
||||
// 11. Emit event
|
||||
await _eventBus.PublishAsync(new BundleImported
|
||||
{
|
||||
SiteId = bundle.Manifest.SiteId,
|
||||
BundleHash = bundle.Manifest.BundleHash,
|
||||
Cursor = bundle.Manifest.ExportCursor,
|
||||
Counts = counts
|
||||
}, ct);
|
||||
|
||||
return new BundleImportResult
|
||||
{
|
||||
BundleHash = bundle.Manifest.BundleHash,
|
||||
ImportedCursor = bundle.Manifest.ExportCursor,
|
||||
Counts = counts,
|
||||
Conflicts = conflicts,
|
||||
Success = true,
|
||||
Duration = stopwatch.Elapsed
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoint
|
||||
|
||||
```csharp
|
||||
// POST /api/v1/federation/import
|
||||
app.MapPost("/api/v1/federation/import", async (
|
||||
HttpRequest request,
|
||||
[FromQuery] bool dry_run = false,
|
||||
[FromQuery] bool skip_signature = false,
|
||||
IBundleImportService importService,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var options = new BundleImportOptions
|
||||
{
|
||||
DryRun = dry_run,
|
||||
SkipSignatureVerification = skip_signature
|
||||
};
|
||||
|
||||
var result = await importService.ImportAsync(request.Body, options, ct);
|
||||
|
||||
return result.Success
|
||||
? Results.Ok(result)
|
||||
: Results.BadRequest(result);
|
||||
})
|
||||
.WithName("ImportBundle")
|
||||
.WithSummary("Import federation bundle")
|
||||
.Accepts<IFormFile>("application/zstd")
|
||||
.Produces<BundleImportResult>(200)
|
||||
.Produces<BundleImportResult>(400);
|
||||
|
||||
// GET /api/v1/federation/sites
|
||||
app.MapGet("/api/v1/federation/sites", async (
|
||||
ISyncLedgerRepository ledgerRepo,
|
||||
CancellationToken ct) =>
|
||||
{
|
||||
var policies = await ledgerRepo.GetAllPoliciesAsync(enabledOnly: false, ct);
|
||||
var sites = new List<FederationSiteInfo>();
|
||||
|
||||
foreach (var policy in policies)
|
||||
{
|
||||
var latest = await ledgerRepo.GetLatestAsync(policy.SiteId, ct);
|
||||
sites.Add(new FederationSiteInfo
|
||||
{
|
||||
SiteId = policy.SiteId,
|
||||
DisplayName = policy.DisplayName,
|
||||
Enabled = policy.Enabled,
|
||||
LastCursor = latest?.Cursor,
|
||||
LastSyncAt = latest?.ImportedAt,
|
||||
BundlesImported = await ledgerRepo.GetHistoryAsync(policy.SiteId, 1000, ct).CountAsync()
|
||||
});
|
||||
}
|
||||
|
||||
return Results.Ok(sites);
|
||||
})
|
||||
.WithName("ListFederationSites")
|
||||
.Produces<IReadOnlyList<FederationSiteInfo>>(200);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```csharp
|
||||
// feedser bundle import <file> [--dry-run] [--skip-signature]
|
||||
[Command("bundle import", Description = "Import federation bundle")]
|
||||
public class BundleImportCommand : ICommand
|
||||
{
|
||||
[Argument(0, Description = "Bundle file path (or - for stdin)")]
|
||||
public string Input { get; set; } = "-";
|
||||
|
||||
[Option('n', "dry-run", Description = "Validate without importing")]
|
||||
public bool DryRun { get; set; }
|
||||
|
||||
[Option("skip-signature", Description = "Skip signature verification (DANGEROUS)")]
|
||||
public bool SkipSignature { get; set; }
|
||||
|
||||
public async ValueTask ExecuteAsync(IConsole console)
|
||||
{
|
||||
var options = new BundleImportOptions
|
||||
{
|
||||
DryRun = DryRun,
|
||||
SkipSignatureVerification = SkipSignature
|
||||
};
|
||||
|
||||
Stream input = Input == "-"
|
||||
? Console.OpenStandardInput()
|
||||
: File.OpenRead(Input);
|
||||
|
||||
try
|
||||
{
|
||||
var result = await _importService.ImportAsync(input, options);
|
||||
|
||||
if (result.Success)
|
||||
{
|
||||
console.Output.WriteLine($"Import successful: {result.Counts.CanonicalCreated} created, {result.Counts.CanonicalUpdated} updated");
|
||||
console.Output.WriteLine($"New cursor: {result.ImportedCursor}");
|
||||
}
|
||||
else
|
||||
{
|
||||
console.Error.WriteLine($"Import failed: {result.FailureReason}");
|
||||
Environment.ExitCode = 1;
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (Input != "-")
|
||||
{
|
||||
await input.DisposeAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// feedser sites list
|
||||
[Command("sites list", Description = "List federation sites")]
|
||||
public class SitesListCommand : ICommand
|
||||
{
|
||||
public async ValueTask ExecuteAsync(IConsole console)
|
||||
{
|
||||
var sites = await _ledgerRepository.GetAllPoliciesAsync();
|
||||
|
||||
console.Output.WriteLine("SITE ID STATUS LAST SYNC CURSOR");
|
||||
console.Output.WriteLine("───────────────────────── ──────── ─────────────────── ──────────────────────────");
|
||||
|
||||
foreach (var site in sites)
|
||||
{
|
||||
var latest = await _ledgerRepository.GetLatestAsync(site.SiteId);
|
||||
var status = site.Enabled ? "enabled" : "disabled";
|
||||
var lastSync = latest?.ImportedAt.ToString("yyyy-MM-dd HH:mm") ?? "never";
|
||||
var cursor = latest?.Cursor ?? "-";
|
||||
|
||||
console.Output.WriteLine($"{site.SiteId,-26} {status,-8} {lastSync,-19} {cursor}");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
@@ -0,0 +1,451 @@
|
||||
# Sprint 8200.0015.0001 - Backport Integration
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Implement **backport-aware precision** by integrating `BackportProofService` into the canonical deduplication flow. This sprint delivers:
|
||||
|
||||
1. **provenance_scope table**: Track distro-specific backport versions and patch lineage
|
||||
2. **Proof Integration**: Wire BackportProofService evidence into merge decisions
|
||||
3. **Policy Lattice**: Configurable vendor vs distro precedence with backport awareness
|
||||
4. **Enhanced Dedup**: Same CVE with different backport status = different canonicals
|
||||
|
||||
**Working directory:** `src/Concelier/__Libraries/StellaOps.Concelier.Merge/`
|
||||
|
||||
**Evidence:** CVE-2024-1234 with Debian backport and RHEL backport produce correct distinct or merged canonicals based on evidence.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** SPRINT_8200_0012_0003 (canonical service), existing BackportProofService
|
||||
- **Blocks:** Nothing (completes Phase D)
|
||||
- **Safe to run in parallel with:** Phase C sprints
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/implplan/SPRINT_8200_0012_0000_FEEDSER_master_plan.md`
|
||||
- `src/Concelier/__Libraries/StellaOps.Concelier.ProofService/BackportProofService.cs`
|
||||
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owner | Task Definition |
|
||||
|---|---------|--------|----------------|-------|-----------------|
|
||||
| **Wave 0: Schema** | | | | | |
|
||||
| 0 | BACKPORT-8200-000 | TODO | Canonical service | Platform Guild | Create migration `20250501000001_CreateProvenanceScope.sql` |
|
||||
| 1 | BACKPORT-8200-001 | TODO | Task 0 | Concelier Guild | Create `ProvenanceScopeEntity` record |
|
||||
| 2 | BACKPORT-8200-002 | TODO | Task 1 | Concelier Guild | Define `IProvenanceScopeRepository` interface |
|
||||
| 3 | BACKPORT-8200-003 | TODO | Task 2 | Concelier Guild | Implement `PostgresProvenanceScopeRepository` |
|
||||
| 4 | BACKPORT-8200-004 | TODO | Task 3 | QA Guild | Unit tests for repository CRUD |
|
||||
| **Wave 1: Proof Service Integration** | | | | | |
|
||||
| 5 | BACKPORT-8200-005 | TODO | Task 4 | Concelier Guild | Define `IBackportEvidenceResolver` interface |
|
||||
| 6 | BACKPORT-8200-006 | TODO | Task 5 | Concelier Guild | Implement resolver calling BackportProofService |
|
||||
| 7 | BACKPORT-8200-007 | TODO | Task 6 | Concelier Guild | Extract patch lineage from proof evidence |
|
||||
| 8 | BACKPORT-8200-008 | TODO | Task 7 | Concelier Guild | Map proof confidence to merge_hash inclusion |
|
||||
| 9 | BACKPORT-8200-009 | TODO | Task 8 | QA Guild | Test evidence extraction from 4 tiers |
|
||||
| **Wave 2: Merge Hash Enhancement** | | | | | |
|
||||
| 10 | BACKPORT-8200-010 | TODO | Task 9 | Concelier Guild | Modify `MergeHashCalculator` to include patch lineage |
|
||||
| 11 | BACKPORT-8200-011 | TODO | Task 10 | Concelier Guild | Implement patch lineage normalization |
|
||||
| 12 | BACKPORT-8200-012 | TODO | Task 11 | Concelier Guild | Update golden corpus with backport test cases |
|
||||
| 13 | BACKPORT-8200-013 | TODO | Task 12 | QA Guild | Test merge_hash differentiation for backports |
|
||||
| **Wave 3: Provenance Scope Population** | | | | | |
|
||||
| 14 | BACKPORT-8200-014 | TODO | Task 13 | Concelier Guild | Create provenance_scope on canonical creation |
|
||||
| 15 | BACKPORT-8200-015 | TODO | Task 14 | Concelier Guild | Link evidence_ref to proofchain.proof_entries |
|
||||
| 16 | BACKPORT-8200-016 | TODO | Task 15 | Concelier Guild | Update provenance_scope on new evidence |
|
||||
| 17 | BACKPORT-8200-017 | TODO | Task 16 | QA Guild | Test provenance scope lifecycle |
|
||||
| **Wave 4: Policy Lattice** | | | | | |
|
||||
| 18 | BACKPORT-8200-018 | TODO | Task 17 | Concelier Guild | Define `ISourcePrecedenceLattice` interface |
|
||||
| 19 | BACKPORT-8200-019 | TODO | Task 18 | Concelier Guild | Implement configurable precedence rules |
|
||||
| 20 | BACKPORT-8200-020 | TODO | Task 19 | Concelier Guild | Add backport-aware overrides (distro > vendor for backports) |
|
||||
| 21 | BACKPORT-8200-021 | TODO | Task 20 | Concelier Guild | Implement exception rules (specific CVE/source pairs) |
|
||||
| 22 | BACKPORT-8200-022 | TODO | Task 21 | QA Guild | Test lattice precedence in various scenarios |
|
||||
| **Wave 5: API & Integration** | | | | | |
|
||||
| 23 | BACKPORT-8200-023 | TODO | Task 22 | Concelier Guild | Add provenance_scope to canonical advisory response |
|
||||
| 24 | BACKPORT-8200-024 | TODO | Task 23 | Concelier Guild | Create `GET /api/v1/canonical/{id}/provenance` endpoint |
|
||||
| 25 | BACKPORT-8200-025 | TODO | Task 24 | Concelier Guild | Add backport evidence to merge decision audit log |
|
||||
| 26 | BACKPORT-8200-026 | TODO | Task 25 | QA Guild | End-to-end test: ingest distro advisory with backport, verify provenance |
|
||||
| 27 | BACKPORT-8200-027 | TODO | Task 26 | Docs Guild | Document backport-aware deduplication |
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
-- Migration: 20250501000001_CreateProvenanceScope.sql
|
||||
|
||||
CREATE TABLE vuln.provenance_scope (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
canonical_id UUID NOT NULL REFERENCES vuln.advisory_canonical(id) ON DELETE CASCADE,
|
||||
distro_release TEXT NOT NULL, -- e.g., 'debian:bookworm', 'rhel:9.2', 'ubuntu:22.04'
|
||||
backport_semver TEXT, -- distro's backported version if different from upstream
|
||||
patch_id TEXT, -- upstream commit SHA or patch identifier
|
||||
patch_origin TEXT, -- 'upstream', 'distro', 'vendor'
|
||||
evidence_ref UUID, -- FK to proofchain.proof_entries
|
||||
confidence NUMERIC(3,2) DEFAULT 0.5, -- 0.0-1.0 confidence from BackportProofService
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT uq_provenance_scope_canonical_distro UNIQUE (canonical_id, distro_release)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_provenance_scope_canonical ON vuln.provenance_scope(canonical_id);
|
||||
CREATE INDEX idx_provenance_scope_distro ON vuln.provenance_scope(distro_release);
|
||||
CREATE INDEX idx_provenance_scope_patch ON vuln.provenance_scope(patch_id) WHERE patch_id IS NOT NULL;
|
||||
|
||||
CREATE TRIGGER trg_provenance_scope_updated
|
||||
BEFORE UPDATE ON vuln.provenance_scope
|
||||
FOR EACH ROW EXECUTE FUNCTION vuln.update_timestamp();
|
||||
|
||||
COMMENT ON TABLE vuln.provenance_scope IS 'Distro-specific backport and patch provenance per canonical';
|
||||
COMMENT ON COLUMN vuln.provenance_scope.backport_semver IS 'Distro version containing backport (may differ from upstream fixed version)';
|
||||
COMMENT ON COLUMN vuln.provenance_scope.evidence_ref IS 'Reference to BackportProofService evidence';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Domain Models
|
||||
|
||||
```csharp
|
||||
namespace StellaOps.Concelier.Merge.Backport;
|
||||
|
||||
/// <summary>
|
||||
/// Distro-specific provenance for a canonical advisory.
|
||||
/// </summary>
|
||||
public sealed record ProvenanceScope
|
||||
{
|
||||
public Guid Id { get; init; }
|
||||
public Guid CanonicalId { get; init; }
|
||||
public required string DistroRelease { get; init; }
|
||||
public string? BackportSemver { get; init; }
|
||||
public string? PatchId { get; init; }
|
||||
public PatchOrigin? PatchOrigin { get; init; }
|
||||
public Guid? EvidenceRef { get; init; }
|
||||
public double Confidence { get; init; }
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
public DateTimeOffset UpdatedAt { get; init; }
|
||||
}
|
||||
|
||||
public enum PatchOrigin
|
||||
{
|
||||
Upstream, // Patch from upstream project
|
||||
Distro, // Distro-specific patch
|
||||
Vendor // Vendor-specific patch
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evidence used in backport determination.
|
||||
/// </summary>
|
||||
public sealed record BackportEvidence
|
||||
{
|
||||
public required string CveId { get; init; }
|
||||
public required string PackagePurl { get; init; }
|
||||
public required string DistroRelease { get; init; }
|
||||
public BackportEvidenceTier Tier { get; init; }
|
||||
public double Confidence { get; init; }
|
||||
public string? PatchId { get; init; }
|
||||
public string? BackportVersion { get; init; }
|
||||
public DateTimeOffset EvidenceDate { get; init; }
|
||||
}
|
||||
|
||||
public enum BackportEvidenceTier
|
||||
{
|
||||
DistroAdvisory = 1, // Tier 1: Direct distro advisory
|
||||
ChangelogMention = 2, // Tier 2: Changelog mentions CVE
|
||||
PatchHeader = 3, // Tier 3: Patch header or HunkSig
|
||||
BinaryFingerprint = 4 // Tier 4: Binary fingerprint match
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evidence Resolution
|
||||
|
||||
```csharp
|
||||
public interface IBackportEvidenceResolver
|
||||
{
|
||||
/// <summary>Resolve backport evidence for CVE + package combination.</summary>
|
||||
Task<BackportEvidence?> ResolveAsync(
|
||||
string cveId,
|
||||
string packagePurl,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>Resolve evidence for multiple packages.</summary>
|
||||
Task<IReadOnlyList<BackportEvidence>> ResolveBatchAsync(
|
||||
string cveId,
|
||||
IEnumerable<string> packagePurls,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public sealed class BackportEvidenceResolver : IBackportEvidenceResolver
|
||||
{
|
||||
private readonly BackportProofService _proofService;
|
||||
|
||||
public async Task<BackportEvidence?> ResolveAsync(
|
||||
string cveId,
|
||||
string packagePurl,
|
||||
CancellationToken ct)
|
||||
{
|
||||
// Call existing BackportProofService
|
||||
var proof = await _proofService.GenerateProofAsync(cveId, packagePurl, ct);
|
||||
|
||||
if (proof is null || proof.Confidence < 0.1)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
// Extract highest-tier evidence
|
||||
var distroRelease = ExtractDistroRelease(packagePurl);
|
||||
var patchId = ExtractPatchId(proof);
|
||||
var backportVersion = ExtractBackportVersion(proof);
|
||||
var tier = DetermineHighestTier(proof);
|
||||
|
||||
return new BackportEvidence
|
||||
{
|
||||
CveId = cveId,
|
||||
PackagePurl = packagePurl,
|
||||
DistroRelease = distroRelease,
|
||||
Tier = tier,
|
||||
Confidence = proof.Confidence,
|
||||
PatchId = patchId,
|
||||
BackportVersion = backportVersion,
|
||||
EvidenceDate = proof.GeneratedAt
|
||||
};
|
||||
}
|
||||
|
||||
private BackportEvidenceTier DetermineHighestTier(ProofBlob proof)
|
||||
{
|
||||
// Check evidence types present
|
||||
if (proof.Evidences.Any(e => e.Type == EvidenceType.DistroAdvisory))
|
||||
return BackportEvidenceTier.DistroAdvisory;
|
||||
if (proof.Evidences.Any(e => e.Type == EvidenceType.ChangelogMention))
|
||||
return BackportEvidenceTier.ChangelogMention;
|
||||
if (proof.Evidences.Any(e => e.Type == EvidenceType.PatchHeader))
|
||||
return BackportEvidenceTier.PatchHeader;
|
||||
if (proof.Evidences.Any(e => e.Type == EvidenceType.BinaryFingerprint))
|
||||
return BackportEvidenceTier.BinaryFingerprint;
|
||||
|
||||
return BackportEvidenceTier.DistroAdvisory; // Default
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Merge Hash with Patch Lineage
|
||||
|
||||
```csharp
|
||||
public string ComputeMergeHash(MergeHashInput input)
|
||||
{
|
||||
// Normalize inputs
|
||||
var normalizedCve = NormalizeCve(input.Cve);
|
||||
var normalizedAffects = NormalizeAffectsKey(input.AffectsKey);
|
||||
var normalizedRange = NormalizeVersionRange(input.VersionRange);
|
||||
var normalizedWeaknesses = NormalizeWeaknesses(input.Weaknesses);
|
||||
|
||||
// NEW: Include patch lineage when available
|
||||
var normalizedLineage = NormalizePatchLineage(input.PatchLineage);
|
||||
|
||||
// Build canonical string
|
||||
var builder = new StringBuilder();
|
||||
builder.Append(normalizedCve);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedAffects);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedRange);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedWeaknesses);
|
||||
builder.Append('|');
|
||||
builder.Append(normalizedLineage);
|
||||
|
||||
// SHA256 hash
|
||||
var bytes = Encoding.UTF8.GetBytes(builder.ToString());
|
||||
var hash = SHA256.HashData(bytes);
|
||||
return Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
|
||||
private string NormalizePatchLineage(string? lineage)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(lineage))
|
||||
{
|
||||
return "";
|
||||
}
|
||||
|
||||
// Extract commit SHA if present
|
||||
var commitMatch = Regex.Match(lineage, @"[0-9a-fA-F]{40}");
|
||||
if (commitMatch.Success)
|
||||
{
|
||||
return commitMatch.Value.ToLowerInvariant();
|
||||
}
|
||||
|
||||
// Normalize patch identifier
|
||||
return lineage.Trim().ToLowerInvariant();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Policy Lattice
|
||||
|
||||
```csharp
|
||||
public interface ISourcePrecedenceLattice
|
||||
{
|
||||
/// <summary>Get precedence rank for source (lower = higher priority).</summary>
|
||||
int GetPrecedence(string source, BackportContext? context = null);
|
||||
|
||||
/// <summary>Compare two sources with optional backport context.</summary>
|
||||
SourceComparison Compare(string source1, string source2, BackportContext? context = null);
|
||||
}
|
||||
|
||||
public sealed record BackportContext
|
||||
{
|
||||
public required string CveId { get; init; }
|
||||
public string? DistroRelease { get; init; }
|
||||
public bool HasBackportEvidence { get; init; }
|
||||
public double EvidenceConfidence { get; init; }
|
||||
}
|
||||
|
||||
public enum SourceComparison
|
||||
{
|
||||
Source1Higher,
|
||||
Source2Higher,
|
||||
Equal
|
||||
}
|
||||
|
||||
public sealed class ConfigurableSourcePrecedenceLattice : ISourcePrecedenceLattice
|
||||
{
|
||||
private readonly PrecedenceConfig _config;
|
||||
|
||||
public int GetPrecedence(string source, BackportContext? context)
|
||||
{
|
||||
// Check for specific overrides
|
||||
if (context?.CveId != null && _config.Overrides.TryGetValue(
|
||||
$"{context.CveId}:{source}", out var overridePrecedence))
|
||||
{
|
||||
return overridePrecedence;
|
||||
}
|
||||
|
||||
// Apply backport boost if distro has evidence
|
||||
if (context?.HasBackportEvidence == true &&
|
||||
IsDistroSource(source) &&
|
||||
context.EvidenceConfidence >= _config.BackportBoostThreshold)
|
||||
{
|
||||
var basePrecedence = _config.DefaultPrecedence.GetValueOrDefault(source, 100);
|
||||
return basePrecedence - _config.BackportBoostAmount; // Lower = higher priority
|
||||
}
|
||||
|
||||
return _config.DefaultPrecedence.GetValueOrDefault(source, 100);
|
||||
}
|
||||
|
||||
private bool IsDistroSource(string source)
|
||||
{
|
||||
return source.ToLowerInvariant() switch
|
||||
{
|
||||
"debian" or "redhat" or "suse" or "ubuntu" or "alpine" or "astra" => true,
|
||||
_ => false
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record PrecedenceConfig
|
||||
{
|
||||
public Dictionary<string, int> DefaultPrecedence { get; init; } = new()
|
||||
{
|
||||
["vendor-psirt"] = 10,
|
||||
["debian"] = 20,
|
||||
["redhat"] = 20,
|
||||
["suse"] = 20,
|
||||
["ubuntu"] = 20,
|
||||
["alpine"] = 20,
|
||||
["astra"] = 20,
|
||||
["osv"] = 30,
|
||||
["ghsa"] = 35,
|
||||
["nvd"] = 40,
|
||||
["cert"] = 50
|
||||
};
|
||||
|
||||
public Dictionary<string, int> Overrides { get; init; } = new();
|
||||
|
||||
public double BackportBoostThreshold { get; init; } = 0.7;
|
||||
public int BackportBoostAmount { get; init; } = 15;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Golden Corpus: Backport Test Cases
|
||||
|
||||
```json
|
||||
{
|
||||
"corpus": "dedup-backport-variants",
|
||||
"items": [
|
||||
{
|
||||
"id": "CVE-2024-1234-debian-backport",
|
||||
"description": "Debian backported fix to different version than upstream",
|
||||
"sources": [
|
||||
{
|
||||
"source": "nvd",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:generic/openssl@1.1.1",
|
||||
"fixed_version": "1.1.1w",
|
||||
"patch_lineage": null
|
||||
},
|
||||
{
|
||||
"source": "debian",
|
||||
"cve": "CVE-2024-1234",
|
||||
"affects_key": "pkg:deb/debian/openssl@1.1.1n-0+deb11u5",
|
||||
"fixed_version": "1.1.1n-0+deb11u6",
|
||||
"patch_lineage": "abc123def456"
|
||||
}
|
||||
],
|
||||
"expected": {
|
||||
"same_canonical": true,
|
||||
"rationale": "Same CVE, same root cause, Debian backported upstream fix",
|
||||
"provenance_scopes": [
|
||||
{
|
||||
"distro_release": "debian:bullseye",
|
||||
"backport_semver": "1.1.1n-0+deb11u6",
|
||||
"patch_origin": "upstream"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "CVE-2024-5678-distro-specific-fix",
|
||||
"description": "Distro-specific fix different from upstream",
|
||||
"sources": [
|
||||
{
|
||||
"source": "nvd",
|
||||
"cve": "CVE-2024-5678",
|
||||
"affects_key": "pkg:generic/nginx@1.20.0",
|
||||
"fixed_version": "1.20.3",
|
||||
"patch_lineage": "upstream-commit-xyz"
|
||||
},
|
||||
{
|
||||
"source": "redhat",
|
||||
"cve": "CVE-2024-5678",
|
||||
"affects_key": "pkg:rpm/redhat/nginx@1.20.1-14.el9",
|
||||
"fixed_version": "1.20.1-14.el9_2.1",
|
||||
"patch_lineage": "rhel-specific-patch-001"
|
||||
}
|
||||
],
|
||||
"expected": {
|
||||
"same_canonical": false,
|
||||
"rationale": "Different patch lineage = different canonical (RHEL has distro-specific fix)",
|
||||
"notes": "Two canonicals created, each with own provenance_scope"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from gap analysis | Project Mgmt |
|
||||
222
docs/implplan/SPRINT_8200_REPRODUCIBILITY_EPIC_SUMMARY.md
Normal file
222
docs/implplan/SPRINT_8200_REPRODUCIBILITY_EPIC_SUMMARY.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Epic 8200 · SBOM/VEX Pipeline Reproducibility
|
||||
|
||||
## Overview
|
||||
|
||||
This epic implements the reproducibility, verifiability, and audit-readiness requirements identified in the product advisory analysis of December 2024.
|
||||
|
||||
**Goal:** Ensure StellaOps produces byte-for-byte identical outputs given identical inputs, with full attestation and offline verification capabilities.
|
||||
|
||||
## Epic Timeline
|
||||
|
||||
| Phase | Sprints | Duration | Focus |
|
||||
|-------|---------|----------|-------|
|
||||
| **Phase 1: Foundation** | 8200.0001.0001 | Week 1 | VerdictId content-addressing (critical fix) |
|
||||
| **Phase 2: Validation** | 8200.0001.0002, 8200.0001.0003 | Week 1-2 | DSSE round-trips, schema validation |
|
||||
| **Phase 3: E2E** | 8200.0001.0004 | Week 2-3 | Full pipeline reproducibility test |
|
||||
| **Phase 4: Packaging** | 8200.0001.0005, 8200.0001.0006 | Week 3 | Sigstore bundles, budget attestation |
|
||||
|
||||
## Sprint Summary
|
||||
|
||||
### P0: SPRINT_8200_0001_0001 — Verdict ID Content-Addressing
|
||||
**Status:** TODO | **Effort:** 2 days | **Blocks:** All other sprints
|
||||
|
||||
**Problem:** `DeltaVerdict.VerdictId` uses random GUID instead of content hash.
|
||||
**Solution:** Implement `VerdictIdGenerator` using SHA-256 of canonical JSON.
|
||||
|
||||
| Task Count | Files Modified | Tests Added |
|
||||
|------------|----------------|-------------|
|
||||
| 12 tasks | 5 files | 4 tests |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] `VerdictIdGenerator` helper class
|
||||
- [ ] Content-addressed VerdictId in all verdict creation sites
|
||||
- [ ] Regression tests for determinism
|
||||
|
||||
---
|
||||
|
||||
### P1: SPRINT_8200_0001_0002 — DSSE Round-Trip Testing
|
||||
**Status:** TODO | **Effort:** 3 days | **Depends on:** P0
|
||||
|
||||
**Problem:** No tests validate sign → verify → re-bundle → re-verify cycle.
|
||||
**Solution:** Comprehensive round-trip test suite with cosign compatibility.
|
||||
|
||||
| Task Count | Files Created | Tests Added |
|
||||
|------------|---------------|-------------|
|
||||
| 20 tasks | 4 files | 15 tests |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] `DsseRoundtripTestFixture` with key management
|
||||
- [ ] Round-trip serialization tests
|
||||
- [ ] Cosign compatibility verification
|
||||
- [ ] Multi-signature envelope handling
|
||||
|
||||
---
|
||||
|
||||
### P2: SPRINT_8200_0001_0003 — SBOM Schema Validation CI
|
||||
**Status:** TODO | **Effort:** 1 day | **Depends on:** None
|
||||
|
||||
**Problem:** No external validator confirms schema compliance.
|
||||
**Solution:** Integrate sbom-utility for CycloneDX 1.6 and SPDX 3.0.1 validation.
|
||||
|
||||
| Task Count | Files Created | CI Jobs Added |
|
||||
|------------|---------------|---------------|
|
||||
| 17 tasks | 7 files | 4 jobs |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] Schema files committed to repo
|
||||
- [ ] `schema-validation.yml` workflow
|
||||
- [ ] Validation scripts for all SBOM formats
|
||||
- [ ] Required PR check
|
||||
|
||||
---
|
||||
|
||||
### P3: SPRINT_8200_0001_0004 — Full E2E Reproducibility Test
|
||||
**Status:** TODO | **Effort:** 5 days | **Depends on:** P0, P1
|
||||
|
||||
**Problem:** No test covers full pipeline: ingest → normalize → diff → decide → attest → bundle.
|
||||
**Solution:** Create `StellaOps.Integration.E2E` project with cross-platform verification.
|
||||
|
||||
| Task Count | Files Created | CI Jobs Added |
|
||||
|------------|---------------|---------------|
|
||||
| 26 tasks | 8 files | 4 jobs |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] Full pipeline test fixture
|
||||
- [ ] Cross-platform hash comparison (Linux, Windows, macOS)
|
||||
- [ ] Golden baseline fixtures
|
||||
- [ ] Nightly reproducibility gate
|
||||
|
||||
---
|
||||
|
||||
### P4: SPRINT_8200_0001_0005 — Sigstore Bundle Implementation
|
||||
**Status:** TODO | **Effort:** 3 days | **Depends on:** P1
|
||||
|
||||
**Problem:** Sigstore bundle type defined but not implemented.
|
||||
**Solution:** Implement v0.3 bundle marshalling/unmarshalling with offline verification.
|
||||
|
||||
| Task Count | Files Created | Tests Added |
|
||||
|------------|---------------|-------------|
|
||||
| 24 tasks | 9 files | 4 tests |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] `StellaOps.Attestor.Bundle` library
|
||||
- [ ] `SigstoreBundleBuilder` and `SigstoreBundleVerifier`
|
||||
- [ ] cosign bundle compatibility
|
||||
- [ ] CLI command `stella attest bundle`
|
||||
|
||||
---
|
||||
|
||||
### P6: SPRINT_8200_0001_0006 — Budget Threshold Attestation
|
||||
**Status:** TODO | **Effort:** 2 days | **Depends on:** P0
|
||||
|
||||
**Problem:** Unknown budget thresholds not attested in DSSE bundles.
|
||||
**Solution:** Create `BudgetCheckPredicate` and include in verdict attestations.
|
||||
|
||||
| Task Count | Files Created/Modified | Tests Added |
|
||||
|------------|------------------------|-------------|
|
||||
| 18 tasks | 7 files | 4 tests |
|
||||
|
||||
**Key Deliverables:**
|
||||
- [ ] `BudgetCheckPredicate` model
|
||||
- [ ] Budget config hash for determinism
|
||||
- [ ] Integration with `VerdictPredicateBuilder`
|
||||
- [ ] Verification rule for config drift
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ P0: Verdict │
|
||||
│ Content-Hash │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┼──────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ P1: DSSE │ │ P2: Schema │ │ P6: Budget │
|
||||
│ Round-Trip │ │ Validation │ │ Attestation │
|
||||
└────────┬────────┘ └─────────────────┘ └─────────────────┘
|
||||
│
|
||||
┌────────┴────────┐
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ P3: E2E Test │ │ P4: Sigstore │
|
||||
│ │ │ Bundle │
|
||||
└─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Total Effort Summary
|
||||
|
||||
| Sprint | Priority | Effort | Tasks | Status |
|
||||
|--------|----------|--------|-------|--------|
|
||||
| 8200.0001.0001 | P0 | 2 days | 12 | TODO |
|
||||
| 8200.0001.0002 | P1 | 3 days | 20 | TODO |
|
||||
| 8200.0001.0003 | P2 | 1 day | 17 | TODO |
|
||||
| 8200.0001.0004 | P3 | 5 days | 26 | TODO |
|
||||
| 8200.0001.0005 | P4 | 3 days | 24 | TODO |
|
||||
| 8200.0001.0006 | P6 | 2 days | 18 | TODO |
|
||||
| **Total** | — | **16 days** | **117 tasks** | — |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Must Have (Phase 1-2)
|
||||
- [ ] VerdictId is content-addressed (SHA-256)
|
||||
- [ ] DSSE round-trip tests pass
|
||||
- [ ] Schema validation in CI
|
||||
- [ ] All existing tests pass (no regressions)
|
||||
|
||||
### Should Have (Phase 3)
|
||||
- [ ] Full E2E pipeline test
|
||||
- [ ] Cross-platform reproducibility verified
|
||||
- [ ] Golden baseline established
|
||||
|
||||
### Nice to Have (Phase 4)
|
||||
- [ ] Sigstore bundle support
|
||||
- [ ] Budget attestation in verdicts
|
||||
- [ ] cosign interoperability
|
||||
|
||||
## Documentation Deliverables
|
||||
|
||||
| Document | Sprint | Status |
|
||||
|----------|--------|--------|
|
||||
| `docs/reproducibility.md` | Pre-req | DONE |
|
||||
| `docs/testing/schema-validation.md` | P2 | TODO |
|
||||
| `docs/testing/e2e-reproducibility.md` | P3 | TODO |
|
||||
| `docs/modules/attestor/bundle-format.md` | P4 | TODO |
|
||||
| `docs/modules/policy/budget-attestation.md` | P6 | TODO |
|
||||
|
||||
## Risk Register
|
||||
|
||||
| Risk | Impact | Probability | Mitigation | Owner |
|
||||
|------|--------|-------------|------------|-------|
|
||||
| Breaking change for stored verdicts | High | Medium | Migration logic for old GUID format | Policy Guild |
|
||||
| Cross-platform determinism failures | High | Medium | Canonical serialization; path normalization | Platform Guild |
|
||||
| Sigstore spec changes | Medium | Low | Pin to v0.3; monitor upstream | Attestor Guild |
|
||||
| CI performance impact | Medium | Medium | Parallelize validation jobs | Platform Guild |
|
||||
|
||||
## Execution Checkpoints
|
||||
|
||||
| Checkpoint | Date | Criteria |
|
||||
|------------|------|----------|
|
||||
| Phase 1 Complete | Week 1 end | VerdictId fix merged; tests green |
|
||||
| Phase 2 Complete | Week 2 end | DSSE round-trips pass; schema validation active |
|
||||
| Phase 3 Complete | Week 3 end | E2E test running nightly; baselines established |
|
||||
| Phase 4 Complete | Week 3 end | Sigstore bundles working; budget attestation active |
|
||||
| Epic Complete | Week 3 end | All success criteria met; docs complete |
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Product Advisory Analysis](../product-advisories/) — Original gap analysis
|
||||
- [Reproducibility Specification](../reproducibility.md) — Verdict ID formula and replay procedure
|
||||
- [Determinism Verification](../testing/determinism-verification.md) — Existing determinism infrastructure
|
||||
- [Attestor Module](../modules/attestor/README.md) — DSSE and attestation architecture
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Version | Changes |
|
||||
|------|---------|---------|
|
||||
| 2025-12-24 | 1.0 | Initial epic creation based on product advisory gap analysis |
|
||||
@@ -0,0 +1,220 @@
|
||||
# Sprint Epoch 9100: Deterministic Resolver Implementation Index
|
||||
|
||||
## Overview
|
||||
|
||||
This document serves as the master index for the Deterministic Resolver implementation initiative. It defines the complete implementation plan for a unified, auditor-friendly resolver that guarantees: **same inputs → same traversal → same verdicts → same digest**.
|
||||
|
||||
**Epoch:** 9100
|
||||
**Start Date:** 2025-12-24
|
||||
**Advisory:** `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md`
|
||||
|
||||
---
|
||||
|
||||
## Sprint Dependency Graph
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ SPRINT 9100.0001.0001 │
|
||||
│ Core Resolver Package │
|
||||
│ (StellaOps.Resolver) │
|
||||
└──────────────────┬───────────────────────┘
|
||||
│
|
||||
┌───────────────────────────┼───────────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
|
||||
│ SPRINT 9100.0001.0002│ │ SPRINT 9100.0001.0003│ │ SPRINT 9100.0002.0001│
|
||||
│ Cycle-Cut Edges │ │ EdgeId │ │ FinalDigest │
|
||||
└──────────────────────┘ └──────────────────────┘ └──────────┬───────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────────────┐
|
||||
│ │ SPRINT 9100.0002.0002│
|
||||
│ │ VerdictDigest │
|
||||
│ └──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ SPRINT 9100.0003.0002│
|
||||
│ Validation & NFC │◄───────────────────────────┐
|
||||
└──────────────────────┘ │
|
||||
│
|
||||
┌──────────────────────┐ │
|
||||
│ SPRINT 9100.0003.0001│ │
|
||||
│ Runtime Purity │────────────────────────────┘
|
||||
└──────────────────────┘ (parallel)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sprint Summary
|
||||
|
||||
| Sprint ID | Title | Priority | Tasks | Dependencies | Status |
|
||||
|-----------|-------|----------|-------|--------------|--------|
|
||||
| 9100.0001.0001 | Core Resolver Package | P0 | 24 | None | TODO |
|
||||
| 9100.0001.0002 | Cycle-Cut Edge Support | P1 | 21 | 9100.0001.0001 | TODO |
|
||||
| 9100.0001.0003 | Content-Addressed EdgeId | P2 | 19 | 9100.0001.0001 | TODO |
|
||||
| 9100.0002.0001 | FinalDigest Implementation | P1 | 24 | 9100.0001.0001 | TODO |
|
||||
| 9100.0002.0002 | Per-Node VerdictDigest | P2 | 21 | 9100.0002.0001 | TODO |
|
||||
| 9100.0003.0001 | Runtime Purity Enforcement | P1 | 28 | 9100.0001.0001 | TODO |
|
||||
| 9100.0003.0002 | Graph Validation & NFC | P3 | 28 | 9100.0001.0002 | TODO |
|
||||
|
||||
**Total Tasks:** 165
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Foundation (Sprints 9100.0001.*)
|
||||
|
||||
**Goal:** Create the core `StellaOps.Resolver` library with unified resolver pattern.
|
||||
|
||||
| Sprint | Scope | Key Deliverables |
|
||||
|--------|-------|------------------|
|
||||
| 9100.0001.0001 | Core Resolver | `DeterministicResolver`, `ResolutionResult`, `NodeId`, `Verdict` |
|
||||
| 9100.0001.0002 | Cycle Handling | `IsCycleCut` edges, cycle validation, `InvalidGraphException` |
|
||||
| 9100.0001.0003 | Edge IDs | `EdgeId`, edge delta detection, Merkle tree integration |
|
||||
|
||||
**Phase 1 Exit Criteria:**
|
||||
- `resolver.Run(graph)` returns complete `ResolutionResult`
|
||||
- Cycles require explicit `IsCycleCut` marking
|
||||
- Both NodeIds and EdgeIds are content-addressed
|
||||
- All Phase 1 tests pass
|
||||
|
||||
### Phase 2: Digest Chain (Sprints 9100.0002.*)
|
||||
|
||||
**Goal:** Implement comprehensive digest infrastructure for verification.
|
||||
|
||||
| Sprint | Scope | Key Deliverables |
|
||||
|--------|-------|------------------|
|
||||
| 9100.0002.0001 | FinalDigest | Composite run-level digest, attestation integration, verification API |
|
||||
| 9100.0002.0002 | VerdictDigest | Per-verdict digests, delta detection, diff reporting |
|
||||
|
||||
**Phase 2 Exit Criteria:**
|
||||
- `FinalDigest` enables single-value verification
|
||||
- Per-node `VerdictDigest` enables drill-down debugging
|
||||
- Attestation includes `FinalDigest` in subject
|
||||
- CLI supports `--output-digest` and `--expected-digest`
|
||||
|
||||
### Phase 3: Hardening (Sprints 9100.0003.*)
|
||||
|
||||
**Goal:** Harden determinism guarantees with runtime enforcement and validation.
|
||||
|
||||
| Sprint | Scope | Key Deliverables |
|
||||
|--------|-------|------------------|
|
||||
| 9100.0003.0001 | Runtime Purity | Prohibited service implementations, fail-fast on ambient access, audit logging |
|
||||
| 9100.0003.0002 | Validation + NFC | Pre-traversal validation, NFC normalization, evidence completeness checks |
|
||||
|
||||
**Phase 3 Exit Criteria:**
|
||||
- Runtime guards catch ambient access attempts
|
||||
- NFC normalization ensures consistent string handling
|
||||
- Graph validation prevents implicit data
|
||||
- All Phase 3 tests pass
|
||||
|
||||
---
|
||||
|
||||
## Critical Path
|
||||
|
||||
The minimum viable implementation requires:
|
||||
|
||||
1. **9100.0001.0001** (Core Resolver) — Foundation for everything
|
||||
2. **9100.0002.0001** (FinalDigest) — Primary verification artifact
|
||||
3. **9100.0001.0002** (Cycle-Cut) — Auditor transparency
|
||||
|
||||
All other sprints enhance but are not required for basic functionality.
|
||||
|
||||
---
|
||||
|
||||
## Parallel Execution Opportunities
|
||||
|
||||
The following sprints can run in parallel:
|
||||
|
||||
| Parallel Group | Sprints | Reason |
|
||||
|----------------|---------|--------|
|
||||
| After Core | 9100.0001.0002, 9100.0001.0003, 9100.0002.0001, 9100.0003.0001 | All depend only on Core Resolver |
|
||||
| After Cycle-Cut | 9100.0003.0002 | Depends on Cycle-Cut, not others |
|
||||
| After FinalDigest | 9100.0002.0002 | Depends on FinalDigest, not others |
|
||||
|
||||
**Recommended Parallelization:**
|
||||
- Team A: 9100.0001.0001 → 9100.0001.0002 → 9100.0003.0002
|
||||
- Team B: 9100.0001.0001 → 9100.0002.0001 → 9100.0002.0002
|
||||
- Team C: 9100.0001.0001 → 9100.0001.0003
|
||||
- Team D: 9100.0001.0001 → 9100.0003.0001
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Test Types by Sprint
|
||||
|
||||
| Sprint | Unit Tests | Property Tests | Integration Tests | Snapshot Tests |
|
||||
|--------|:----------:|:--------------:|:-----------------:|:--------------:|
|
||||
| 9100.0001.0001 | 6 | 3 | 0 | 1 |
|
||||
| 9100.0001.0002 | 4 | 1 | 1 | 0 |
|
||||
| 9100.0001.0003 | 3 | 1 | 1 | 0 |
|
||||
| 9100.0002.0001 | 5 | 1 | 1 | 1 |
|
||||
| 9100.0002.0002 | 4 | 1 | 1 | 0 |
|
||||
| 9100.0003.0001 | 6 | 1 | 1 | 0 |
|
||||
| 9100.0003.0002 | 6 | 1 | 1 | 0 |
|
||||
|
||||
### Mandatory Test Patterns
|
||||
|
||||
All sprints must include:
|
||||
1. **Replay Test:** Same input → identical output
|
||||
2. **Idempotency Test:** Multiple runs → same result
|
||||
3. **Determinism Test:** Order-independent processing
|
||||
|
||||
---
|
||||
|
||||
## Module Ownership
|
||||
|
||||
| Module | Sprints | Guild |
|
||||
|--------|---------|-------|
|
||||
| `StellaOps.Resolver` | 9100.0001.*, 9100.0002.0002, 9100.0003.0002 | Resolver Guild |
|
||||
| `StellaOps.Attestor.ProofChain` | 9100.0001.0003, 9100.0002.0001 | Attestor Guild |
|
||||
| `StellaOps.Policy.Engine` | 9100.0003.0001 | Policy Guild |
|
||||
| `StellaOps.Cli` | 9100.0002.0001, 9100.0002.0002 | CLI Guild |
|
||||
|
||||
---
|
||||
|
||||
## Risk Register
|
||||
|
||||
| Risk | Impact | Probability | Mitigation | Owner |
|
||||
|------|--------|-------------|------------|-------|
|
||||
| Existing code uses DateTime.UtcNow | Breaking change | High | Audit before enforcement; migration guide | Policy Guild |
|
||||
| Large graphs with many cycles | Performance | Medium | Optimize Tarjan; limit SCC reporting | Resolver Guild |
|
||||
| NFC normalization changes existing IDs | Hash mismatch | Medium | Migration path; version graph schema | Resolver Guild |
|
||||
| Canonical serialization drift | Non-deterministic | Low | Single serializer; integration tests | Resolver Guild |
|
||||
| TrustLatticeEngine API incompatible | Adapter complexity | Low | Thin wrapper; document contract | Resolver Guild |
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Replay Test Pass Rate | 100% | CI pipeline |
|
||||
| Permutation Test Pass Rate | 100% | CI pipeline |
|
||||
| Performance Overhead | < 10% | Benchmark vs current |
|
||||
| FinalDigest Verification | Single-value comparison | Auditor validation |
|
||||
| Runtime Purity Violations | 0 in production | Telemetry |
|
||||
|
||||
---
|
||||
|
||||
## Documentation Deliverables
|
||||
|
||||
| Document | Sprint | Status |
|
||||
|----------|--------|--------|
|
||||
| Product Advisory | Pre-sprints | Complete |
|
||||
| API Reference | 9100.0001.0001 | TODO |
|
||||
| Integration Guide | 9100.0002.0001 | TODO |
|
||||
| Auditor Guide | 9100.0002.0001 | TODO |
|
||||
| Migration Guide | 9100.0003.0002 | TODO |
|
||||
|
||||
---
|
||||
|
||||
## Revision History
|
||||
|
||||
| Date | Version | Changes | Author |
|
||||
|------|---------|---------|--------|
|
||||
| 2025-12-24 | 1.0 | Initial creation | Project Mgmt |
|
||||
99
docs/implplan/SPRINT_9100_0001_0001_LB_resolver_core.md
Normal file
99
docs/implplan/SPRINT_9100_0001_0001_LB_resolver_core.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Sprint 9100.0001.0001 - Core Resolver Package
|
||||
|
||||
## Topic & Scope
|
||||
- Create unified `StellaOps.Resolver` library implementing the deterministic resolver pattern.
|
||||
- Single entry point: `DeterministicResolver.Run(graph) → ResolutionResult`.
|
||||
- Integrate with existing `DeterministicGraphOrderer`, `TrustLatticeEngine`, and `CanonicalJsonSerializer`.
|
||||
- Produce `ResolutionResult` containing: TraversalSequence, Verdicts[], GraphDigest, PolicyDigest, FinalDigest.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`.
|
||||
- **Evidence:** `resolver.Run(graph)` returns complete `ResolutionResult`; replay tests pass; determinism tests pass.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: None (uses existing components).
|
||||
- Blocks: Sprint 9100.0001.0002 (Cycle-Cut), Sprint 9100.0002.0001 (FinalDigest), Sprint 9100.0002.0002 (VerdictDigest).
|
||||
- Safe to run in parallel with: Sprint 9100.0003.0001 (Runtime Purity).
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md`
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Ordering/DeterministicGraphOrderer.cs`
|
||||
- `src/Policy/__Libraries/StellaOps.Policy/TrustLattice/K4Lattice.cs`
|
||||
- `src/__Libraries/StellaOps.Canonicalization/Json/CanonicalJsonSerializer.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: Core Models** | | | | | |
|
||||
| 1 | RESOLVER-9100-001 | TODO | None | Resolver Guild | Create `StellaOps.Resolver` project with net10.0 target. Add project to solution. |
|
||||
| 2 | RESOLVER-9100-002 | TODO | RESOLVER-9100-001 | Resolver Guild | Define `NodeId` record with SHA256 computation, ordinal comparison, and `From(kind, normalizedKey)` factory. |
|
||||
| 3 | RESOLVER-9100-003 | TODO | RESOLVER-9100-002 | Resolver Guild | Define `Node` record with `NodeId Id`, `string Kind`, `JsonElement Attrs`. |
|
||||
| 4 | RESOLVER-9100-004 | TODO | RESOLVER-9100-002 | Resolver Guild | Define `Edge` record with `NodeId Src`, `string Kind`, `NodeId Dst`, `JsonElement Attrs`. |
|
||||
| 5 | RESOLVER-9100-005 | TODO | RESOLVER-9100-002 | Resolver Guild | Define `Policy` record with `string Version`, `JsonElement Rules`, `string ConstantsDigest`. |
|
||||
| 6 | RESOLVER-9100-006 | TODO | RESOLVER-9100-003 | Resolver Guild | Define `EvidenceGraph` record holding `ImmutableArray<Node> Nodes`, `ImmutableArray<Edge> Edges`. |
|
||||
| 7 | RESOLVER-9100-007 | TODO | RESOLVER-9100-002 | Resolver Guild | Define `Verdict` record with `NodeId Node`, `string Status`, `JsonElement Evidence`, `string VerdictDigest`. |
|
||||
| 8 | RESOLVER-9100-008 | TODO | RESOLVER-9100-007 | Resolver Guild | Define `ResolutionResult` record with `ImmutableArray<NodeId> TraversalSequence`, `ImmutableArray<Verdict> Verdicts`, `string GraphDigest`, `string PolicyDigest`, `string FinalDigest`. |
|
||||
| **Phase 2: Resolver Implementation** | | | | | |
|
||||
| 9 | RESOLVER-9100-009 | TODO | RESOLVER-9100-008 | Resolver Guild | Create `IDeterministicResolver` interface with `ResolutionResult Run(EvidenceGraph graph)`. |
|
||||
| 10 | RESOLVER-9100-010 | TODO | RESOLVER-9100-009 | Resolver Guild | Create `DeterministicResolver` class implementing `IDeterministicResolver`. Constructor takes `Policy`, `IGraphOrderer`, `ITrustLatticeEvaluator`, `ICanonicalSerializer`. |
|
||||
| 11 | RESOLVER-9100-011 | TODO | RESOLVER-9100-010 | Resolver Guild | Implement `Run()` method: canonicalize graph, compute traversal order, evaluate each node, compute digests. |
|
||||
| 12 | RESOLVER-9100-012 | TODO | RESOLVER-9100-011 | Resolver Guild | Implement `GatherInboundEvidence(graph, nodeId)` helper: returns all edges where `Dst == nodeId`. |
|
||||
| 13 | RESOLVER-9100-013 | TODO | RESOLVER-9100-011 | Resolver Guild | Implement `EvaluatePure(node, inbound, policy)` helper: pure evaluation function, no IO. |
|
||||
| 14 | RESOLVER-9100-014 | TODO | RESOLVER-9100-011 | Resolver Guild | Implement `ComputeFinalDigest()`: SHA256 of canonical JSON containing graphDigest, policyDigest, verdicts[]. |
|
||||
| **Phase 3: Adapters & Integration** | | | | | |
|
||||
| 15 | RESOLVER-9100-015 | TODO | RESOLVER-9100-010 | Resolver Guild | Create `IGraphOrderer` interface adapter wrapping `DeterministicGraphOrderer`. |
|
||||
| 16 | RESOLVER-9100-016 | TODO | RESOLVER-9100-010 | Resolver Guild | Create `ITrustLatticeEvaluator` interface adapter wrapping `TrustLatticeEngine`. |
|
||||
| 17 | RESOLVER-9100-017 | TODO | RESOLVER-9100-010 | Resolver Guild | Create `ICanonicalSerializer` interface adapter wrapping `CanonicalJsonSerializer`. |
|
||||
| 18 | RESOLVER-9100-018 | TODO | RESOLVER-9100-017 | Resolver Guild | Create `ResolverServiceCollectionExtensions` for DI registration. |
|
||||
| **Phase 4: Testing** | | | | | |
|
||||
| 19 | RESOLVER-9100-019 | TODO | RESOLVER-9100-011 | Resolver Guild | Create `StellaOps.Resolver.Tests` project with xUnit. |
|
||||
| 20 | RESOLVER-9100-020 | TODO | RESOLVER-9100-019 | Resolver Guild | Add replay test: same input twice → identical `FinalDigest`. |
|
||||
| 21 | RESOLVER-9100-021 | TODO | RESOLVER-9100-019 | Resolver Guild | Add permutation test: shuffle nodes/edges → identical `FinalDigest`. |
|
||||
| 22 | RESOLVER-9100-022 | TODO | RESOLVER-9100-019 | Resolver Guild | Add property test: resolver is idempotent. |
|
||||
| 23 | RESOLVER-9100-023 | TODO | RESOLVER-9100-019 | Resolver Guild | Add property test: traversal sequence matches expected topological order. |
|
||||
| 24 | RESOLVER-9100-024 | TODO | RESOLVER-9100-019 | Resolver Guild | Add snapshot test: `ResolutionResult` canonical JSON structure. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (Models):** Tasks 1-8.
|
||||
- **Wave 2 (Resolver):** Tasks 9-14.
|
||||
- **Wave 3 (Adapters):** Tasks 15-18.
|
||||
- **Wave 4 (Tests):** Tasks 19-24.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** All core records defined; NodeId, Verdict, ResolutionResult compilable.
|
||||
- **Wave 2 evidence:** `DeterministicResolver.Run()` returns complete result; digests computed.
|
||||
- **Wave 3 evidence:** DI registration works; adapters integrate with existing components.
|
||||
- **Wave 4 evidence:** All 6 tests pass; replay/permutation/idempotency verified.
|
||||
|
||||
## Interlocks
|
||||
- `DeterministicGraphOrderer` must support `IGraphOrderer` interface or be wrapped.
|
||||
- `TrustLatticeEngine` must expose pure evaluation method.
|
||||
- `CanonicalJsonSerializer` must be injectable.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 1 complete: Core models defined.
|
||||
- Wave 2 complete: Resolver implementation functional.
|
||||
- Wave 3 complete: Integration with existing components.
|
||||
- Wave 4 complete: All tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review core model design. | Architecture Guild |
|
||||
| TBD | Review resolver implementation. | Resolver Guild |
|
||||
| TBD | Run determinism test suite. | QA Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** Use existing `DeterministicGraphOrderer` rather than reimplementing.
|
||||
- **Decision:** Adapters wrap existing services to maintain backward compatibility.
|
||||
- **Decision:** `ResolutionResult` is immutable record for thread safety.
|
||||
- **Decision:** `FinalDigest` includes verdicts array to detect per-node changes.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| TrustLatticeEngine API incompatible | Adapter complexity | Create thin wrapper; document API contract | Resolver Guild |
|
||||
| Performance regression | Slow resolution | Profile; optimize hot paths; cache policy digest | Resolver Guild |
|
||||
| Serialization differences | Non-deterministic digests | Use single canonical serializer throughout | Resolver Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
93
docs/implplan/SPRINT_9100_0001_0002_LB_cycle_cut_edges.md
Normal file
93
docs/implplan/SPRINT_9100_0001_0002_LB_cycle_cut_edges.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# Sprint 9100.0001.0002 - Cycle-Cut Edge Support
|
||||
|
||||
## Topic & Scope
|
||||
- Add explicit cycle-cut edge support to the resolver graph model.
|
||||
- Edges with `IsCycleCut = true` break cycles for topological ordering.
|
||||
- Graphs with unmarked cycles → validation error before traversal.
|
||||
- Provides auditor visibility into cycle handling.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`, `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/`.
|
||||
- **Evidence:** Cycle detection validates all cycles have cut edges; unmarked cycles throw `InvalidGraphException`; audit log shows cycle-cut decisions.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver).
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0002.* (Digest sprints).
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: Cycle-Cut Edges)
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Ordering/DeterministicGraphOrderer.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: Model Extension** | | | | | |
|
||||
| 1 | CYCLE-9100-001 | TODO | Core Resolver | Resolver Guild | Add `bool IsCycleCut` property to `Edge` record (default false). |
|
||||
| 2 | CYCLE-9100-002 | TODO | CYCLE-9100-001 | Resolver Guild | Define `CycleInfo` record with `ImmutableArray<NodeId> CycleNodes`, `Edge? CutEdge`. |
|
||||
| 3 | CYCLE-9100-003 | TODO | CYCLE-9100-002 | Resolver Guild | Define `GraphValidationResult` record with `bool IsValid`, `ImmutableArray<CycleInfo> Cycles`, `ImmutableArray<string> Errors`. |
|
||||
| **Phase 2: Cycle Detection** | | | | | |
|
||||
| 4 | CYCLE-9100-004 | TODO | CYCLE-9100-003 | Resolver Guild | Implement `ICycleDetector` interface with `ImmutableArray<CycleInfo> DetectCycles(EvidenceGraph graph)`. |
|
||||
| 5 | CYCLE-9100-005 | TODO | CYCLE-9100-004 | Resolver Guild | Implement `TarjanCycleDetector` using Tarjan's SCC algorithm for cycle detection. |
|
||||
| 6 | CYCLE-9100-006 | TODO | CYCLE-9100-005 | Resolver Guild | For each detected SCC, identify if any edge in the cycle has `IsCycleCut = true`. |
|
||||
| 7 | CYCLE-9100-007 | TODO | CYCLE-9100-006 | Resolver Guild | Return `CycleInfo` with cycle nodes and the cut edge (if present). |
|
||||
| **Phase 3: Graph Validation** | | | | | |
|
||||
| 8 | CYCLE-9100-008 | TODO | CYCLE-9100-007 | Resolver Guild | Implement `IGraphValidator` interface with `GraphValidationResult Validate(EvidenceGraph graph)`. |
|
||||
| 9 | CYCLE-9100-009 | TODO | CYCLE-9100-008 | Resolver Guild | Implement `DefaultGraphValidator` that runs cycle detection. |
|
||||
| 10 | CYCLE-9100-010 | TODO | CYCLE-9100-009 | Resolver Guild | For cycles without cut edges, add error: "Cycle detected without IsCycleCut edge: {nodeIds}". |
|
||||
| 11 | CYCLE-9100-011 | TODO | CYCLE-9100-010 | Resolver Guild | Define `InvalidGraphException` with `GraphValidationResult ValidationResult` property. |
|
||||
| 12 | CYCLE-9100-012 | TODO | CYCLE-9100-011 | Resolver Guild | Integrate validation into `DeterministicResolver.Run()` before traversal. |
|
||||
| **Phase 4: Orderer Integration** | | | | | |
|
||||
| 13 | CYCLE-9100-013 | TODO | CYCLE-9100-012 | Resolver Guild | Update `DeterministicGraphOrderer` to skip `IsCycleCut` edges during topological sort. |
|
||||
| 14 | CYCLE-9100-014 | TODO | CYCLE-9100-013 | Resolver Guild | Ensure cycle-cut edges are still included in canonical edge ordering (for digest). |
|
||||
| 15 | CYCLE-9100-015 | TODO | CYCLE-9100-014 | Resolver Guild | Document cycle-cut semantics: edge is evidence but not traversal dependency. |
|
||||
| **Phase 5: Testing** | | | | | |
|
||||
| 16 | CYCLE-9100-016 | TODO | CYCLE-9100-015 | Resolver Guild | Add test: graph with marked cycle-cut edge → valid, traversal completes. |
|
||||
| 17 | CYCLE-9100-017 | TODO | CYCLE-9100-016 | Resolver Guild | Add test: graph with unmarked cycle → `InvalidGraphException` thrown. |
|
||||
| 18 | CYCLE-9100-018 | TODO | CYCLE-9100-016 | Resolver Guild | Add test: multiple cycles, all marked → valid. |
|
||||
| 19 | CYCLE-9100-019 | TODO | CYCLE-9100-016 | Resolver Guild | Add test: multiple cycles, one unmarked → exception includes cycle info. |
|
||||
| 20 | CYCLE-9100-020 | TODO | CYCLE-9100-016 | Resolver Guild | Add property test: cycle detection is deterministic (same graph → same cycles). |
|
||||
| 21 | CYCLE-9100-021 | TODO | CYCLE-9100-016 | Resolver Guild | Add test: cycle-cut edge included in graph digest. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (Models):** Tasks 1-3.
|
||||
- **Wave 2 (Detection):** Tasks 4-7.
|
||||
- **Wave 3 (Validation):** Tasks 8-12.
|
||||
- **Wave 4 (Integration):** Tasks 13-15.
|
||||
- **Wave 5 (Tests):** Tasks 16-21.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** `Edge.IsCycleCut` property defined; `CycleInfo` and `GraphValidationResult` records exist.
|
||||
- **Wave 2 evidence:** Tarjan's algorithm detects all SCCs; cycles identified correctly.
|
||||
- **Wave 3 evidence:** Validation runs before traversal; unmarked cycles throw exception.
|
||||
- **Wave 4 evidence:** Topological sort skips cut edges; digests include cut edges.
|
||||
- **Wave 5 evidence:** All 6 tests pass; cycle handling is auditable.
|
||||
|
||||
## Interlocks
|
||||
- Requires `Edge` record from Sprint 9100.0001.0001.
|
||||
- `DeterministicGraphOrderer` must be modified to respect `IsCycleCut`.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 3 complete: Validation integrated into resolver.
|
||||
- Wave 5 complete: All cycle tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review Tarjan implementation. | Architecture Guild |
|
||||
| TBD | Verify cycle-cut semantics with auditors. | Compliance Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** Use Tarjan's algorithm for SCC detection (O(V+E) complexity).
|
||||
- **Decision:** Cycle-cut edges are included in digest but excluded from traversal dependencies.
|
||||
- **Decision:** Unmarked cycles are a hard error, not a warning.
|
||||
- **Decision:** Multiple edges in a cycle can be marked; only one is required.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Large graphs with many cycles | Performance | Optimize Tarjan; limit SCC size for reporting | Resolver Guild |
|
||||
| Existing graphs have unmarked cycles | Breaking change | Migration guide; add IsCycleCut to existing edges | Resolver Guild |
|
||||
| Auditors unclear on cycle-cut semantics | Confusion | Document in proof chain spec | Docs Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
@@ -0,0 +1,87 @@
|
||||
# Sprint 9100.0001.0003 - Content-Addressed EdgeId
|
||||
|
||||
## Topic & Scope
|
||||
- Implement content-addressed edge identifiers analogous to `NodeId`.
|
||||
- `EdgeId = sha256(srcId || "->" || edgeKind || "->" || dstId)`.
|
||||
- Enable edge-level attestations, delta detection, and Merkle tree inclusion.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`, `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`.
|
||||
- **Evidence:** `EdgeId` computed deterministically; edges included in Merkle tree; edge-level delta detection works.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver) for `NodeId`.
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0002.* (Digest sprints).
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: Edge Key Computation)
|
||||
- `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Identifiers/ContentAddressedId.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: EdgeId Implementation** | | | | | |
|
||||
| 1 | EDGEID-9100-001 | TODO | Core Resolver | Resolver Guild | Define `EdgeId` record extending content-addressed pattern: `sha256(src->kind->dst)`. |
|
||||
| 2 | EDGEID-9100-002 | TODO | EDGEID-9100-001 | Resolver Guild | Implement `EdgeId.From(NodeId src, string kind, NodeId dst)` factory method. |
|
||||
| 3 | EDGEID-9100-003 | TODO | EDGEID-9100-002 | Resolver Guild | Implement `IComparable<EdgeId>` for deterministic ordering. |
|
||||
| 4 | EDGEID-9100-004 | TODO | EDGEID-9100-003 | Resolver Guild | Add `EdgeId Id` property to `Edge` record (computed on construction). |
|
||||
| 5 | EDGEID-9100-005 | TODO | EDGEID-9100-004 | Resolver Guild | Ensure `EdgeId` uses lowercase hex and normalized inputs. |
|
||||
| **Phase 2: Graph Integration** | | | | | |
|
||||
| 6 | EDGEID-9100-006 | TODO | EDGEID-9100-005 | Resolver Guild | Update `EvidenceGraph` to expose `ImmutableArray<EdgeId> EdgeIds` (computed). |
|
||||
| 7 | EDGEID-9100-007 | TODO | EDGEID-9100-006 | Resolver Guild | Update `ComputeCanonicalHash()` to include sorted EdgeIds in hash input. |
|
||||
| 8 | EDGEID-9100-008 | TODO | EDGEID-9100-007 | Resolver Guild | Verify EdgeId ordering matches edge ordering in canonical output. |
|
||||
| **Phase 3: Merkle Tree Integration** | | | | | |
|
||||
| 9 | EDGEID-9100-009 | TODO | EDGEID-9100-008 | Attestor Guild | Update `ContentAddressedIdGenerator.GraphRevisionId` to include EdgeIds in Merkle tree. |
|
||||
| 10 | EDGEID-9100-010 | TODO | EDGEID-9100-009 | Attestor Guild | Ensure EdgeIds are sorted before Merkle tree construction. |
|
||||
| 11 | EDGEID-9100-011 | TODO | EDGEID-9100-010 | Attestor Guild | Add `EdgeId` to `StellaOps.Attestor.ProofChain.Identifiers` namespace. |
|
||||
| **Phase 4: Delta Detection** | | | | | |
|
||||
| 12 | EDGEID-9100-012 | TODO | EDGEID-9100-011 | Resolver Guild | Implement `IEdgeDeltaDetector` interface: `EdgeDelta Detect(EvidenceGraph old, EvidenceGraph new)`. |
|
||||
| 13 | EDGEID-9100-013 | TODO | EDGEID-9100-012 | Resolver Guild | `EdgeDelta` contains: `AddedEdges`, `RemovedEdges`, `ModifiedEdges` (by EdgeId). |
|
||||
| 14 | EDGEID-9100-014 | TODO | EDGEID-9100-013 | Resolver Guild | Edge modification detected by: same (src, kind, dst) but different Attrs hash. |
|
||||
| **Phase 5: Testing** | | | | | |
|
||||
| 15 | EDGEID-9100-015 | TODO | EDGEID-9100-014 | Resolver Guild | Add test: EdgeId computed deterministically from src, kind, dst. |
|
||||
| 16 | EDGEID-9100-016 | TODO | EDGEID-9100-015 | Resolver Guild | Add test: EdgeId ordering is consistent with string ordering. |
|
||||
| 17 | EDGEID-9100-017 | TODO | EDGEID-9100-015 | Resolver Guild | Add test: Graph hash changes when edge added/removed. |
|
||||
| 18 | EDGEID-9100-018 | TODO | EDGEID-9100-015 | Resolver Guild | Add test: EdgeDelta correctly identifies added/removed/modified edges. |
|
||||
| 19 | EDGEID-9100-019 | TODO | EDGEID-9100-015 | Resolver Guild | Add property test: EdgeId is idempotent (same inputs → same id). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (EdgeId):** Tasks 1-5.
|
||||
- **Wave 2 (Graph):** Tasks 6-8.
|
||||
- **Wave 3 (Merkle):** Tasks 9-11.
|
||||
- **Wave 4 (Delta):** Tasks 12-14.
|
||||
- **Wave 5 (Tests):** Tasks 15-19.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** `EdgeId` record defined; factory method works.
|
||||
- **Wave 2 evidence:** Graph hash includes EdgeIds; ordering verified.
|
||||
- **Wave 3 evidence:** Merkle tree includes both NodeIds and EdgeIds.
|
||||
- **Wave 4 evidence:** Delta detection identifies edge changes.
|
||||
- **Wave 5 evidence:** All 5 tests pass.
|
||||
|
||||
## Interlocks
|
||||
- Requires `NodeId` from Sprint 9100.0001.0001.
|
||||
- `ContentAddressedIdGenerator` must be extended for EdgeId.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 2 complete: EdgeIds integrated into graph.
|
||||
- Wave 5 complete: All edge tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review EdgeId format with attestor team. | Attestor Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** EdgeId format: `sha256(srcId->kind->dstId)` with arrow separator.
|
||||
- **Decision:** EdgeId is immutable; computed once at edge construction.
|
||||
- **Decision:** Edge attrs are NOT included in EdgeId (only in attrs hash for modification detection).
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| EdgeId collisions | Incorrect deduplication | SHA256 collision is practically impossible | Resolver Guild |
|
||||
| Performance overhead | Slower graph construction | Cache EdgeId computation; lazy evaluation | Resolver Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
99
docs/implplan/SPRINT_9100_0002_0001_ATTESTOR_final_digest.md
Normal file
99
docs/implplan/SPRINT_9100_0002_0001_ATTESTOR_final_digest.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Sprint 9100.0002.0001 - FinalDigest Implementation
|
||||
|
||||
## Topic & Scope
|
||||
- Implement composite `FinalDigest` for complete resolution run verification.
|
||||
- `FinalDigest = sha256(canonical({graphDigest, policyDigest, verdicts[]}))`
|
||||
- Single digest enables: auditor verification, CI/CD gate assertions, vendor replay validation.
|
||||
- Integrate with attestation system for signed proofs.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`, `src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/`.
|
||||
- **Evidence:** `FinalDigest` computed correctly; same inputs → same digest; attestation includes FinalDigest.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver) for `ResolutionResult`.
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0001.0002, Sprint 9100.0001.0003.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: FinalDigest)
|
||||
- `src/__Libraries/StellaOps.Canonicalization/Json/CanonicalJsonSerializer.cs`
|
||||
- `docs/modules/attestor/proof-chain-specification.md`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: Digest Computation** | | | | | |
|
||||
| 1 | DIGEST-9100-001 | TODO | Core Resolver | Resolver Guild | Define `DigestInput` record: `{ GraphDigest, PolicyDigest, Verdicts[] }`. |
|
||||
| 2 | DIGEST-9100-002 | TODO | DIGEST-9100-001 | Resolver Guild | Implement `IFinalDigestComputer` interface with `string Compute(DigestInput input)`. |
|
||||
| 3 | DIGEST-9100-003 | TODO | DIGEST-9100-002 | Resolver Guild | Implement `Sha256FinalDigestComputer`: serialize input canonically, compute SHA256. |
|
||||
| 4 | DIGEST-9100-004 | TODO | DIGEST-9100-003 | Resolver Guild | Ensure verdicts array is sorted by NodeId before serialization. |
|
||||
| 5 | DIGEST-9100-005 | TODO | DIGEST-9100-004 | Resolver Guild | Integrate `IFinalDigestComputer` into `DeterministicResolver.Run()`. |
|
||||
| **Phase 2: Attestation Integration** | | | | | |
|
||||
| 6 | DIGEST-9100-006 | TODO | DIGEST-9100-005 | Attestor Guild | Define `ResolutionAttestation` predicate type for in-toto statements. |
|
||||
| 7 | DIGEST-9100-007 | TODO | DIGEST-9100-006 | Attestor Guild | Include `FinalDigest` in `ResolutionAttestation` subject descriptor. |
|
||||
| 8 | DIGEST-9100-008 | TODO | DIGEST-9100-007 | Attestor Guild | Include `GraphDigest` and `PolicyDigest` in predicate body. |
|
||||
| 9 | DIGEST-9100-009 | TODO | DIGEST-9100-008 | Attestor Guild | Add `ResolutionAttestationBuilder` to `IStatementBuilder` factory. |
|
||||
| 10 | DIGEST-9100-010 | TODO | DIGEST-9100-009 | Attestor Guild | Register predicate schema: `resolution.v1.schema.json`. |
|
||||
| **Phase 3: Verification API** | | | | | |
|
||||
| 11 | DIGEST-9100-011 | TODO | DIGEST-9100-010 | Resolver Guild | Implement `IResolutionVerifier` interface with `VerificationResult Verify(ResolutionResult expected, ResolutionResult actual)`. |
|
||||
| 12 | DIGEST-9100-012 | TODO | DIGEST-9100-011 | Resolver Guild | `VerificationResult` includes: `bool Match`, `string ExpectedDigest`, `string ActualDigest`, `ImmutableArray<string> Differences`. |
|
||||
| 13 | DIGEST-9100-013 | TODO | DIGEST-9100-012 | Resolver Guild | If `FinalDigest` matches, consider verified without deep comparison. |
|
||||
| 14 | DIGEST-9100-014 | TODO | DIGEST-9100-013 | Resolver Guild | If `FinalDigest` differs, drill down: compare GraphDigest, PolicyDigest, then per-verdict. |
|
||||
| **Phase 4: CLI Integration** | | | | | |
|
||||
| 15 | DIGEST-9100-015 | TODO | DIGEST-9100-014 | CLI Guild | Add `stellaops resolve --output-digest` option to emit FinalDigest. |
|
||||
| 16 | DIGEST-9100-016 | TODO | DIGEST-9100-015 | CLI Guild | Add `stellaops verify --expected-digest <hash>` option for verification. |
|
||||
| 17 | DIGEST-9100-017 | TODO | DIGEST-9100-016 | CLI Guild | Exit code 0 if match, non-zero if mismatch with diff output. |
|
||||
| **Phase 5: Testing** | | | | | |
|
||||
| 18 | DIGEST-9100-018 | TODO | DIGEST-9100-017 | Resolver Guild | Add test: FinalDigest is deterministic (same inputs → same digest). |
|
||||
| 19 | DIGEST-9100-019 | TODO | DIGEST-9100-018 | Resolver Guild | Add test: FinalDigest changes when any verdict changes. |
|
||||
| 20 | DIGEST-9100-020 | TODO | DIGEST-9100-018 | Resolver Guild | Add test: FinalDigest changes when graph changes. |
|
||||
| 21 | DIGEST-9100-021 | TODO | DIGEST-9100-018 | Resolver Guild | Add test: FinalDigest changes when policy changes. |
|
||||
| 22 | DIGEST-9100-022 | TODO | DIGEST-9100-018 | Resolver Guild | Add test: Verification API correctly identifies match/mismatch. |
|
||||
| 23 | DIGEST-9100-023 | TODO | DIGEST-9100-018 | Resolver Guild | Add test: Attestation includes FinalDigest in subject. |
|
||||
| 24 | DIGEST-9100-024 | TODO | DIGEST-9100-018 | Resolver Guild | Add property test: FinalDigest is collision-resistant (different inputs → different digest). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (Computation):** Tasks 1-5.
|
||||
- **Wave 2 (Attestation):** Tasks 6-10.
|
||||
- **Wave 3 (Verification):** Tasks 11-14.
|
||||
- **Wave 4 (CLI):** Tasks 15-17.
|
||||
- **Wave 5 (Tests):** Tasks 18-24.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** `FinalDigest` computed and included in `ResolutionResult`.
|
||||
- **Wave 2 evidence:** Attestation predicate includes FinalDigest; schema registered.
|
||||
- **Wave 3 evidence:** Verification API identifies mismatches with drill-down.
|
||||
- **Wave 4 evidence:** CLI commands work for digest output and verification.
|
||||
- **Wave 5 evidence:** All 7 tests pass; determinism verified.
|
||||
|
||||
## Interlocks
|
||||
- Requires `ResolutionResult` from Sprint 9100.0001.0001.
|
||||
- Attestor schema registry must accept new predicate type.
|
||||
- CLI must have access to resolver service.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 1 complete: FinalDigest computed.
|
||||
- Wave 2 complete: Attestation integration.
|
||||
- Wave 5 complete: All tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review digest format with auditors. | Compliance Guild |
|
||||
| TBD | Register predicate schema. | Attestor Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** FinalDigest is SHA256 of canonical JSON (not Merkle root).
|
||||
- **Decision:** Verdicts sorted by NodeId in digest input.
|
||||
- **Decision:** FinalDigest is the primary verification artifact; drill-down is optional.
|
||||
- **Decision:** Attestation subject is `sha256:<FinalDigest>`.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Canonical serialization drift | Different digests | Use single serializer; integration tests | Resolver Guild |
|
||||
| Large verdict arrays | Performance | Stream computation; don't materialize full JSON | Resolver Guild |
|
||||
| Attestation schema changes | Breaking change | Versioned schemas; migration path | Attestor Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
90
docs/implplan/SPRINT_9100_0002_0002_LB_verdict_digest.md
Normal file
90
docs/implplan/SPRINT_9100_0002_0002_LB_verdict_digest.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Sprint 9100.0002.0002 - Per-Node VerdictDigest
|
||||
|
||||
## Topic & Scope
|
||||
- Implement content-addressed digest for each individual verdict.
|
||||
- `VerdictDigest = sha256(canonical(verdict))` for drill-down debugging.
|
||||
- Enables identification of which specific node's verdict changed between runs.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`.
|
||||
- **Evidence:** Each `Verdict` has `VerdictDigest`; delta detection shows per-node changes; debugging identifies changed verdict.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver) for `Verdict` record.
|
||||
- Depends on: Sprint 9100.0002.0001 (FinalDigest) for integration.
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0003.*.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: Per-Node VerdictDigest)
|
||||
- `src/__Libraries/StellaOps.Canonicalization/Json/CanonicalJsonSerializer.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: VerdictDigest Computation** | | | | | |
|
||||
| 1 | VDIGEST-9100-001 | TODO | Core Resolver | Resolver Guild | Ensure `Verdict` record includes `string VerdictDigest` property. |
|
||||
| 2 | VDIGEST-9100-002 | TODO | VDIGEST-9100-001 | Resolver Guild | Implement `IVerdictDigestComputer` interface with `string Compute(Verdict verdict)`. |
|
||||
| 3 | VDIGEST-9100-003 | TODO | VDIGEST-9100-002 | Resolver Guild | Implement `Sha256VerdictDigestComputer`: exclude `VerdictDigest` field from input, serialize rest canonically, compute SHA256. |
|
||||
| 4 | VDIGEST-9100-004 | TODO | VDIGEST-9100-003 | Resolver Guild | Integrate digest computation into `DeterministicResolver.Run()` after each verdict. |
|
||||
| 5 | VDIGEST-9100-005 | TODO | VDIGEST-9100-004 | Resolver Guild | Ensure VerdictDigest is computed before adding to verdicts array. |
|
||||
| **Phase 2: Delta Detection** | | | | | |
|
||||
| 6 | VDIGEST-9100-006 | TODO | VDIGEST-9100-005 | Resolver Guild | Implement `IVerdictDeltaDetector` interface with `VerdictDelta Detect(ResolutionResult old, ResolutionResult new)`. |
|
||||
| 7 | VDIGEST-9100-007 | TODO | VDIGEST-9100-006 | Resolver Guild | `VerdictDelta` contains: `ChangedVerdicts` (by NodeId), `AddedVerdicts`, `RemovedVerdicts`. |
|
||||
| 8 | VDIGEST-9100-008 | TODO | VDIGEST-9100-007 | Resolver Guild | For each NodeId in both results, compare `VerdictDigest` to detect changes. |
|
||||
| 9 | VDIGEST-9100-009 | TODO | VDIGEST-9100-008 | Resolver Guild | Emit detailed diff for changed verdicts: old status vs new status, evidence changes. |
|
||||
| **Phase 3: Debugging Support** | | | | | |
|
||||
| 10 | VDIGEST-9100-010 | TODO | VDIGEST-9100-009 | Resolver Guild | Add `VerdictDiffReport` model with human-readable changes. |
|
||||
| 11 | VDIGEST-9100-011 | TODO | VDIGEST-9100-010 | Resolver Guild | Implement `IVerdictDiffReporter` for generating diff reports. |
|
||||
| 12 | VDIGEST-9100-012 | TODO | VDIGEST-9100-011 | Resolver Guild | Include NodeId, old digest, new digest, status change, evidence diff. |
|
||||
| **Phase 4: CLI Integration** | | | | | |
|
||||
| 13 | VDIGEST-9100-013 | TODO | VDIGEST-9100-012 | CLI Guild | Add `stellaops resolve diff <old-result> <new-result>` command. |
|
||||
| 14 | VDIGEST-9100-014 | TODO | VDIGEST-9100-013 | CLI Guild | Output changed verdicts with NodeId and status changes. |
|
||||
| 15 | VDIGEST-9100-015 | TODO | VDIGEST-9100-014 | CLI Guild | Add `--verbose` flag for full evidence diff. |
|
||||
| **Phase 5: Testing** | | | | | |
|
||||
| 16 | VDIGEST-9100-016 | TODO | VDIGEST-9100-015 | Resolver Guild | Add test: VerdictDigest is deterministic for same verdict. |
|
||||
| 17 | VDIGEST-9100-017 | TODO | VDIGEST-9100-016 | Resolver Guild | Add test: VerdictDigest changes when status changes. |
|
||||
| 18 | VDIGEST-9100-018 | TODO | VDIGEST-9100-016 | Resolver Guild | Add test: VerdictDigest changes when evidence changes. |
|
||||
| 19 | VDIGEST-9100-019 | TODO | VDIGEST-9100-016 | Resolver Guild | Add test: Delta detection correctly identifies changed verdicts. |
|
||||
| 20 | VDIGEST-9100-020 | TODO | VDIGEST-9100-016 | Resolver Guild | Add test: Delta detection handles added/removed nodes. |
|
||||
| 21 | VDIGEST-9100-021 | TODO | VDIGEST-9100-016 | Resolver Guild | Add property test: VerdictDigest excludes itself from computation (no recursion). |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (Computation):** Tasks 1-5.
|
||||
- **Wave 2 (Delta):** Tasks 6-9.
|
||||
- **Wave 3 (Debugging):** Tasks 10-12.
|
||||
- **Wave 4 (CLI):** Tasks 13-15.
|
||||
- **Wave 5 (Tests):** Tasks 16-21.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** Each verdict has VerdictDigest computed.
|
||||
- **Wave 2 evidence:** Delta detection identifies changed verdicts by NodeId.
|
||||
- **Wave 3 evidence:** Diff reports show human-readable changes.
|
||||
- **Wave 4 evidence:** CLI diff command works.
|
||||
- **Wave 5 evidence:** All 6 tests pass.
|
||||
|
||||
## Interlocks
|
||||
- Requires `Verdict` record from Sprint 9100.0001.0001.
|
||||
- Canonical serializer must handle circular reference (VerdictDigest in Verdict).
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 1 complete: VerdictDigest computed.
|
||||
- Wave 5 complete: All tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review VerdictDigest format. | Architecture Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** VerdictDigest excludes itself from computation (serialize without VerdictDigest field).
|
||||
- **Decision:** Delta detection uses NodeId as key for matching.
|
||||
- **Decision:** Evidence diff uses JSON diff algorithm.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Circular reference in serialization | Stack overflow | Explicit exclusion of VerdictDigest field | Resolver Guild |
|
||||
| Large evidence objects | Slow diff | Limit evidence size; use digest comparison first | Resolver Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
104
docs/implplan/SPRINT_9100_0003_0001_POLICY_runtime_purity.md
Normal file
104
docs/implplan/SPRINT_9100_0003_0001_POLICY_runtime_purity.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Sprint 9100.0003.0001 - Runtime Purity Enforcement
|
||||
|
||||
## Topic & Scope
|
||||
- Extend determinism enforcement from static analysis to runtime guards.
|
||||
- Prevent evaluation functions from accessing ambient state (time, network, filesystem, environment).
|
||||
- Implement dependency injection shims that fail-fast on ambient access attempts.
|
||||
- **Working directory:** `src/Policy/StellaOps.Policy.Engine/DeterminismGuard/`, `src/__Libraries/StellaOps.Resolver/`.
|
||||
- **Evidence:** Runtime guards catch ambient access; tests verify no IO during evaluation; audit log shows blocked attempts.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver) for evaluation integration.
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0002.* (Digest sprints).
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: Evidence-Only Evaluation)
|
||||
- `src/Policy/StellaOps.Policy.Engine/DeterminismGuard/ProhibitedPatternAnalyzer.cs`
|
||||
- `src/Policy/StellaOps.Policy.Engine/Evaluation/PolicyEvaluationContext.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: Ambient Service Interfaces** | | | | | |
|
||||
| 1 | PURITY-9100-001 | TODO | None | Policy Guild | Define `IAmbientTimeProvider` interface with `DateTimeOffset Now { get; }`. |
|
||||
| 2 | PURITY-9100-002 | TODO | PURITY-9100-001 | Policy Guild | Define `IAmbientNetworkAccessor` interface (empty marker for detection). |
|
||||
| 3 | PURITY-9100-003 | TODO | PURITY-9100-002 | Policy Guild | Define `IAmbientFileSystemAccessor` interface (empty marker for detection). |
|
||||
| 4 | PURITY-9100-004 | TODO | PURITY-9100-003 | Policy Guild | Define `IAmbientEnvironmentAccessor` interface with `string? GetVariable(string name)`. |
|
||||
| **Phase 2: Fail-Fast Implementations** | | | | | |
|
||||
| 5 | PURITY-9100-005 | TODO | PURITY-9100-004 | Policy Guild | Implement `ProhibitedTimeProvider` that throws `AmbientAccessViolationException` on access. |
|
||||
| 6 | PURITY-9100-006 | TODO | PURITY-9100-005 | Policy Guild | Implement `ProhibitedNetworkAccessor` that throws on any method call. |
|
||||
| 7 | PURITY-9100-007 | TODO | PURITY-9100-006 | Policy Guild | Implement `ProhibitedFileSystemAccessor` that throws on any method call. |
|
||||
| 8 | PURITY-9100-008 | TODO | PURITY-9100-007 | Policy Guild | Implement `ProhibitedEnvironmentAccessor` that throws on `GetVariable()`. |
|
||||
| 9 | PURITY-9100-009 | TODO | PURITY-9100-008 | Policy Guild | Define `AmbientAccessViolationException` with category, attempted operation, and stack trace. |
|
||||
| **Phase 3: Evaluation Context Integration** | | | | | |
|
||||
| 10 | PURITY-9100-010 | TODO | PURITY-9100-009 | Policy Guild | Update `PolicyEvaluationContext` to accept ambient service interfaces via constructor. |
|
||||
| 11 | PURITY-9100-011 | TODO | PURITY-9100-010 | Policy Guild | Default context uses prohibited implementations for all ambient services. |
|
||||
| 12 | PURITY-9100-012 | TODO | PURITY-9100-011 | Policy Guild | Add `InjectedNow` property that returns the pre-configured timestamp. |
|
||||
| 13 | PURITY-9100-013 | TODO | PURITY-9100-012 | Policy Guild | Update all evaluation code to use `context.InjectedNow` instead of `DateTime.UtcNow`. |
|
||||
| **Phase 4: Resolver Integration** | | | | | |
|
||||
| 14 | PURITY-9100-014 | TODO | PURITY-9100-013 | Resolver Guild | `DeterministicResolver` creates evaluation context with prohibited implementations. |
|
||||
| 15 | PURITY-9100-015 | TODO | PURITY-9100-014 | Resolver Guild | Add `EnsureNoAmbientInputs()` check before evaluation loop. |
|
||||
| 16 | PURITY-9100-016 | TODO | PURITY-9100-015 | Resolver Guild | Catch `AmbientAccessViolationException` and include in resolution failure. |
|
||||
| 17 | PURITY-9100-017 | TODO | PURITY-9100-016 | Resolver Guild | Add telemetry for blocked ambient access attempts. |
|
||||
| **Phase 5: Audit Logging** | | | | | |
|
||||
| 18 | PURITY-9100-018 | TODO | PURITY-9100-017 | Policy Guild | Log blocked attempts with: category, operation, caller stack, timestamp. |
|
||||
| 19 | PURITY-9100-019 | TODO | PURITY-9100-018 | Policy Guild | Include blocked attempts in resolution audit trail. |
|
||||
| 20 | PURITY-9100-020 | TODO | PURITY-9100-019 | Policy Guild | Add `PurityViolation` event for observability. |
|
||||
| **Phase 6: Testing** | | | | | |
|
||||
| 21 | PURITY-9100-021 | TODO | PURITY-9100-020 | Policy Guild | Add test: ProhibitedTimeProvider throws on access. |
|
||||
| 22 | PURITY-9100-022 | TODO | PURITY-9100-021 | Policy Guild | Add test: ProhibitedNetworkAccessor throws on access. |
|
||||
| 23 | PURITY-9100-023 | TODO | PURITY-9100-021 | Policy Guild | Add test: ProhibitedFileSystemAccessor throws on access. |
|
||||
| 24 | PURITY-9100-024 | TODO | PURITY-9100-021 | Policy Guild | Add test: ProhibitedEnvironmentAccessor throws on access. |
|
||||
| 25 | PURITY-9100-025 | TODO | PURITY-9100-021 | Policy Guild | Add test: Evaluation with InjectedNow works correctly. |
|
||||
| 26 | PURITY-9100-026 | TODO | PURITY-9100-021 | Policy Guild | Add test: Resolver catches AmbientAccessViolationException. |
|
||||
| 27 | PURITY-9100-027 | TODO | PURITY-9100-021 | Policy Guild | Add integration test: Full resolution completes without ambient access. |
|
||||
| 28 | PURITY-9100-028 | TODO | PURITY-9100-021 | Policy Guild | Add property test: Any code path using DateTime.UtcNow in evaluation fails. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (Interfaces):** Tasks 1-4.
|
||||
- **Wave 2 (Fail-Fast):** Tasks 5-9.
|
||||
- **Wave 3 (Context):** Tasks 10-13.
|
||||
- **Wave 4 (Resolver):** Tasks 14-17.
|
||||
- **Wave 5 (Audit):** Tasks 18-20.
|
||||
- **Wave 6 (Tests):** Tasks 21-28.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** All ambient service interfaces defined.
|
||||
- **Wave 2 evidence:** Prohibited implementations throw on access.
|
||||
- **Wave 3 evidence:** Evaluation context uses injected timestamp.
|
||||
- **Wave 4 evidence:** Resolver blocks ambient access during evaluation.
|
||||
- **Wave 5 evidence:** Blocked attempts are logged and auditable.
|
||||
- **Wave 6 evidence:** All 8 tests pass.
|
||||
|
||||
## Interlocks
|
||||
- `PolicyEvaluationContext` must be updated for new interfaces.
|
||||
- All evaluation code must use context instead of ambient services.
|
||||
- `ProhibitedPatternAnalyzer` continues to catch static violations.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 3 complete: Evaluation uses injected services.
|
||||
- Wave 6 complete: All tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Audit existing evaluation code for ambient access. | Policy Guild |
|
||||
| TBD | Review exception types with error handling team. | Platform Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** Prohibited implementations throw immediately (fail-fast).
|
||||
- **Decision:** Use interfaces for all ambient services to enable injection.
|
||||
- **Decision:** `InjectedNow` replaces all `DateTime.UtcNow` usage in evaluation.
|
||||
- **Decision:** Audit log includes stack trace for debugging.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Existing code uses DateTime.UtcNow | Breaking change | Audit and refactor before enforcement | Policy Guild |
|
||||
| Performance overhead from interfaces | Slower evaluation | Virtual call overhead is negligible | Policy Guild |
|
||||
| Missing ambient access points | Runtime violations | Comprehensive test coverage; static analyzer | Policy Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
101
docs/implplan/SPRINT_9100_0003_0002_LB_validation_nfc.md
Normal file
101
docs/implplan/SPRINT_9100_0003_0002_LB_validation_nfc.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Sprint 9100.0003.0002 - Graph Validation & NFC Normalization
|
||||
|
||||
## Topic & Scope
|
||||
- Implement pre-traversal graph validation ("no implicit data" assertion).
|
||||
- Add Unicode NFC normalization for string fields in graph model.
|
||||
- Ensure all evidence is explicitly present in graph before evaluation.
|
||||
- **Working directory:** `src/__Libraries/StellaOps.Resolver/`, `src/__Libraries/StellaOps.Canonicalization/`.
|
||||
- **Evidence:** Graph validation runs before traversal; NFC normalization applied to string fields; implicit data detected and rejected.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on: Sprint 9100.0001.0001 (Core Resolver) for `EvidenceGraph`.
|
||||
- Depends on: Sprint 9100.0001.0002 (Cycle-Cut) for cycle validation.
|
||||
- Blocks: None.
|
||||
- Safe to run in parallel with: Sprint 9100.0002.*.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/product-advisories/24-Dec-2025 - Deterministic Resolver Architecture.md` (Section: Graph Validation, NFC)
|
||||
- `src/__Libraries/StellaOps.Canonicalization/Json/CanonicalJsonSerializer.cs`
|
||||
|
||||
## Delivery Tracker
|
||||
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Phase 1: NFC Normalization** | | | | | |
|
||||
| 1 | VALID-9100-001 | TODO | None | Resolver Guild | Define `IStringNormalizer` interface with `string Normalize(string input)`. |
|
||||
| 2 | VALID-9100-002 | TODO | VALID-9100-001 | Resolver Guild | Implement `NfcStringNormalizer` using `string.Normalize(NormalizationForm.FormC)`. |
|
||||
| 3 | VALID-9100-003 | TODO | VALID-9100-002 | Resolver Guild | Apply NFC normalization to `NodeId` input key before hashing. |
|
||||
| 4 | VALID-9100-004 | TODO | VALID-9100-003 | Resolver Guild | Apply NFC normalization to `Edge.Kind` before EdgeId computation. |
|
||||
| 5 | VALID-9100-005 | TODO | VALID-9100-004 | Resolver Guild | Apply NFC normalization to node attribute string values. |
|
||||
| 6 | VALID-9100-006 | TODO | VALID-9100-005 | Resolver Guild | Document NFC normalization in API documentation. |
|
||||
| **Phase 2: Implicit Data Detection** | | | | | |
|
||||
| 7 | VALID-9100-007 | TODO | VALID-9100-006 | Resolver Guild | Define `ImplicitDataViolation` record: `{ ViolationType, NodeId?, Description }`. |
|
||||
| 8 | VALID-9100-008 | TODO | VALID-9100-007 | Resolver Guild | Implement `IImplicitDataDetector` interface with `ImmutableArray<ImplicitDataViolation> Detect(EvidenceGraph graph)`. |
|
||||
| 9 | VALID-9100-009 | TODO | VALID-9100-008 | Resolver Guild | Detect: edges referencing non-existent nodes. |
|
||||
| 10 | VALID-9100-010 | TODO | VALID-9100-009 | Resolver Guild | Detect: nodes with required attributes missing. |
|
||||
| 11 | VALID-9100-011 | TODO | VALID-9100-010 | Resolver Guild | Detect: duplicate NodeIds in graph. |
|
||||
| 12 | VALID-9100-012 | TODO | VALID-9100-011 | Resolver Guild | Detect: duplicate EdgeIds in graph (same src, kind, dst). |
|
||||
| **Phase 3: Evidence Completeness** | | | | | |
|
||||
| 13 | VALID-9100-013 | TODO | VALID-9100-012 | Resolver Guild | Define `IEvidenceCompletenessChecker` interface. |
|
||||
| 14 | VALID-9100-014 | TODO | VALID-9100-013 | Resolver Guild | Check: all nodes have at least one evidence edge (except roots). |
|
||||
| 15 | VALID-9100-015 | TODO | VALID-9100-014 | Resolver Guild | Check: evidence edge `proofDigest` attributes are present (if required by policy). |
|
||||
| 16 | VALID-9100-016 | TODO | VALID-9100-015 | Resolver Guild | Configurable strictness: warn vs error for missing evidence. |
|
||||
| **Phase 4: Unified Validation** | | | | | |
|
||||
| 17 | VALID-9100-017 | TODO | VALID-9100-016 | Resolver Guild | Extend `IGraphValidator` from Sprint 9100.0001.0002 with implicit data and completeness checks. |
|
||||
| 18 | VALID-9100-018 | TODO | VALID-9100-017 | Resolver Guild | `GraphValidationResult` includes: `Cycles`, `ImplicitDataViolations`, `CompletenessWarnings`. |
|
||||
| 19 | VALID-9100-019 | TODO | VALID-9100-018 | Resolver Guild | Integrate unified validation into `DeterministicResolver.Run()` before traversal. |
|
||||
| 20 | VALID-9100-020 | TODO | VALID-9100-019 | Resolver Guild | Fail-fast on errors; continue with warnings (logged). |
|
||||
| **Phase 5: Testing** | | | | | |
|
||||
| 21 | VALID-9100-021 | TODO | VALID-9100-020 | Resolver Guild | Add test: NFC normalization produces consistent NodeIds for equivalent Unicode. |
|
||||
| 22 | VALID-9100-022 | TODO | VALID-9100-021 | Resolver Guild | Add test: Edge referencing non-existent node detected. |
|
||||
| 23 | VALID-9100-023 | TODO | VALID-9100-021 | Resolver Guild | Add test: Duplicate NodeIds detected. |
|
||||
| 24 | VALID-9100-024 | TODO | VALID-9100-021 | Resolver Guild | Add test: Duplicate EdgeIds detected. |
|
||||
| 25 | VALID-9100-025 | TODO | VALID-9100-021 | Resolver Guild | Add test: Missing required attribute detected. |
|
||||
| 26 | VALID-9100-026 | TODO | VALID-9100-021 | Resolver Guild | Add test: Node without evidence edge detected (except roots). |
|
||||
| 27 | VALID-9100-027 | TODO | VALID-9100-021 | Resolver Guild | Add test: Valid graph passes all checks. |
|
||||
| 28 | VALID-9100-028 | TODO | VALID-9100-021 | Resolver Guild | Add property test: NFC normalization is idempotent. |
|
||||
|
||||
## Wave Coordination
|
||||
- **Wave 1 (NFC):** Tasks 1-6.
|
||||
- **Wave 2 (Implicit):** Tasks 7-12.
|
||||
- **Wave 3 (Completeness):** Tasks 13-16.
|
||||
- **Wave 4 (Unified):** Tasks 17-20.
|
||||
- **Wave 5 (Tests):** Tasks 21-28.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- **Wave 1 evidence:** NFC normalization applied to all string inputs.
|
||||
- **Wave 2 evidence:** Implicit data violations detected and reported.
|
||||
- **Wave 3 evidence:** Evidence completeness checked per policy.
|
||||
- **Wave 4 evidence:** Unified validation runs before traversal.
|
||||
- **Wave 5 evidence:** All 8 tests pass.
|
||||
|
||||
## Interlocks
|
||||
- Requires `EvidenceGraph` from Sprint 9100.0001.0001.
|
||||
- Extends `IGraphValidator` from Sprint 9100.0001.0002.
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Wave 1 complete: NFC normalization working.
|
||||
- Wave 4 complete: Unified validation integrated.
|
||||
- Wave 5 complete: All tests passing.
|
||||
|
||||
## Action Tracker
|
||||
| Date (UTC) | Action | Owner |
|
||||
| --- | --- | --- |
|
||||
| TBD | Review NFC normalization with i18n team. | Platform Guild |
|
||||
| TBD | Define required vs optional attributes per node kind. | Architecture Guild |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Decision:** Use NormalizationForm.FormC (canonical composition).
|
||||
- **Decision:** NFC normalization is applied during NodeId/EdgeId construction.
|
||||
- **Decision:** Missing evidence is a warning by default, error if policy requires.
|
||||
- **Decision:** Duplicate IDs are always an error.
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| NFC normalization breaks existing IDs | Hash mismatch | Migration path; version graph schema | Resolver Guild |
|
||||
| Over-strict validation | Valid graphs rejected | Configurable strictness; warning mode | Resolver Guild |
|
||||
| Performance overhead | Slow validation | Validate incrementally; cache results | Resolver Guild |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-12-24 | Sprint created based on product advisory. | Project Mgmt |
|
||||
187
docs/implplan/SPRINT_9200_0001_0000_TRIAGE_master_plan.md
Normal file
187
docs/implplan/SPRINT_9200_0001_0000_TRIAGE_master_plan.md
Normal file
@@ -0,0 +1,187 @@
|
||||
# Sprint 9200.0001.0000 · Quiet-by-Design Triage - Master Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This master plan coordinates implementation of **Quiet-by-Design Triage + Evidence-First Panels** - a UX pattern that gates noise at the source and surfaces proof with one click.
|
||||
|
||||
### Business Value
|
||||
|
||||
Most scanners dump every finding into a big list and let users filter. This:
|
||||
- Overwhelms teams with non-actionable noise
|
||||
- Hides what's actually exploitable
|
||||
- Slows compliance audits with scattered evidence
|
||||
|
||||
**Quiet-by-Design** inverts this:
|
||||
- **Default view = only actionable** (reachable, policy-relevant, unattested-but-material)
|
||||
- **Collapsed chips** for gated buckets (+N unreachable, +N policy-dismissed, +N backported)
|
||||
- **One-click proof** (SBOM, Reachability, VEX, Attestations, Deltas in one panel)
|
||||
- **Deterministic replay** (copy command to reproduce any verdict)
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis Summary
|
||||
|
||||
### Backend Foundations (Already Implemented)
|
||||
|
||||
| Capability | Implementation | Status |
|
||||
|------------|---------------|--------|
|
||||
| Policy verdicts | `PolicyVerdictStatus` enum with Pass/Blocked/Ignored/Warned/Deferred/Escalated | Done |
|
||||
| Reachability analysis | Three-layer stack (static, binary resolution, runtime gating) | Done |
|
||||
| VEX trust scoring | `VexSourceTrustScore` with multi-dimensional scoring | Done |
|
||||
| Evidence bundles | `EvidenceBundle`, `ProofBundle` with attestations | Done |
|
||||
| Delta comparison | `DeltaCompareResponseDto` for scan diffs | Done |
|
||||
| Replay commands | `stella replay`, `replay verify`, `replay snapshot` | Done |
|
||||
| Triage lanes | `TriageLane` enum with MutedReach, MutedVex | Done |
|
||||
|
||||
### Gaps to Fill (This Sprint Series)
|
||||
|
||||
| Gap | Description | Sprint |
|
||||
|-----|-------------|--------|
|
||||
| **Gated bucket counts** | Bulk API doesn't aggregate counts by gating reason | 9200.0001.0001 |
|
||||
| **`gating_reason` field** | Finding DTO lacks explicit gating reason | 9200.0001.0001 |
|
||||
| **VEX trust score in triage** | `TriageVexStatusDto` doesn't expose trust score | 9200.0001.0001 |
|
||||
| **SubgraphId/DeltasId linkage** | Finding DTO lacks links to evidence artifacts | 9200.0001.0001 |
|
||||
| **Unified evidence endpoint** | No single endpoint for all evidence tabs | 9200.0001.0002 |
|
||||
| **Copy-ready replay command** | No backend generates the one-liner | 9200.0001.0003 |
|
||||
| **Frontend gated chips** | UI needs to consume new backend data | 9200.0001.0004 |
|
||||
|
||||
---
|
||||
|
||||
## Sprint Breakdown
|
||||
|
||||
### Sprint 9200.0001.0001 - Gated Triage Contracts (Scanner)
|
||||
|
||||
**Focus:** Extend triage DTOs with gating explainability
|
||||
|
||||
| Deliverable | Description |
|
||||
|-------------|-------------|
|
||||
| `GatingReason` field | Add to `FindingTriageStatusDto`: "unreachable" / "policy_dismissed" / "backported" / "vex_not_affected" |
|
||||
| `IsHiddenByDefault` field | Boolean indicating if finding is gated by default view |
|
||||
| `SubgraphId` field | Link to reachability subgraph for one-click drill-down |
|
||||
| `DeltasId` field | Link to delta comparison for "what changed" |
|
||||
| VEX trust score fields | Add `TrustScore`, `PolicyTrustThreshold`, `MeetsPolicyThreshold` to `TriageVexStatusDto` |
|
||||
| Gated bucket counts | Add `GatedBucketsSummaryDto` to `BulkTriageQueryResponseDto` |
|
||||
|
||||
**Working Directory:** `src/Scanner/StellaOps.Scanner.WebService/`
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Blocks:** Sprint 9200.0001.0002, 9200.0001.0004
|
||||
|
||||
---
|
||||
|
||||
### Sprint 9200.0001.0002 - Unified Evidence Endpoint (Scanner)
|
||||
|
||||
**Focus:** Single API call for complete evidence panel
|
||||
|
||||
| Deliverable | Description |
|
||||
|-------------|-------------|
|
||||
| `GET /v1/triage/findings/{id}/evidence` | Unified endpoint returning all evidence tabs |
|
||||
| `UnifiedEvidenceResponseDto` | Contains SBOM ref, reachability subgraph, VEX claims, attestations, deltas |
|
||||
| Manifest hashes | Include manifest hashes for determinism verification |
|
||||
| Verification status | Green/red check based on evidence hash drift detection |
|
||||
|
||||
**Working Directory:** `src/Scanner/StellaOps.Scanner.WebService/`
|
||||
|
||||
**Dependencies:** Sprint 9200.0001.0001
|
||||
|
||||
**Blocks:** Sprint 9200.0001.0004
|
||||
|
||||
---
|
||||
|
||||
### Sprint 9200.0001.0003 - Replay Command Generator (CLI/Scanner)
|
||||
|
||||
**Focus:** Generate copy-ready replay commands
|
||||
|
||||
| Deliverable | Description |
|
||||
|-------------|-------------|
|
||||
| `ReplayCommandGenerator` service | Builds replay command string with all necessary hashes |
|
||||
| `ReplayCommand` field in DTO | Add to `FindingTriageStatusDto` or unified evidence response |
|
||||
| Command format | `stella scan replay --artifact <digest> --manifest <hash> --feeds <hash> --policy <hash>` |
|
||||
| Evidence bundle download | Generate downloadable ZIP/TAR with all evidence |
|
||||
|
||||
**Working Directory:** `src/Scanner/StellaOps.Scanner.WebService/`, `src/Cli/StellaOps.Cli/`
|
||||
|
||||
**Dependencies:** Sprint 9200.0001.0001
|
||||
|
||||
**Blocks:** Sprint 9200.0001.0004
|
||||
|
||||
---
|
||||
|
||||
### Sprint 9200.0001.0004 - Quiet Triage UI (Frontend)
|
||||
|
||||
**Focus:** Consume new backend APIs in Angular frontend
|
||||
|
||||
| Deliverable | Description |
|
||||
|-------------|-------------|
|
||||
| Gated bucket chips | `+N unreachable`, `+N policy-dismissed`, `+N backported` with expand/collapse |
|
||||
| "Why hidden?" explainer | Modal/panel explaining gating reason with examples |
|
||||
| VEX trust threshold display | Show "Score 0.62 vs required 0.8" in VEX tab |
|
||||
| One-click replay command | Copy button in evidence panel |
|
||||
| Evidence panel delta tab | Integrate delta comparison into evidence panel |
|
||||
|
||||
**Working Directory:** `src/Web/StellaOps.Web/`
|
||||
|
||||
**Dependencies:** Sprint 9200.0001.0001, 9200.0001.0002, 9200.0001.0003
|
||||
|
||||
**Blocks:** None (final sprint)
|
||||
|
||||
---
|
||||
|
||||
## Coordination Matrix
|
||||
|
||||
```
|
||||
0001 (Contracts)
|
||||
|
|
||||
+-------------+-------------+
|
||||
| |
|
||||
0002 (Evidence API) 0003 (Replay Command)
|
||||
| |
|
||||
+-------------+-------------+
|
||||
|
|
||||
0004 (Frontend)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| Gated bucket visibility | 100% of hidden findings have `gating_reason` | API contract tests |
|
||||
| VEX trust transparency | Trust score exposed for 100% of VEX statuses | API response validation |
|
||||
| Replay command coverage | Replay command available for 100% of findings | Integration tests |
|
||||
| Evidence panel latency | < 500ms for unified evidence endpoint | Performance benchmarks |
|
||||
| Frontend adoption | Gated chips render correctly | E2E Playwright tests |
|
||||
|
||||
---
|
||||
|
||||
## Risk Register
|
||||
|
||||
| Risk | Impact | Probability | Mitigation |
|
||||
|------|--------|-------------|------------|
|
||||
| Backend data not computed | DTOs return nulls | Low | Data already exists in backend, just not exposed |
|
||||
| Frontend/backend contract mismatch | UI errors | Medium | Shared TypeScript types, contract tests |
|
||||
| Performance regression | Slow triage views | Low | Unified endpoint reduces round-trips |
|
||||
| Gating logic complexity | Incorrect classification | Medium | Comprehensive test cases for each gating reason |
|
||||
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
| Sprint | Focus | Estimated Effort |
|
||||
|--------|-------|------------------|
|
||||
| 9200.0001.0001 | Contracts | ~3 days |
|
||||
| 9200.0001.0002 | Evidence API | ~2 days |
|
||||
| 9200.0001.0003 | Replay Command | ~2 days |
|
||||
| 9200.0001.0004 | Frontend | ~3 days |
|
||||
|
||||
**Total:** ~10 days (can parallelize 0002 and 0003 after 0001)
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Master plan created from Quiet-by-Design Triage product advisory gap analysis. | Project Mgmt |
|
||||
@@ -0,0 +1,535 @@
|
||||
# Sprint 9200.0001.0001 · Gated Triage Contracts
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Extend Scanner triage DTOs with **gating explainability** - exposing why findings are hidden by default and providing links to supporting evidence. This sprint delivers:
|
||||
|
||||
1. **GatingReason field**: Explicit reason why a finding is gated (unreachable, policy_dismissed, backported, vex_not_affected)
|
||||
2. **IsHiddenByDefault field**: Boolean flag for default view filtering
|
||||
3. **SubgraphId/DeltasId fields**: Links to reachability subgraph and delta comparison
|
||||
4. **VEX trust score fields**: Trust score, policy threshold, and threshold comparison
|
||||
5. **Gated bucket counts**: Summary counts of hidden findings by gating reason in bulk queries
|
||||
6. **Backend wiring**: Service logic to compute gating reasons from existing data
|
||||
|
||||
**Working directory:** `src/Scanner/StellaOps.Scanner.WebService/`
|
||||
|
||||
**Evidence:** All triage DTOs include gating fields; bulk queries return bucket counts; integration tests verify correct classification.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** None (extends existing contracts)
|
||||
- **Blocks:** Sprint 9200.0001.0002 (Unified Evidence), Sprint 9200.0001.0004 (Frontend)
|
||||
- **Safe to run in parallel with:** Sprint 9200.0001.0003 (Replay Command) after Wave 1 completes
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/triage/proof-bundle-spec.md` (existing proof bundle design)
|
||||
- `docs/modules/scanner/README.md` (Scanner module architecture)
|
||||
- Product Advisory: Quiet-by-Design Triage + Evidence-First Panels
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
The `FindingTriageStatusDto` lacks explicit gating information:
|
||||
|
||||
```csharp
|
||||
// Current - no gating visibility
|
||||
public sealed record FindingTriageStatusDto
|
||||
{
|
||||
public required string FindingId { get; init; }
|
||||
public required string Lane { get; init; } // MutedReach, MutedVex, etc.
|
||||
public required string Verdict { get; init; }
|
||||
public string? Reason { get; init; } // Generic reason
|
||||
public TriageVexStatusDto? VexStatus { get; init; } // No trust score
|
||||
public TriageReachabilityDto? Reachability { get; init; }
|
||||
// Missing: Why is this hidden? Link to evidence? Trust threshold?
|
||||
}
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Frontend cannot show "Why hidden?" without inferring from Lane
|
||||
- No link to reachability subgraph or delta comparison
|
||||
- VEX trust score computed but not surfaced
|
||||
- Bulk queries don't aggregate gated bucket counts
|
||||
|
||||
### Target State
|
||||
|
||||
Extended DTOs with explicit gating explainability:
|
||||
|
||||
```csharp
|
||||
// Target - explicit gating visibility
|
||||
public sealed record FindingTriageStatusDto
|
||||
{
|
||||
// Existing fields...
|
||||
|
||||
// NEW: Gating explainability
|
||||
public string? GatingReason { get; init; } // "unreachable" | "policy_dismissed" | "backported" | "vex_not_affected"
|
||||
public bool IsHiddenByDefault { get; init; } // true if gated
|
||||
public string? SubgraphId { get; init; } // link to reachability graph
|
||||
public string? DeltasId { get; init; } // link to delta comparison
|
||||
}
|
||||
|
||||
public sealed record TriageVexStatusDto
|
||||
{
|
||||
// Existing fields...
|
||||
|
||||
// NEW: Trust scoring
|
||||
public double? TrustScore { get; init; } // 0.0-1.0 composite score
|
||||
public double? PolicyTrustThreshold { get; init; } // policy-defined minimum
|
||||
public bool? MeetsPolicyThreshold { get; init; } // TrustScore >= Threshold
|
||||
}
|
||||
|
||||
public sealed record BulkTriageQueryResponseDto
|
||||
{
|
||||
// Existing fields...
|
||||
|
||||
// NEW: Gated bucket counts
|
||||
public GatedBucketsSummaryDto? GatedBuckets { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### GatingReason Enum
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Contracts/GatingReason.cs
|
||||
|
||||
/// <summary>
|
||||
/// Reasons why a finding is hidden by default in quiet-by-design triage.
|
||||
/// </summary>
|
||||
public enum GatingReason
|
||||
{
|
||||
/// <summary>Not gated - visible in default view.</summary>
|
||||
None = 0,
|
||||
|
||||
/// <summary>Finding is not reachable from any entrypoint.</summary>
|
||||
Unreachable = 1,
|
||||
|
||||
/// <summary>Policy rule dismissed this finding (waived, tolerated).</summary>
|
||||
PolicyDismissed = 2,
|
||||
|
||||
/// <summary>Patched via distro backport; version comparison confirms fixed.</summary>
|
||||
Backported = 3,
|
||||
|
||||
/// <summary>VEX statement declares not_affected with sufficient trust.</summary>
|
||||
VexNotAffected = 4,
|
||||
|
||||
/// <summary>Superseded by newer advisory or CVE.</summary>
|
||||
Superseded = 5,
|
||||
|
||||
/// <summary>Muted by user decision (explicit acknowledgement).</summary>
|
||||
UserMuted = 6
|
||||
}
|
||||
```
|
||||
|
||||
### Extended FindingTriageStatusDto
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Contracts/TriageContracts.cs
|
||||
|
||||
/// <summary>
|
||||
/// Response DTO for finding triage status with gating explainability.
|
||||
/// </summary>
|
||||
public sealed record FindingTriageStatusDto
|
||||
{
|
||||
// === Existing Fields ===
|
||||
|
||||
/// <summary>Unique finding identifier.</summary>
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
/// <summary>Current triage lane.</summary>
|
||||
public required string Lane { get; init; }
|
||||
|
||||
/// <summary>Final verdict (Ship/Block/Exception).</summary>
|
||||
public required string Verdict { get; init; }
|
||||
|
||||
/// <summary>Human-readable reason for the current status.</summary>
|
||||
public string? Reason { get; init; }
|
||||
|
||||
/// <summary>VEX status if applicable.</summary>
|
||||
public TriageVexStatusDto? VexStatus { get; init; }
|
||||
|
||||
/// <summary>Reachability determination if applicable.</summary>
|
||||
public TriageReachabilityDto? Reachability { get; init; }
|
||||
|
||||
/// <summary>Risk score information.</summary>
|
||||
public TriageRiskScoreDto? RiskScore { get; init; }
|
||||
|
||||
/// <summary>Policy counterfactuals - what would flip this to Ship.</summary>
|
||||
public IReadOnlyList<string>? WouldPassIf { get; init; }
|
||||
|
||||
/// <summary>Attached evidence artifacts.</summary>
|
||||
public IReadOnlyList<TriageEvidenceDto>? Evidence { get; init; }
|
||||
|
||||
/// <summary>When this status was last computed.</summary>
|
||||
public DateTimeOffset? ComputedAt { get; init; }
|
||||
|
||||
/// <summary>Link to proof bundle for this finding.</summary>
|
||||
public string? ProofBundleUri { get; init; }
|
||||
|
||||
// === NEW: Gating Explainability (Sprint 9200.0001.0001) ===
|
||||
|
||||
/// <summary>
|
||||
/// Reason why this finding is hidden in the default view.
|
||||
/// Null or "none" if finding is visible by default.
|
||||
/// </summary>
|
||||
public string? GatingReason { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// True if this finding is hidden by default in quiet-by-design triage.
|
||||
/// </summary>
|
||||
public bool IsHiddenByDefault { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Content-addressed ID of the reachability subgraph for this finding.
|
||||
/// Enables one-click drill-down to call path visualization.
|
||||
/// </summary>
|
||||
public string? SubgraphId { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// ID of the delta comparison showing what changed for this finding.
|
||||
/// Links to the most recent scan delta involving this finding.
|
||||
/// </summary>
|
||||
public string? DeltasId { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Human-readable explanation of why this finding is gated.
|
||||
/// Suitable for "Why hidden?" tooltip/modal.
|
||||
/// </summary>
|
||||
public string? GatingExplanation { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Extended TriageVexStatusDto
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// VEX status DTO with trust scoring.
|
||||
/// </summary>
|
||||
public sealed record TriageVexStatusDto
|
||||
{
|
||||
// === Existing Fields ===
|
||||
|
||||
/// <summary>Status value (Affected, NotAffected, UnderInvestigation, Unknown).</summary>
|
||||
public required string Status { get; init; }
|
||||
|
||||
/// <summary>Justification category for NotAffected status.</summary>
|
||||
public string? Justification { get; init; }
|
||||
|
||||
/// <summary>Impact statement explaining the decision.</summary>
|
||||
public string? ImpactStatement { get; init; }
|
||||
|
||||
/// <summary>Who issued the VEX statement.</summary>
|
||||
public string? IssuedBy { get; init; }
|
||||
|
||||
/// <summary>When the VEX statement was issued.</summary>
|
||||
public DateTimeOffset? IssuedAt { get; init; }
|
||||
|
||||
/// <summary>Reference to the VEX document.</summary>
|
||||
public string? VexDocumentRef { get; init; }
|
||||
|
||||
// === NEW: Trust Scoring (Sprint 9200.0001.0001) ===
|
||||
|
||||
/// <summary>
|
||||
/// Composite trust score for the VEX source [0.0-1.0].
|
||||
/// Higher = more trustworthy source.
|
||||
/// </summary>
|
||||
public double? TrustScore { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Policy-defined minimum trust threshold for VEX acceptance.
|
||||
/// If TrustScore < PolicyTrustThreshold, VEX is not sufficient to gate.
|
||||
/// </summary>
|
||||
public double? PolicyTrustThreshold { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// True if TrustScore >= PolicyTrustThreshold.
|
||||
/// When false, finding remains actionable despite VEX not_affected.
|
||||
/// </summary>
|
||||
public bool? MeetsPolicyThreshold { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Breakdown of trust score components for transparency.
|
||||
/// </summary>
|
||||
public TrustScoreBreakdownDto? TrustBreakdown { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Breakdown of VEX trust score components.
|
||||
/// </summary>
|
||||
public sealed record TrustScoreBreakdownDto
|
||||
{
|
||||
/// <summary>Authority score [0-1]: Issuer reputation and category.</summary>
|
||||
public double Authority { get; init; }
|
||||
|
||||
/// <summary>Accuracy score [0-1]: Historical correctness.</summary>
|
||||
public double Accuracy { get; init; }
|
||||
|
||||
/// <summary>Timeliness score [0-1]: Response speed.</summary>
|
||||
public double Timeliness { get; init; }
|
||||
|
||||
/// <summary>Verification score [0-1]: Signature validity.</summary>
|
||||
public double Verification { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### GatedBucketsSummaryDto
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Summary of findings hidden by gating reason for chip display.
|
||||
/// </summary>
|
||||
public sealed record GatedBucketsSummaryDto
|
||||
{
|
||||
/// <summary>Findings hidden because not reachable from entrypoints.</summary>
|
||||
public int UnreachableCount { get; init; }
|
||||
|
||||
/// <summary>Findings hidden by policy rules (waived, tolerated).</summary>
|
||||
public int PolicyDismissedCount { get; init; }
|
||||
|
||||
/// <summary>Findings hidden because backported/patched.</summary>
|
||||
public int BackportedCount { get; init; }
|
||||
|
||||
/// <summary>Findings hidden by VEX not_affected with sufficient trust.</summary>
|
||||
public int VexNotAffectedCount { get; init; }
|
||||
|
||||
/// <summary>Findings hidden because superseded by newer advisory.</summary>
|
||||
public int SupersededCount { get; init; }
|
||||
|
||||
/// <summary>Findings explicitly muted by users.</summary>
|
||||
public int UserMutedCount { get; init; }
|
||||
|
||||
/// <summary>Total hidden findings across all gating reasons.</summary>
|
||||
public int TotalHiddenCount =>
|
||||
UnreachableCount + PolicyDismissedCount + BackportedCount +
|
||||
VexNotAffectedCount + SupersededCount + UserMutedCount;
|
||||
}
|
||||
```
|
||||
|
||||
### Extended BulkTriageQueryResponseDto
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Bulk triage query response with gated bucket summary.
|
||||
/// </summary>
|
||||
public sealed record BulkTriageQueryResponseDto
|
||||
{
|
||||
// === Existing Fields ===
|
||||
|
||||
/// <summary>The findings matching the query.</summary>
|
||||
public required IReadOnlyList<FindingTriageStatusDto> Findings { get; init; }
|
||||
|
||||
/// <summary>Total count matching the query.</summary>
|
||||
public int TotalCount { get; init; }
|
||||
|
||||
/// <summary>Next cursor for pagination.</summary>
|
||||
public string? NextCursor { get; init; }
|
||||
|
||||
/// <summary>Summary statistics.</summary>
|
||||
public TriageSummaryDto? Summary { get; init; }
|
||||
|
||||
// === NEW: Gated Buckets (Sprint 9200.0001.0001) ===
|
||||
|
||||
/// <summary>
|
||||
/// Summary of findings hidden by each gating reason.
|
||||
/// Enables "+N unreachable", "+N policy-dismissed" chip display.
|
||||
/// </summary>
|
||||
public GatedBucketsSummaryDto? GatedBuckets { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Count of actionable findings (visible in default view).
|
||||
/// </summary>
|
||||
public int ActionableCount { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gating Logic Specification
|
||||
|
||||
### Gating Reason Computation
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Services/GatingReasonResolver.cs
|
||||
|
||||
public interface IGatingReasonResolver
|
||||
{
|
||||
(GatingReason Reason, string Explanation) Resolve(
|
||||
TriageFinding finding,
|
||||
TriageReachabilityResult? reachability,
|
||||
TriageEffectiveVex? vex,
|
||||
TriageRiskResult? risk,
|
||||
VexSourceTrustScore? trustScore,
|
||||
double policyTrustThreshold);
|
||||
}
|
||||
|
||||
public class GatingReasonResolver : IGatingReasonResolver
|
||||
{
|
||||
public (GatingReason Reason, string Explanation) Resolve(...)
|
||||
{
|
||||
// Priority order for gating (first match wins):
|
||||
|
||||
// 1. Unreachable - no path from entrypoint
|
||||
if (reachability?.Reachable == TriageReachability.No)
|
||||
{
|
||||
return (GatingReason.Unreachable,
|
||||
$"Not reachable from any entrypoint (confidence: {reachability.Confidence}%)");
|
||||
}
|
||||
|
||||
// 2. Backported - version comparison confirms patched
|
||||
if (risk?.Lane == TriageLane.MutedReach && versionEvidence?.IsFixed == true)
|
||||
{
|
||||
return (GatingReason.Backported,
|
||||
$"Patched via backport ({versionEvidence.Comparator}: {versionEvidence.InstalledVersion} >= {versionEvidence.FixedVersion})");
|
||||
}
|
||||
|
||||
// 3. VEX not_affected with sufficient trust
|
||||
if (vex?.Status == TriageVexStatus.NotAffected)
|
||||
{
|
||||
if (trustScore != null && trustScore.CompositeScore >= policyTrustThreshold)
|
||||
{
|
||||
return (GatingReason.VexNotAffected,
|
||||
$"VEX: not_affected by {vex.Issuer} (trust: {trustScore.CompositeScore:P0} >= {policyTrustThreshold:P0})");
|
||||
}
|
||||
// VEX exists but trust insufficient - still actionable
|
||||
}
|
||||
|
||||
// 4. Policy dismissed (waived, tolerated)
|
||||
if (risk?.Verdict == TriageVerdict.Ship && risk.Lane == TriageLane.MutedVex)
|
||||
{
|
||||
return (GatingReason.PolicyDismissed,
|
||||
$"Policy rule '{risk.PolicyId}' waived this finding: {risk.Why}");
|
||||
}
|
||||
|
||||
// 5. User explicitly muted
|
||||
if (finding.Decisions.Any(d => d.Kind == DecisionKind.Mute))
|
||||
{
|
||||
var mute = finding.Decisions.First(d => d.Kind == DecisionKind.Mute);
|
||||
return (GatingReason.UserMuted,
|
||||
$"Muted by {mute.Actor} on {mute.AppliedAt:u}: {mute.Reason}");
|
||||
}
|
||||
|
||||
// Not gated - visible in default view
|
||||
return (GatingReason.None, null);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Contract Definitions)** | | | | | |
|
||||
| 1 | GTR-9200-001 | TODO | None | Scanner Guild | Define `GatingReason` enum in `Contracts/GatingReason.cs`. |
|
||||
| 2 | GTR-9200-002 | TODO | Task 1 | Scanner Guild | Add gating fields to `FindingTriageStatusDto`: `GatingReason`, `IsHiddenByDefault`, `SubgraphId`, `DeltasId`, `GatingExplanation`. |
|
||||
| 3 | GTR-9200-003 | TODO | Task 1 | Scanner Guild | Add trust fields to `TriageVexStatusDto`: `TrustScore`, `PolicyTrustThreshold`, `MeetsPolicyThreshold`, `TrustBreakdown`. |
|
||||
| 4 | GTR-9200-004 | TODO | Task 1 | Scanner Guild | Define `TrustScoreBreakdownDto` for trust score decomposition. |
|
||||
| 5 | GTR-9200-005 | TODO | Task 1 | Scanner Guild | Define `GatedBucketsSummaryDto` for bucket counts. |
|
||||
| 6 | GTR-9200-006 | TODO | Task 5 | Scanner Guild | Add `GatedBuckets` and `ActionableCount` to `BulkTriageQueryResponseDto`. |
|
||||
| **Wave 1 (Gating Logic)** | | | | | |
|
||||
| 7 | GTR-9200-007 | TODO | Task 2 | Scanner Guild | Define `IGatingReasonResolver` interface. |
|
||||
| 8 | GTR-9200-008 | TODO | Task 7 | Scanner Guild | Implement `GatingReasonResolver` with priority-ordered gating logic. |
|
||||
| 9 | GTR-9200-009 | TODO | Task 8 | Scanner Guild | Wire gating resolver into `TriageStatusService.GetFindingStatusAsync()`. |
|
||||
| 10 | GTR-9200-010 | TODO | Task 3 | Scanner Guild | Wire `VexSourceTrustScore` into `TriageVexStatusDto` mapping. |
|
||||
| 11 | GTR-9200-011 | TODO | Task 10 | Scanner Guild | Add policy trust threshold lookup from configuration. |
|
||||
| **Wave 2 (Bucket Aggregation)** | | | | | |
|
||||
| 12 | GTR-9200-012 | TODO | Tasks 8, 9 | Scanner Guild | Implement bucket counting logic in `TriageStatusService.QueryBulkAsync()`. |
|
||||
| 13 | GTR-9200-013 | TODO | Task 12 | Scanner Guild | Add `ActionableCount` computation (total - hidden). |
|
||||
| 14 | GTR-9200-014 | TODO | Task 12 | Scanner Guild | Optimize bucket counting with single DB query using GROUP BY. |
|
||||
| **Wave 3 (Evidence Linking)** | | | | | |
|
||||
| 15 | GTR-9200-015 | TODO | Task 2 | Scanner Guild | Wire `SubgraphId` from reachability stack to DTO. |
|
||||
| 16 | GTR-9200-016 | TODO | Task 2 | Scanner Guild | Wire `DeltasId` from most recent delta comparison to DTO. |
|
||||
| 17 | GTR-9200-017 | TODO | Tasks 15, 16 | Scanner Guild | Add caching for subgraph/delta ID lookups. |
|
||||
| **Wave 4 (Tests)** | | | | | |
|
||||
| 18 | GTR-9200-018 | TODO | Tasks 1-6 | QA Guild | Add unit tests for all new DTO fields and serialization. |
|
||||
| 19 | GTR-9200-019 | TODO | Task 8 | QA Guild | Add unit tests for `GatingReasonResolver` - all gating reason paths. |
|
||||
| 20 | GTR-9200-020 | TODO | Task 12 | QA Guild | Add unit tests for bucket counting logic. |
|
||||
| 21 | GTR-9200-021 | TODO | Task 10 | QA Guild | Add unit tests for VEX trust threshold comparison. |
|
||||
| 22 | GTR-9200-022 | TODO | All | QA Guild | Add integration tests: triage endpoint returns gating fields. |
|
||||
| 23 | GTR-9200-023 | TODO | All | QA Guild | Add integration tests: bulk query returns bucket counts. |
|
||||
| 24 | GTR-9200-024 | TODO | All | QA Guild | Add snapshot tests for DTO JSON structure. |
|
||||
| **Wave 5 (Documentation)** | | | | | |
|
||||
| 25 | GTR-9200-025 | TODO | All | Docs Guild | Update `docs/modules/scanner/README.md` with gating explainability. |
|
||||
| 26 | GTR-9200-026 | TODO | All | Docs Guild | Add API reference for new DTO fields. |
|
||||
| 27 | GTR-9200-027 | TODO | All | Docs Guild | Update triage API OpenAPI spec. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-6 | Contract definitions | All DTOs compile; fields defined |
|
||||
| **Wave 1** | 7-11 | Gating logic | Resolver works; VEX trust wired |
|
||||
| **Wave 2** | 12-14 | Bucket aggregation | Bulk queries return counts |
|
||||
| **Wave 3** | 15-17 | Evidence linking | SubgraphId/DeltasId populated |
|
||||
| **Wave 4** | 18-24 | Tests | All tests pass |
|
||||
| **Wave 5** | 25-27 | Documentation | Docs updated |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Policy Trust Threshold
|
||||
|
||||
```yaml
|
||||
# etc/scanner.yaml
|
||||
triage:
|
||||
vex:
|
||||
# Minimum trust score for VEX not_affected to gate a finding
|
||||
trust_threshold: 0.8
|
||||
|
||||
gating:
|
||||
# Enable/disable specific gating reasons
|
||||
enabled_reasons:
|
||||
- unreachable
|
||||
- backported
|
||||
- vex_not_affected
|
||||
- policy_dismissed
|
||||
- user_muted
|
||||
|
||||
# Whether to show gated counts in bulk queries
|
||||
include_gated_counts: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Gating reason as string enum | JSON-friendly; avoids int serialization issues |
|
||||
| Trust threshold from config | Different orgs have different VEX acceptance criteria |
|
||||
| Explanation as human-readable string | Frontend can display directly without mapping |
|
||||
| SubgraphId/DeltasId as content-addressed IDs | Enables deterministic linking; cache-friendly |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| VEX trust score not computed for all sources | Null TrustScore | Return null; frontend handles gracefully | Scanner Guild |
|
||||
| Delta comparison not available for new findings | Null DeltasId | Expected behavior; first scan has no delta | Scanner Guild |
|
||||
| Bucket counting performance at scale | Slow bulk queries | Use indexed GROUP BY; consider materialized view | Scanner Guild |
|
||||
| Gating reason conflicts | Unclear classification | Priority-ordered resolution; document order | Scanner Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Quiet-by-Design Triage gap analysis. | Project Mgmt |
|
||||
@@ -0,0 +1,623 @@
|
||||
# Sprint 9200.0001.0002 · Unified Evidence Endpoint
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Create a **single API endpoint** that returns all evidence tabs for a finding in one call, reducing frontend round-trips and providing a complete "Evidence Panel" data package. This sprint delivers:
|
||||
|
||||
1. **Unified Evidence Endpoint**: `GET /v1/triage/findings/{findingId}/evidence`
|
||||
2. **UnifiedEvidenceResponseDto**: Complete response with SBOM, Reachability, VEX, Attestations, Deltas
|
||||
3. **Manifest Hashes**: Include all hashes needed for determinism verification
|
||||
4. **Verification Status**: Green/red check based on evidence hash drift detection
|
||||
5. **Evidence Bundle Download**: Endpoint to export complete evidence package as ZIP/TAR
|
||||
|
||||
**Working directory:** `src/Scanner/StellaOps.Scanner.WebService/`
|
||||
|
||||
**Evidence:** Single API call returns complete evidence panel data; download endpoint produces valid archive; integration tests verify all tabs populated.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 9200.0001.0001 (Gated Triage Contracts) - uses SubgraphId, DeltasId fields
|
||||
- **Blocks:** Sprint 9200.0001.0004 (Frontend) - frontend consumes this endpoint
|
||||
- **Safe to run in parallel with:** Sprint 9200.0001.0003 (Replay Command)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/triage/proof-bundle-spec.md` (Existing proof bundle design)
|
||||
- `docs/modules/scanner/evidence-bundle.md` (Existing evidence bundle design)
|
||||
- Product Advisory: Evidence-First Panels specification
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
Evidence is split across multiple endpoints:
|
||||
|
||||
| Evidence Tab | Current Endpoint | Round-trips |
|
||||
|--------------|-----------------|-------------|
|
||||
| SBOM | `/v1/sbom/{digestt}` | 1 |
|
||||
| Reachability | `/v1/reachability/{graphId}` | 1 |
|
||||
| VEX | `/v1/triage/findings/{id}` (partial) | 1 |
|
||||
| Attestations | `/v1/attestor/entries?artifact={sha}` | 1 |
|
||||
| Deltas | `/v1/delta/compare` | 1 |
|
||||
| Policy | `/v1/triage/findings/{id}` (partial) | 1 |
|
||||
|
||||
**Problems:**
|
||||
- 6 API calls to populate evidence panel
|
||||
- No unified verification status
|
||||
- No single download for audit bundle
|
||||
- Manifest hashes scattered across responses
|
||||
|
||||
### Target State
|
||||
|
||||
Single endpoint returns everything:
|
||||
|
||||
```
|
||||
GET /v1/triage/findings/{findingId}/evidence
|
||||
|
||||
Response:
|
||||
{
|
||||
"sbom": { ... },
|
||||
"reachability": { ... },
|
||||
"vex": [ ... ],
|
||||
"attestations": [ ... ],
|
||||
"deltas": { ... },
|
||||
"policy": { ... },
|
||||
"manifests": { ... },
|
||||
"verification": { "status": "verified", ... },
|
||||
"replayCommand": "stella scan replay --artifact ..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### UnifiedEvidenceResponseDto
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Contracts/UnifiedEvidenceContracts.cs
|
||||
|
||||
/// <summary>
|
||||
/// Complete evidence package for a finding - all tabs in one response.
|
||||
/// </summary>
|
||||
public sealed record UnifiedEvidenceResponseDto
|
||||
{
|
||||
/// <summary>Finding this evidence applies to.</summary>
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
/// <summary>CVE identifier.</summary>
|
||||
public required string CveId { get; init; }
|
||||
|
||||
/// <summary>Affected component PURL.</summary>
|
||||
public required string ComponentPurl { get; init; }
|
||||
|
||||
// === Evidence Tabs ===
|
||||
|
||||
/// <summary>SBOM evidence - component metadata and linkage.</summary>
|
||||
public SbomEvidenceDto? Sbom { get; init; }
|
||||
|
||||
/// <summary>Reachability evidence - call paths to vulnerable code.</summary>
|
||||
public ReachabilityEvidenceDto? Reachability { get; init; }
|
||||
|
||||
/// <summary>VEX claims from all sources with trust scores.</summary>
|
||||
public IReadOnlyList<VexClaimDto>? VexClaims { get; init; }
|
||||
|
||||
/// <summary>Attestations (in-toto/DSSE) for this artifact.</summary>
|
||||
public IReadOnlyList<AttestationSummaryDto>? Attestations { get; init; }
|
||||
|
||||
/// <summary>Delta comparison since last scan.</summary>
|
||||
public DeltaEvidenceDto? Deltas { get; init; }
|
||||
|
||||
/// <summary>Policy evaluation evidence.</summary>
|
||||
public PolicyEvidenceDto? Policy { get; init; }
|
||||
|
||||
// === Manifest Hashes ===
|
||||
|
||||
/// <summary>Content-addressed hashes for determinism verification.</summary>
|
||||
public required ManifestHashesDto Manifests { get; init; }
|
||||
|
||||
// === Verification Status ===
|
||||
|
||||
/// <summary>Overall verification status of evidence chain.</summary>
|
||||
public required VerificationStatusDto Verification { get; init; }
|
||||
|
||||
// === Replay Command ===
|
||||
|
||||
/// <summary>Copy-ready CLI command to replay this verdict.</summary>
|
||||
public string? ReplayCommand { get; init; }
|
||||
|
||||
// === Metadata ===
|
||||
|
||||
/// <summary>When this evidence was assembled.</summary>
|
||||
public required DateTimeOffset GeneratedAt { get; init; }
|
||||
|
||||
/// <summary>Cache key for this response (content-addressed).</summary>
|
||||
public string? CacheKey { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Evidence Tab DTOs
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// SBOM evidence for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record SbomEvidenceDto
|
||||
{
|
||||
/// <summary>SBOM document reference (content-addressed).</summary>
|
||||
public required string SbomRef { get; init; }
|
||||
|
||||
/// <summary>SBOM format (CycloneDX, SPDX).</summary>
|
||||
public required string Format { get; init; }
|
||||
|
||||
/// <summary>Component entry from SBOM.</summary>
|
||||
public required SbomComponentDto Component { get; init; }
|
||||
|
||||
/// <summary>Direct dependencies of this component.</summary>
|
||||
public IReadOnlyList<string>? Dependencies { get; init; }
|
||||
|
||||
/// <summary>Dependents that import this component.</summary>
|
||||
public IReadOnlyList<string>? Dependents { get; init; }
|
||||
|
||||
/// <summary>Layer where component was found (for containers).</summary>
|
||||
public string? LayerDigest { get; init; }
|
||||
}
|
||||
|
||||
public sealed record SbomComponentDto
|
||||
{
|
||||
public required string Purl { get; init; }
|
||||
public required string Name { get; init; }
|
||||
public required string Version { get; init; }
|
||||
public string? License { get; init; }
|
||||
public string? Supplier { get; init; }
|
||||
public IReadOnlyDictionary<string, string>? Hashes { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reachability evidence for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record ReachabilityEvidenceDto
|
||||
{
|
||||
/// <summary>Subgraph ID (content-addressed).</summary>
|
||||
public required string SubgraphId { get; init; }
|
||||
|
||||
/// <summary>Reachability verdict.</summary>
|
||||
public required string Status { get; init; }
|
||||
|
||||
/// <summary>Confidence score [0-1].</summary>
|
||||
public required double Confidence { get; init; }
|
||||
|
||||
/// <summary>Call paths from entrypoints to vulnerable symbol.</summary>
|
||||
public required IReadOnlyList<CallPathDto> Paths { get; init; }
|
||||
|
||||
/// <summary>Entrypoints that can reach the vulnerable code.</summary>
|
||||
public IReadOnlyList<EntrypointDto>? Entrypoints { get; init; }
|
||||
|
||||
/// <summary>Vulnerable symbol information.</summary>
|
||||
public VulnerableSymbolDto? VulnerableSymbol { get; init; }
|
||||
|
||||
/// <summary>Graph digest for determinism.</summary>
|
||||
public required string GraphDigest { get; init; }
|
||||
}
|
||||
|
||||
public sealed record CallPathDto
|
||||
{
|
||||
public required string PathId { get; init; }
|
||||
public required int HopCount { get; init; }
|
||||
public required IReadOnlyList<CallNodeDto> Nodes { get; init; }
|
||||
public double Confidence { get; init; }
|
||||
}
|
||||
|
||||
public sealed record CallNodeDto
|
||||
{
|
||||
public required string Symbol { get; init; }
|
||||
public required string File { get; init; }
|
||||
public int? Line { get; init; }
|
||||
public bool IsEntrypoint { get; init; }
|
||||
public bool IsVulnerable { get; init; }
|
||||
}
|
||||
|
||||
public sealed record EntrypointDto
|
||||
{
|
||||
public required string Symbol { get; init; }
|
||||
public required string Type { get; init; } // HTTP, CLI, MessageHandler, etc.
|
||||
public string? Route { get; init; }
|
||||
}
|
||||
|
||||
public sealed record VulnerableSymbolDto
|
||||
{
|
||||
public required string Symbol { get; init; }
|
||||
public required string File { get; init; }
|
||||
public int? Line { get; init; }
|
||||
public string? Cwe { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// VEX claim for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record VexClaimDto
|
||||
{
|
||||
/// <summary>VEX source identifier.</summary>
|
||||
public required string Source { get; init; }
|
||||
|
||||
/// <summary>VEX status (affected, not_affected, fixed, under_investigation).</summary>
|
||||
public required string Status { get; init; }
|
||||
|
||||
/// <summary>Justification for not_affected.</summary>
|
||||
public string? Justification { get; init; }
|
||||
|
||||
/// <summary>Trust score for this source [0-1].</summary>
|
||||
public double? TrustScore { get; init; }
|
||||
|
||||
/// <summary>When the VEX statement was issued.</summary>
|
||||
public DateTimeOffset? IssuedAt { get; init; }
|
||||
|
||||
/// <summary>Evidence digest for verification.</summary>
|
||||
public string? EvidenceDigest { get; init; }
|
||||
|
||||
/// <summary>Whether signature was verified.</summary>
|
||||
public bool SignatureVerified { get; init; }
|
||||
|
||||
/// <summary>VEX document reference.</summary>
|
||||
public string? DocumentRef { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Attestation summary for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record AttestationSummaryDto
|
||||
{
|
||||
/// <summary>Attestation type (sbom, scan, vex, provenance).</summary>
|
||||
public required string Type { get; init; }
|
||||
|
||||
/// <summary>Signer identity.</summary>
|
||||
public required string Signer { get; init; }
|
||||
|
||||
/// <summary>Subject digest this attestation covers.</summary>
|
||||
public required string SubjectDigest { get; init; }
|
||||
|
||||
/// <summary>DSSE envelope digest.</summary>
|
||||
public required string DsseDigest { get; init; }
|
||||
|
||||
/// <summary>When the attestation was created.</summary>
|
||||
public required DateTimeOffset SignedAt { get; init; }
|
||||
|
||||
/// <summary>Rekor log index if published.</summary>
|
||||
public long? RekorLogIndex { get; init; }
|
||||
|
||||
/// <summary>Verification status.</summary>
|
||||
public required string VerificationStatus { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Delta evidence for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record DeltaEvidenceDto
|
||||
{
|
||||
/// <summary>Delta comparison ID.</summary>
|
||||
public required string DeltasId { get; init; }
|
||||
|
||||
/// <summary>Base scan digest (previous).</summary>
|
||||
public required string BaseDigest { get; init; }
|
||||
|
||||
/// <summary>Target scan digest (current).</summary>
|
||||
public required string TargetDigest { get; init; }
|
||||
|
||||
/// <summary>What changed for this finding.</summary>
|
||||
public required DeltaChangeDto Change { get; init; }
|
||||
|
||||
/// <summary>When the comparison was generated.</summary>
|
||||
public required DateTimeOffset GeneratedAt { get; init; }
|
||||
}
|
||||
|
||||
public sealed record DeltaChangeDto
|
||||
{
|
||||
/// <summary>Change type: Added, Removed, Modified, Unchanged.</summary>
|
||||
public required string ChangeType { get; init; }
|
||||
|
||||
/// <summary>Previous verdict if changed.</summary>
|
||||
public string? PreviousVerdict { get; init; }
|
||||
|
||||
/// <summary>Current verdict.</summary>
|
||||
public string? CurrentVerdict { get; init; }
|
||||
|
||||
/// <summary>Why the verdict changed.</summary>
|
||||
public string? ChangeReason { get; init; }
|
||||
|
||||
/// <summary>Field-level changes.</summary>
|
||||
public IReadOnlyList<FieldChangeDto>? FieldChanges { get; init; }
|
||||
}
|
||||
|
||||
public sealed record FieldChangeDto
|
||||
{
|
||||
public required string Field { get; init; }
|
||||
public string? PreviousValue { get; init; }
|
||||
public string? CurrentValue { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Policy evidence for evidence panel.
|
||||
/// </summary>
|
||||
public sealed record PolicyEvidenceDto
|
||||
{
|
||||
/// <summary>Policy ID that was evaluated.</summary>
|
||||
public required string PolicyId { get; init; }
|
||||
|
||||
/// <summary>Policy version.</summary>
|
||||
public required string PolicyVersion { get; init; }
|
||||
|
||||
/// <summary>Final verdict.</summary>
|
||||
public required string Verdict { get; init; }
|
||||
|
||||
/// <summary>Rules that were evaluated.</summary>
|
||||
public IReadOnlyList<PolicyRuleResultDto>? Rules { get; init; }
|
||||
|
||||
/// <summary>Policy snapshot digest.</summary>
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
/// <summary>Counterfactuals - what would flip to pass.</summary>
|
||||
public IReadOnlyList<string>? WouldPassIf { get; init; }
|
||||
}
|
||||
|
||||
public sealed record PolicyRuleResultDto
|
||||
{
|
||||
public required string RuleId { get; init; }
|
||||
public required string RuleName { get; init; }
|
||||
public required bool Matched { get; init; }
|
||||
public required string Effect { get; init; }
|
||||
public string? Reason { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Manifest Hashes and Verification
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Content-addressed hashes for determinism verification.
|
||||
/// </summary>
|
||||
public sealed record ManifestHashesDto
|
||||
{
|
||||
/// <summary>Artifact digest (image or SBOM).</summary>
|
||||
public required string ArtifactDigest { get; init; }
|
||||
|
||||
/// <summary>Feed snapshot digest.</summary>
|
||||
public required string FeedDigest { get; init; }
|
||||
|
||||
/// <summary>Policy snapshot digest.</summary>
|
||||
public required string PolicyDigest { get; init; }
|
||||
|
||||
/// <summary>Reachability graph digest.</summary>
|
||||
public string? GraphDigest { get; init; }
|
||||
|
||||
/// <summary>Run manifest digest.</summary>
|
||||
public required string ManifestDigest { get; init; }
|
||||
|
||||
/// <summary>Scanner version.</summary>
|
||||
public required string ScannerVersion { get; init; }
|
||||
|
||||
/// <summary>Canonicalization version.</summary>
|
||||
public required string CanonicalizationVersion { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Overall verification status of evidence chain.
|
||||
/// </summary>
|
||||
public sealed record VerificationStatusDto
|
||||
{
|
||||
/// <summary>Overall status: verified, warning, failed, unknown.</summary>
|
||||
public required string Status { get; init; }
|
||||
|
||||
/// <summary>True if all hashes match stored manifests.</summary>
|
||||
public bool HashesMatch { get; init; }
|
||||
|
||||
/// <summary>True if all signatures verified.</summary>
|
||||
public bool SignaturesValid { get; init; }
|
||||
|
||||
/// <summary>True if evidence is fresh (not stale).</summary>
|
||||
public bool IsFresh { get; init; }
|
||||
|
||||
/// <summary>Age of evidence in hours.</summary>
|
||||
public double AgeHours { get; init; }
|
||||
|
||||
/// <summary>Issues found during verification.</summary>
|
||||
public IReadOnlyList<string>? Issues { get; init; }
|
||||
|
||||
/// <summary>When verification was performed.</summary>
|
||||
public required DateTimeOffset VerifiedAt { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoint Specification
|
||||
|
||||
### GET /v1/triage/findings/{findingId}/evidence
|
||||
|
||||
Returns complete evidence package for a finding.
|
||||
|
||||
**Request:**
|
||||
```
|
||||
GET /v1/triage/findings/f-abc123/evidence
|
||||
Authorization: Bearer <token>
|
||||
Accept: application/json
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
```json
|
||||
{
|
||||
"findingId": "f-abc123",
|
||||
"cveId": "CVE-2024-1234",
|
||||
"componentPurl": "pkg:npm/lodash@4.17.20",
|
||||
"sbom": {
|
||||
"sbomRef": "sha256:abc...",
|
||||
"format": "CycloneDX",
|
||||
"component": { ... }
|
||||
},
|
||||
"reachability": {
|
||||
"subgraphId": "sha256:def...",
|
||||
"status": "reachable",
|
||||
"confidence": 0.95,
|
||||
"paths": [ ... ]
|
||||
},
|
||||
"vexClaims": [
|
||||
{
|
||||
"source": "vendor:lodash",
|
||||
"status": "not_affected",
|
||||
"trustScore": 0.62,
|
||||
...
|
||||
}
|
||||
],
|
||||
"attestations": [ ... ],
|
||||
"deltas": { ... },
|
||||
"policy": { ... },
|
||||
"manifests": {
|
||||
"artifactDigest": "sha256:...",
|
||||
"feedDigest": "sha256:...",
|
||||
"policyDigest": "sha256:...",
|
||||
...
|
||||
},
|
||||
"verification": {
|
||||
"status": "verified",
|
||||
"hashesMatch": true,
|
||||
"signaturesValid": true,
|
||||
...
|
||||
},
|
||||
"replayCommand": "stella scan replay --artifact sha256:abc --manifest sha256:def --feeds sha256:ghi --policy sha256:jkl",
|
||||
"generatedAt": "2025-12-24T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### GET /v1/triage/findings/{findingId}/evidence/export
|
||||
|
||||
Downloads complete evidence bundle as archive.
|
||||
|
||||
**Request:**
|
||||
```
|
||||
GET /v1/triage/findings/f-abc123/evidence/export?format=zip
|
||||
Authorization: Bearer <token>
|
||||
Accept: application/zip
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
- Content-Type: `application/zip` or `application/gzip`
|
||||
- Content-Disposition: `attachment; filename="evidence-f-abc123.zip"`
|
||||
|
||||
**Archive Contents:**
|
||||
```
|
||||
evidence-f-abc123/
|
||||
├── manifest.json # Evidence manifest with hashes
|
||||
├── sbom.cdx.json # CycloneDX SBOM slice
|
||||
├── reachability.json # Reachability subgraph
|
||||
├── vex/
|
||||
│ ├── vendor-lodash.json # VEX statements by source
|
||||
│ └── nvd.json
|
||||
├── attestations/
|
||||
│ ├── sbom.dsse.json # DSSE envelopes
|
||||
│ └── scan.dsse.json
|
||||
├── policy/
|
||||
│ ├── snapshot.json # Policy snapshot
|
||||
│ └── evaluation.json # Policy evaluation result
|
||||
├── delta.json # Delta comparison
|
||||
└── replay-command.txt # Copy-ready replay command
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Contract Definitions)** | | | | | |
|
||||
| 1 | UEE-9200-001 | TODO | Sprint 0001 | Scanner Guild | Define `UnifiedEvidenceResponseDto` with all evidence tabs. |
|
||||
| 2 | UEE-9200-002 | TODO | Task 1 | Scanner Guild | Define `SbomEvidenceDto` and related component DTOs. |
|
||||
| 3 | UEE-9200-003 | TODO | Task 1 | Scanner Guild | Define `ReachabilityEvidenceDto` and call path DTOs. |
|
||||
| 4 | UEE-9200-004 | TODO | Task 1 | Scanner Guild | Define `VexClaimDto` with trust score. |
|
||||
| 5 | UEE-9200-005 | TODO | Task 1 | Scanner Guild | Define `AttestationSummaryDto`. |
|
||||
| 6 | UEE-9200-006 | TODO | Task 1 | Scanner Guild | Define `DeltaEvidenceDto` and change DTOs. |
|
||||
| 7 | UEE-9200-007 | TODO | Task 1 | Scanner Guild | Define `PolicyEvidenceDto` and rule result DTOs. |
|
||||
| 8 | UEE-9200-008 | TODO | Task 1 | Scanner Guild | Define `ManifestHashesDto` and `VerificationStatusDto`. |
|
||||
| **Wave 1 (Evidence Aggregator)** | | | | | |
|
||||
| 9 | UEE-9200-009 | TODO | Tasks 1-8 | Scanner Guild | Define `IUnifiedEvidenceService` interface. |
|
||||
| 10 | UEE-9200-010 | TODO | Task 9 | Scanner Guild | Implement `UnifiedEvidenceService.GetEvidenceAsync()`. |
|
||||
| 11 | UEE-9200-011 | TODO | Task 10 | Scanner Guild | Wire SBOM evidence from `ISbomRepository`. |
|
||||
| 12 | UEE-9200-012 | TODO | Task 10 | Scanner Guild | Wire reachability evidence from `IReachabilityResolver`. |
|
||||
| 13 | UEE-9200-013 | TODO | Task 10 | Scanner Guild | Wire VEX claims from `IVexClaimService`. |
|
||||
| 14 | UEE-9200-014 | TODO | Task 10 | Scanner Guild | Wire attestations from `IAttestorEntryRepository`. |
|
||||
| 15 | UEE-9200-015 | TODO | Task 10 | Scanner Guild | Wire delta evidence from `IDeltaCompareService`. |
|
||||
| 16 | UEE-9200-016 | TODO | Task 10 | Scanner Guild | Wire policy evidence from `IPolicyExplanationStore`. |
|
||||
| **Wave 2 (Verification & Manifests)** | | | | | |
|
||||
| 17 | UEE-9200-017 | TODO | Task 10 | Scanner Guild | Implement manifest hash collection from run manifest. |
|
||||
| 18 | UEE-9200-018 | TODO | Task 17 | Scanner Guild | Implement verification status computation. |
|
||||
| 19 | UEE-9200-019 | TODO | Task 18 | Scanner Guild | Implement hash drift detection. |
|
||||
| 20 | UEE-9200-020 | TODO | Task 18 | Scanner Guild | Implement signature verification status aggregation. |
|
||||
| **Wave 3 (Endpoints)** | | | | | |
|
||||
| 21 | UEE-9200-021 | TODO | Task 10 | Scanner Guild | Create `UnifiedEvidenceEndpoints.cs`. |
|
||||
| 22 | UEE-9200-022 | TODO | Task 21 | Scanner Guild | Implement `GET /v1/triage/findings/{id}/evidence`. |
|
||||
| 23 | UEE-9200-023 | TODO | Task 22 | Scanner Guild | Add caching for evidence response (content-addressed key). |
|
||||
| 24 | UEE-9200-024 | TODO | Task 22 | Scanner Guild | Add ETag/If-None-Match support. |
|
||||
| **Wave 4 (Export)** | | | | | |
|
||||
| 25 | UEE-9200-025 | TODO | Task 22 | Scanner Guild | Implement `IEvidenceBundleExporter` interface. |
|
||||
| 26 | UEE-9200-026 | TODO | Task 25 | Scanner Guild | Implement ZIP archive generation. |
|
||||
| 27 | UEE-9200-027 | TODO | Task 25 | Scanner Guild | Implement TAR.GZ archive generation. |
|
||||
| 28 | UEE-9200-028 | TODO | Task 26 | Scanner Guild | Implement `GET /v1/triage/findings/{id}/evidence/export`. |
|
||||
| 29 | UEE-9200-029 | TODO | Task 28 | Scanner Guild | Add archive manifest with hashes. |
|
||||
| **Wave 5 (Tests)** | | | | | |
|
||||
| 30 | UEE-9200-030 | TODO | Tasks 1-8 | QA Guild | Add unit tests for all DTO serialization. |
|
||||
| 31 | UEE-9200-031 | TODO | Task 10 | QA Guild | Add unit tests for evidence aggregation. |
|
||||
| 32 | UEE-9200-032 | TODO | Task 18 | QA Guild | Add unit tests for verification status. |
|
||||
| 33 | UEE-9200-033 | TODO | Task 22 | QA Guild | Add integration tests for evidence endpoint. |
|
||||
| 34 | UEE-9200-034 | TODO | Task 28 | QA Guild | Add integration tests for export endpoint. |
|
||||
| 35 | UEE-9200-035 | TODO | All | QA Guild | Add snapshot tests for response JSON structure. |
|
||||
| **Wave 6 (Documentation)** | | | | | |
|
||||
| 36 | UEE-9200-036 | TODO | All | Docs Guild | Update OpenAPI spec with new endpoints. |
|
||||
| 37 | UEE-9200-037 | TODO | All | Docs Guild | Add evidence bundle format documentation. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-8 | Contract definitions | All DTOs compile |
|
||||
| **Wave 1** | 9-16 | Evidence aggregation | Service assembles all tabs |
|
||||
| **Wave 2** | 17-20 | Verification | Hashes and signatures checked |
|
||||
| **Wave 3** | 21-24 | GET endpoint | Evidence endpoint works |
|
||||
| **Wave 4** | 25-29 | Export | Archive download works |
|
||||
| **Wave 5** | 30-35 | Tests | All tests pass |
|
||||
| **Wave 6** | 36-37 | Documentation | Docs updated |
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Single aggregated response | Reduces frontend round-trips from 6 to 1 |
|
||||
| Optional tabs (null if unavailable) | Graceful degradation for missing evidence |
|
||||
| Content-addressed cache key | Enables efficient caching and ETag |
|
||||
| ZIP and TAR.GZ export formats | Industry standard; works in all environments |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Large response size | Network latency | Compression; pagination for lists | Scanner Guild |
|
||||
| Slow aggregation | Endpoint latency | Parallel fetch; caching | Scanner Guild |
|
||||
| Missing evidence sources | Null tabs | Graceful handling; document expected nulls | Scanner Guild |
|
||||
| Export archive size | Download time | Stream generation; progress indicator | Scanner Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Quiet-by-Design Triage gap analysis. | Project Mgmt |
|
||||
@@ -0,0 +1,726 @@
|
||||
# Sprint 9200.0001.0003 · Replay Command Generator
|
||||
|
||||
## Topic & Scope
|
||||
|
||||
Generate **copy-ready replay commands** for deterministic verdict reproduction. This sprint delivers:
|
||||
|
||||
1. **ReplayCommandGenerator service**: Builds command strings with all necessary hashes
|
||||
2. **ReplayCommand field in DTOs**: Add to evidence response for frontend copy button
|
||||
3. **Evidence bundle export**: Generate downloadable ZIP with all evidence artifacts
|
||||
4. **Command format standardization**: `stella scan replay --artifact <digest> --manifest <hash> --feeds <hash> --policy <hash>`
|
||||
|
||||
**Working directory:** `src/Scanner/StellaOps.Scanner.WebService/`, `src/Cli/StellaOps.Cli/`
|
||||
|
||||
**Evidence:** Replay commands are generated for all findings; evidence bundles are downloadable; replay commands reproduce identical verdicts.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Concurrency
|
||||
|
||||
- **Depends on:** Sprint 9200.0001.0001 (Gated Triage Contracts) for DTO integration
|
||||
- **Blocks:** Sprint 9200.0001.0004 (Frontend) for copy button
|
||||
- **Safe to run in parallel with:** Sprint 9200.0001.0002 (Unified Evidence)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Prerequisites
|
||||
|
||||
- `docs/modules/scanner/README.md` (Scanner module architecture)
|
||||
- `src/Cli/StellaOps.Cli/Commands/ReplayCommandGroup.cs` (existing replay CLI)
|
||||
- `src/Testing/StellaOps.Testing.Manifests/` (run manifest models)
|
||||
- Product Advisory: Quiet-by-Design Triage + Evidence-First Panels
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Current State
|
||||
|
||||
The CLI has comprehensive replay capabilities, but:
|
||||
|
||||
```csharp
|
||||
// Current: User must manually construct replay command
|
||||
// No backend service generates the command string
|
||||
// Evidence bundles require manual assembly
|
||||
|
||||
// Existing CLI commands:
|
||||
// - stella replay --manifest <file>
|
||||
// - stella replay verify --manifest <file>
|
||||
// - stella replay snapshot --artifact <digest> --snapshot <id>
|
||||
|
||||
// Missing: Single-click command generation from finding
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Users must manually assemble replay parameters
|
||||
- Evidence bundle download requires multiple API calls
|
||||
- No standardized command format for frontend copy button
|
||||
- Replay parameters scattered across multiple sources
|
||||
|
||||
### Target State
|
||||
|
||||
Backend generates copy-ready replay commands:
|
||||
|
||||
```csharp
|
||||
// Target: ReplayCommandGenerator generates command strings
|
||||
public interface IReplayCommandGenerator
|
||||
{
|
||||
ReplayCommandInfo GenerateCommand(FindingContext context);
|
||||
}
|
||||
|
||||
// Returns:
|
||||
// Command: stella scan replay --artifact sha256:abc... --manifest sha256:def... --feeds sha256:ghi... --policy sha256:jkl...
|
||||
// ShortCommand: stella replay snapshot --verdict V-12345
|
||||
// BundleUrl: /v1/triage/findings/{id}/evidence/export
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Specification
|
||||
|
||||
### ReplayCommandGenerator Interface
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Services/ReplayCommandGenerator.cs
|
||||
|
||||
/// <summary>
|
||||
/// Generates copy-ready CLI commands for deterministic replay.
|
||||
/// </summary>
|
||||
public interface IReplayCommandGenerator
|
||||
{
|
||||
/// <summary>
|
||||
/// Generate replay command info for a specific finding.
|
||||
/// </summary>
|
||||
ReplayCommandInfo GenerateForFinding(FindingReplayContext context);
|
||||
|
||||
/// <summary>
|
||||
/// Generate replay command for a scan run.
|
||||
/// </summary>
|
||||
ReplayCommandInfo GenerateForRun(ScanRunReplayContext context);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Context for generating finding-specific replay command.
|
||||
/// </summary>
|
||||
public sealed record FindingReplayContext
|
||||
{
|
||||
/// <summary>Finding ID.</summary>
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
/// <summary>Scan run ID containing this finding.</summary>
|
||||
public required string ScanRunId { get; init; }
|
||||
|
||||
/// <summary>Artifact digest (sha256:...).</summary>
|
||||
public required string ArtifactDigest { get; init; }
|
||||
|
||||
/// <summary>Run manifest hash.</summary>
|
||||
public required string ManifestHash { get; init; }
|
||||
|
||||
/// <summary>Feed snapshot hash at time of scan.</summary>
|
||||
public required string FeedSnapshotHash { get; init; }
|
||||
|
||||
/// <summary>Policy ruleset hash at time of scan.</summary>
|
||||
public required string PolicyHash { get; init; }
|
||||
|
||||
/// <summary>Knowledge snapshot ID if available.</summary>
|
||||
public string? KnowledgeSnapshotId { get; init; }
|
||||
|
||||
/// <summary>Verdict ID for snapshot-based replay.</summary>
|
||||
public string? VerdictId { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Context for generating run-level replay command.
|
||||
/// </summary>
|
||||
public sealed record ScanRunReplayContext
|
||||
{
|
||||
/// <summary>Scan run ID.</summary>
|
||||
public required string ScanRunId { get; init; }
|
||||
|
||||
/// <summary>Run manifest hash.</summary>
|
||||
public required string ManifestHash { get; init; }
|
||||
|
||||
/// <summary>All artifact digests in the run.</summary>
|
||||
public required IReadOnlyList<string> ArtifactDigests { get; init; }
|
||||
|
||||
/// <summary>Feed snapshot hash.</summary>
|
||||
public required string FeedSnapshotHash { get; init; }
|
||||
|
||||
/// <summary>Policy hash.</summary>
|
||||
public required string PolicyHash { get; init; }
|
||||
|
||||
/// <summary>Knowledge snapshot ID.</summary>
|
||||
public string? KnowledgeSnapshotId { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### ReplayCommandInfo DTO
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Complete replay command information for frontend display.
|
||||
/// </summary>
|
||||
public sealed record ReplayCommandInfo
|
||||
{
|
||||
/// <summary>
|
||||
/// Full replay command with all parameters.
|
||||
/// Example: stella scan replay --artifact sha256:abc --manifest sha256:def --feeds sha256:ghi --policy sha256:jkl
|
||||
/// </summary>
|
||||
public required string FullCommand { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Short replay command using verdict/snapshot ID.
|
||||
/// Example: stella replay snapshot --verdict V-12345
|
||||
/// </summary>
|
||||
public string? ShortCommand { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// URL to download evidence bundle (ZIP).
|
||||
/// </summary>
|
||||
public string? BundleDownloadUrl { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Manifest hash for verification.
|
||||
/// </summary>
|
||||
public required string ManifestHash { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// All input hashes for determinism verification.
|
||||
/// </summary>
|
||||
public required ReplayInputHashes InputHashes { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When this command info was generated.
|
||||
/// </summary>
|
||||
public DateTimeOffset GeneratedAt { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// All input hashes that determine replay output.
|
||||
/// </summary>
|
||||
public sealed record ReplayInputHashes
|
||||
{
|
||||
/// <summary>Artifact content hash.</summary>
|
||||
public required string ArtifactDigest { get; init; }
|
||||
|
||||
/// <summary>Run manifest hash (includes all scan parameters).</summary>
|
||||
public required string ManifestHash { get; init; }
|
||||
|
||||
/// <summary>Vulnerability feed snapshot hash.</summary>
|
||||
public required string FeedSnapshotHash { get; init; }
|
||||
|
||||
/// <summary>Policy ruleset hash.</summary>
|
||||
public required string PolicyHash { get; init; }
|
||||
|
||||
/// <summary>VEX corpus hash (if applicable).</summary>
|
||||
public string? VexCorpusHash { get; init; }
|
||||
|
||||
/// <summary>Reachability model hash (if applicable).</summary>
|
||||
public string? ReachabilityModelHash { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### ReplayCommandGenerator Implementation
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Generates copy-ready CLI commands for deterministic replay.
|
||||
/// </summary>
|
||||
public class ReplayCommandGenerator : IReplayCommandGenerator
|
||||
{
|
||||
private readonly string _cliName;
|
||||
private readonly IOptions<ReplayCommandOptions> _options;
|
||||
|
||||
public ReplayCommandGenerator(IOptions<ReplayCommandOptions> options)
|
||||
{
|
||||
_options = options;
|
||||
_cliName = options.Value.CliName ?? "stella";
|
||||
}
|
||||
|
||||
public ReplayCommandInfo GenerateForFinding(FindingReplayContext context)
|
||||
{
|
||||
var fullCommand = BuildFullCommand(context);
|
||||
var shortCommand = BuildShortCommand(context);
|
||||
var bundleUrl = BuildBundleUrl(context);
|
||||
|
||||
return new ReplayCommandInfo
|
||||
{
|
||||
FullCommand = fullCommand,
|
||||
ShortCommand = shortCommand,
|
||||
BundleDownloadUrl = bundleUrl,
|
||||
ManifestHash = context.ManifestHash,
|
||||
InputHashes = new ReplayInputHashes
|
||||
{
|
||||
ArtifactDigest = context.ArtifactDigest,
|
||||
ManifestHash = context.ManifestHash,
|
||||
FeedSnapshotHash = context.FeedSnapshotHash,
|
||||
PolicyHash = context.PolicyHash
|
||||
},
|
||||
GeneratedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
}
|
||||
|
||||
private string BuildFullCommand(FindingReplayContext context)
|
||||
{
|
||||
var sb = new StringBuilder();
|
||||
sb.Append(_cliName);
|
||||
sb.Append(" scan replay");
|
||||
sb.Append($" --artifact {context.ArtifactDigest}");
|
||||
sb.Append($" --manifest {context.ManifestHash}");
|
||||
sb.Append($" --feeds {context.FeedSnapshotHash}");
|
||||
sb.Append($" --policy {context.PolicyHash}");
|
||||
|
||||
if (context.KnowledgeSnapshotId is not null)
|
||||
{
|
||||
sb.Append($" --snapshot {context.KnowledgeSnapshotId}");
|
||||
}
|
||||
|
||||
return sb.ToString();
|
||||
}
|
||||
|
||||
private string? BuildShortCommand(FindingReplayContext context)
|
||||
{
|
||||
if (context.VerdictId is null)
|
||||
return null;
|
||||
|
||||
return $"{_cliName} replay snapshot --verdict {context.VerdictId}";
|
||||
}
|
||||
|
||||
private string BuildBundleUrl(FindingReplayContext context)
|
||||
{
|
||||
return $"/v1/triage/findings/{context.FindingId}/evidence/export";
|
||||
}
|
||||
|
||||
public ReplayCommandInfo GenerateForRun(ScanRunReplayContext context)
|
||||
{
|
||||
var sb = new StringBuilder();
|
||||
sb.Append(_cliName);
|
||||
sb.Append(" scan replay");
|
||||
sb.Append($" --manifest {context.ManifestHash}");
|
||||
sb.Append($" --feeds {context.FeedSnapshotHash}");
|
||||
sb.Append($" --policy {context.PolicyHash}");
|
||||
|
||||
foreach (var artifact in context.ArtifactDigests)
|
||||
{
|
||||
sb.Append($" --artifact {artifact}");
|
||||
}
|
||||
|
||||
return new ReplayCommandInfo
|
||||
{
|
||||
FullCommand = sb.ToString(),
|
||||
ShortCommand = context.KnowledgeSnapshotId is not null
|
||||
? $"{_cliName} replay batch --snapshot {context.KnowledgeSnapshotId}"
|
||||
: null,
|
||||
BundleDownloadUrl = $"/v1/runs/{context.ScanRunId}/evidence/export",
|
||||
ManifestHash = context.ManifestHash,
|
||||
InputHashes = new ReplayInputHashes
|
||||
{
|
||||
ArtifactDigest = string.Join(",", context.ArtifactDigests),
|
||||
ManifestHash = context.ManifestHash,
|
||||
FeedSnapshotHash = context.FeedSnapshotHash,
|
||||
PolicyHash = context.PolicyHash
|
||||
},
|
||||
GeneratedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Evidence Bundle Export
|
||||
|
||||
```csharp
|
||||
// src/Scanner/StellaOps.Scanner.WebService/Services/EvidenceBundleExporter.cs
|
||||
|
||||
/// <summary>
|
||||
/// Exports evidence bundles as downloadable archives.
|
||||
/// </summary>
|
||||
public interface IEvidenceBundleExporter
|
||||
{
|
||||
/// <summary>
|
||||
/// Export evidence bundle for a finding.
|
||||
/// </summary>
|
||||
Task<Stream> ExportFindingBundleAsync(
|
||||
string findingId,
|
||||
EvidenceBundleFormat format,
|
||||
CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Export evidence bundle for a scan run.
|
||||
/// </summary>
|
||||
Task<Stream> ExportRunBundleAsync(
|
||||
string scanRunId,
|
||||
EvidenceBundleFormat format,
|
||||
CancellationToken ct = default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evidence bundle export format.
|
||||
/// </summary>
|
||||
public enum EvidenceBundleFormat
|
||||
{
|
||||
/// <summary>ZIP archive.</summary>
|
||||
Zip,
|
||||
|
||||
/// <summary>TAR.GZ archive.</summary>
|
||||
TarGz
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evidence bundle exporter implementation.
|
||||
/// </summary>
|
||||
public class EvidenceBundleExporter : IEvidenceBundleExporter
|
||||
{
|
||||
private readonly ITriageStatusService _triageService;
|
||||
private readonly IProofBundleRepository _proofBundleRepo;
|
||||
private readonly IReplayCommandGenerator _replayCommandGenerator;
|
||||
private readonly ILogger<EvidenceBundleExporter> _logger;
|
||||
|
||||
public async Task<Stream> ExportFindingBundleAsync(
|
||||
string findingId,
|
||||
EvidenceBundleFormat format,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
var ms = new MemoryStream();
|
||||
|
||||
using var archive = new ZipArchive(ms, ZipArchiveMode.Create, leaveOpen: true);
|
||||
|
||||
// 1. Add finding triage status
|
||||
var triageStatus = await _triageService.GetFindingStatusAsync(findingId, ct);
|
||||
await AddJsonEntry(archive, "finding-status.json", triageStatus);
|
||||
|
||||
// 2. Add proof bundle if available
|
||||
if (triageStatus.ProofBundleUri is not null)
|
||||
{
|
||||
var proofBundle = await _proofBundleRepo.GetAsync(triageStatus.ProofBundleUri, ct);
|
||||
if (proofBundle is not null)
|
||||
{
|
||||
await AddJsonEntry(archive, "proof-bundle.json", proofBundle);
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Add replay command
|
||||
var replayContext = BuildFindingContext(triageStatus);
|
||||
var replayCommand = _replayCommandGenerator.GenerateForFinding(replayContext);
|
||||
await AddJsonEntry(archive, "replay-command.json", replayCommand);
|
||||
|
||||
// 4. Add replay script
|
||||
await AddTextEntry(archive, "replay.sh", BuildReplayScript(replayCommand));
|
||||
await AddTextEntry(archive, "replay.ps1", BuildReplayPowerShellScript(replayCommand));
|
||||
|
||||
// 5. Add README
|
||||
await AddTextEntry(archive, "README.md", BuildReadme(findingId, replayCommand));
|
||||
|
||||
// 6. Add manifest file
|
||||
var manifest = BuildBundleManifest(findingId, replayCommand);
|
||||
await AddJsonEntry(archive, "MANIFEST.json", manifest);
|
||||
|
||||
ms.Position = 0;
|
||||
return ms;
|
||||
}
|
||||
|
||||
private static string BuildReplayScript(ReplayCommandInfo command)
|
||||
{
|
||||
return $"""
|
||||
#!/bin/bash
|
||||
# Evidence Bundle Replay Script
|
||||
# Generated: {command.GeneratedAt:u}
|
||||
|
||||
# Verify hashes before replay
|
||||
echo "Input Hashes:"
|
||||
echo " Artifact: {command.InputHashes.ArtifactDigest}"
|
||||
echo " Manifest: {command.InputHashes.ManifestHash}"
|
||||
echo " Feeds: {command.InputHashes.FeedSnapshotHash}"
|
||||
echo " Policy: {command.InputHashes.PolicyHash}"
|
||||
echo ""
|
||||
|
||||
# Run replay
|
||||
{command.FullCommand}
|
||||
""";
|
||||
}
|
||||
|
||||
private static string BuildReplayPowerShellScript(ReplayCommandInfo command)
|
||||
{
|
||||
return $"""
|
||||
# Evidence Bundle Replay Script (PowerShell)
|
||||
# Generated: {command.GeneratedAt:u}
|
||||
|
||||
Write-Host "Input Hashes:"
|
||||
Write-Host " Artifact: {command.InputHashes.ArtifactDigest}"
|
||||
Write-Host " Manifest: {command.InputHashes.ManifestHash}"
|
||||
Write-Host " Feeds: {command.InputHashes.FeedSnapshotHash}"
|
||||
Write-Host " Policy: {command.InputHashes.PolicyHash}"
|
||||
Write-Host ""
|
||||
|
||||
# Run replay
|
||||
{command.FullCommand}
|
||||
""";
|
||||
}
|
||||
|
||||
private static string BuildReadme(string findingId, ReplayCommandInfo command)
|
||||
{
|
||||
return $"""
|
||||
# Evidence Bundle
|
||||
|
||||
## Finding: {findingId}
|
||||
|
||||
This bundle contains all evidence necessary to reproduce the security verdict for this finding.
|
||||
|
||||
## Quick Replay
|
||||
|
||||
### Full Command (explicit inputs)
|
||||
```bash
|
||||
{command.FullCommand}
|
||||
```
|
||||
|
||||
### Short Command (uses verdict store)
|
||||
```bash
|
||||
{command.ShortCommand ?? "N/A - verdict ID not available"}
|
||||
```
|
||||
|
||||
## Bundle Contents
|
||||
|
||||
- `finding-status.json` - Current triage status and gating information
|
||||
- `proof-bundle.json` - Content-addressable proof bundle
|
||||
- `replay-command.json` - Machine-readable replay command
|
||||
- `replay.sh` - Bash replay script
|
||||
- `replay.ps1` - PowerShell replay script
|
||||
- `MANIFEST.json` - Bundle manifest with hashes
|
||||
|
||||
## Verification
|
||||
|
||||
All inputs are content-addressed. Replay with identical inputs produces identical verdicts.
|
||||
|
||||
| Input | Hash |
|
||||
|-------|------|
|
||||
| Artifact | `{command.InputHashes.ArtifactDigest}` |
|
||||
| Manifest | `{command.InputHashes.ManifestHash}` |
|
||||
| Feeds | `{command.InputHashes.FeedSnapshotHash}` |
|
||||
| Policy | `{command.InputHashes.PolicyHash}` |
|
||||
|
||||
---
|
||||
Generated: {command.GeneratedAt:u}
|
||||
""";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration with Unified Evidence Endpoint
|
||||
|
||||
```csharp
|
||||
// Extension to UnifiedEvidenceResponseDto from Sprint 9200.0001.0002
|
||||
|
||||
public sealed record UnifiedEvidenceResponseDto
|
||||
{
|
||||
// ... existing fields from Sprint 0002 ...
|
||||
|
||||
// === NEW: Replay Command (Sprint 9200.0001.0003) ===
|
||||
|
||||
/// <summary>
|
||||
/// Copy-ready replay command for deterministic reproduction.
|
||||
/// </summary>
|
||||
public ReplayCommandInfo? ReplayCommand { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### CLI Enhancements
|
||||
|
||||
```csharp
|
||||
// src/Cli/StellaOps.Cli/Commands/ScanReplayCommand.cs
|
||||
|
||||
// New subcommand: stella scan replay (distinct from stella replay)
|
||||
// Accepts explicit input hashes for offline replay
|
||||
|
||||
public static Command BuildScanReplayCommand(Option<bool> verboseOption, CancellationToken ct)
|
||||
{
|
||||
var artifactOption = new Option<string>("--artifact")
|
||||
{
|
||||
Description = "Artifact digest (sha256:...)",
|
||||
IsRequired = true
|
||||
};
|
||||
var manifestOption = new Option<string>("--manifest")
|
||||
{
|
||||
Description = "Run manifest hash",
|
||||
IsRequired = true
|
||||
};
|
||||
var feedsOption = new Option<string>("--feeds")
|
||||
{
|
||||
Description = "Feed snapshot hash",
|
||||
IsRequired = true
|
||||
};
|
||||
var policyOption = new Option<string>("--policy")
|
||||
{
|
||||
Description = "Policy ruleset hash",
|
||||
IsRequired = true
|
||||
};
|
||||
var snapshotOption = new Option<string?>("--snapshot")
|
||||
{
|
||||
Description = "Knowledge snapshot ID"
|
||||
};
|
||||
var offlineOption = new Option<bool>("--offline")
|
||||
{
|
||||
Description = "Run completely offline (fail if any input missing)"
|
||||
};
|
||||
var outputOption = new Option<string?>("--output")
|
||||
{
|
||||
Description = "Output verdict JSON path"
|
||||
};
|
||||
|
||||
var replayCmd = new Command("replay", "Replay scan with explicit input hashes");
|
||||
replayCmd.Add(artifactOption);
|
||||
replayCmd.Add(manifestOption);
|
||||
replayCmd.Add(feedsOption);
|
||||
replayCmd.Add(policyOption);
|
||||
replayCmd.Add(snapshotOption);
|
||||
replayCmd.Add(offlineOption);
|
||||
replayCmd.Add(outputOption);
|
||||
replayCmd.Add(verboseOption);
|
||||
|
||||
replayCmd.SetAction(async (parseResult, _) =>
|
||||
{
|
||||
var artifact = parseResult.GetValue(artifactOption) ?? string.Empty;
|
||||
var manifest = parseResult.GetValue(manifestOption) ?? string.Empty;
|
||||
var feeds = parseResult.GetValue(feedsOption) ?? string.Empty;
|
||||
var policy = parseResult.GetValue(policyOption) ?? string.Empty;
|
||||
var snapshot = parseResult.GetValue(snapshotOption);
|
||||
var offline = parseResult.GetValue(offlineOption);
|
||||
var output = parseResult.GetValue(outputOption);
|
||||
var verbose = parseResult.GetValue(verboseOption);
|
||||
|
||||
if (verbose)
|
||||
{
|
||||
Console.WriteLine("Replay Configuration:");
|
||||
Console.WriteLine($" Artifact: {artifact}");
|
||||
Console.WriteLine($" Manifest: {manifest}");
|
||||
Console.WriteLine($" Feeds: {feeds}");
|
||||
Console.WriteLine($" Policy: {policy}");
|
||||
if (snapshot is not null)
|
||||
Console.WriteLine($" Snapshot: {snapshot}");
|
||||
Console.WriteLine($" Offline: {offline}");
|
||||
}
|
||||
|
||||
// ... implementation using ReplayEngine ...
|
||||
|
||||
return 0;
|
||||
});
|
||||
|
||||
return replayCmd;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
| # | Task ID | Status | Key dependency | Owners | Task Definition |
|
||||
|---|---------|--------|----------------|--------|-----------------|
|
||||
| **Wave 0 (Contract Definitions)** | | | | | |
|
||||
| 1 | RCG-9200-001 | TODO | None | Scanner Guild | Define `IReplayCommandGenerator` interface in `Services/`. |
|
||||
| 2 | RCG-9200-002 | TODO | Task 1 | Scanner Guild | Define `FindingReplayContext` record. |
|
||||
| 3 | RCG-9200-003 | TODO | Task 1 | Scanner Guild | Define `ScanRunReplayContext` record. |
|
||||
| 4 | RCG-9200-004 | TODO | Task 1 | Scanner Guild | Define `ReplayCommandInfo` DTO. |
|
||||
| 5 | RCG-9200-005 | TODO | Task 4 | Scanner Guild | Define `ReplayInputHashes` DTO. |
|
||||
| 6 | RCG-9200-006 | TODO | Task 4 | Scanner Guild | Define `ReplayCommandOptions` configuration class. |
|
||||
| **Wave 1 (Generator Implementation)** | | | | | |
|
||||
| 7 | RCG-9200-007 | TODO | Tasks 1-6 | Scanner Guild | Implement `ReplayCommandGenerator.GenerateForFinding()`. |
|
||||
| 8 | RCG-9200-008 | TODO | Task 7 | Scanner Guild | Implement `ReplayCommandGenerator.GenerateForRun()`. |
|
||||
| 9 | RCG-9200-009 | TODO | Task 7 | Scanner Guild | Add short command generation for verdict-based replay. |
|
||||
| 10 | RCG-9200-010 | TODO | Task 7 | Scanner Guild | Wire generator into DI container. |
|
||||
| **Wave 2 (Evidence Bundle Export)** | | | | | |
|
||||
| 11 | RCG-9200-011 | TODO | Task 10 | Scanner Guild | Define `IEvidenceBundleExporter` interface. |
|
||||
| 12 | RCG-9200-012 | TODO | Task 11 | Scanner Guild | Implement `EvidenceBundleExporter.ExportFindingBundleAsync()`. |
|
||||
| 13 | RCG-9200-013 | TODO | Task 12 | Scanner Guild | Add replay script generation (bash). |
|
||||
| 14 | RCG-9200-014 | TODO | Task 12 | Scanner Guild | Add replay script generation (PowerShell). |
|
||||
| 15 | RCG-9200-015 | TODO | Task 12 | Scanner Guild | Add README generation with hash table. |
|
||||
| 16 | RCG-9200-016 | TODO | Task 12 | Scanner Guild | Add MANIFEST.json generation. |
|
||||
| 17 | RCG-9200-017 | TODO | Task 11 | Scanner Guild | Implement `EvidenceBundleExporter.ExportRunBundleAsync()`. |
|
||||
| **Wave 3 (API Endpoints)** | | | | | |
|
||||
| 18 | RCG-9200-018 | TODO | Task 12 | Scanner Guild | Add `GET /v1/triage/findings/{id}/evidence/export` endpoint. |
|
||||
| 19 | RCG-9200-019 | TODO | Task 17 | Scanner Guild | Add `GET /v1/runs/{id}/evidence/export` endpoint. |
|
||||
| 20 | RCG-9200-020 | TODO | Task 10 | Scanner Guild | Wire `ReplayCommand` into `UnifiedEvidenceResponseDto`. |
|
||||
| **Wave 4 (CLI Enhancements)** | | | | | |
|
||||
| 21 | RCG-9200-021 | TODO | None | CLI Guild | Add `stella scan replay` subcommand with explicit hashes. |
|
||||
| 22 | RCG-9200-022 | TODO | Task 21 | CLI Guild | Add `--offline` flag for air-gapped replay. |
|
||||
| 23 | RCG-9200-023 | TODO | Task 21 | CLI Guild | Add input hash verification before replay. |
|
||||
| 24 | RCG-9200-024 | TODO | Task 21 | CLI Guild | Add verbose output with hash confirmation. |
|
||||
| **Wave 5 (Tests)** | | | | | |
|
||||
| 25 | RCG-9200-025 | TODO | Task 7 | QA Guild | Add unit tests for `ReplayCommandGenerator` - all command formats. |
|
||||
| 26 | RCG-9200-026 | TODO | Task 12 | QA Guild | Add unit tests for evidence bundle generation. |
|
||||
| 27 | RCG-9200-027 | TODO | Task 18 | QA Guild | Add integration tests for export endpoints. |
|
||||
| 28 | RCG-9200-028 | TODO | Task 21 | QA Guild | Add CLI integration tests for `stella scan replay`. |
|
||||
| 29 | RCG-9200-029 | TODO | All | QA Guild | Add determinism tests: replay with exported bundle produces identical verdict. |
|
||||
| **Wave 6 (Documentation)** | | | | | |
|
||||
| 30 | RCG-9200-030 | TODO | All | Docs Guild | Update CLI reference for `stella scan replay`. |
|
||||
| 31 | RCG-9200-031 | TODO | All | Docs Guild | Add evidence bundle format specification. |
|
||||
| 32 | RCG-9200-032 | TODO | All | Docs Guild | Update API reference for export endpoints. |
|
||||
|
||||
---
|
||||
|
||||
## Wave Coordination
|
||||
|
||||
| Wave | Tasks | Focus | Evidence |
|
||||
|------|-------|-------|----------|
|
||||
| **Wave 0** | 1-6 | Contract definitions | All DTOs compile |
|
||||
| **Wave 1** | 7-10 | Generator implementation | Commands generated correctly |
|
||||
| **Wave 2** | 11-17 | Evidence bundle export | ZIP bundles contain all artifacts |
|
||||
| **Wave 3** | 18-20 | API endpoints | Endpoints return downloads |
|
||||
| **Wave 4** | 21-24 | CLI enhancements | CLI accepts explicit hashes |
|
||||
| **Wave 5** | 25-29 | Tests | All tests pass |
|
||||
| **Wave 6** | 30-32 | Documentation | Docs updated |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
# etc/scanner.yaml
|
||||
replay:
|
||||
command:
|
||||
# CLI executable name
|
||||
cli_name: stella
|
||||
|
||||
# Include snapshot ID in short command when available
|
||||
include_snapshot_shorthand: true
|
||||
|
||||
bundle:
|
||||
# Default export format
|
||||
default_format: zip
|
||||
|
||||
# Include replay scripts in bundle
|
||||
include_scripts: true
|
||||
|
||||
# Include README in bundle
|
||||
include_readme: true
|
||||
|
||||
# Maximum bundle size (MB)
|
||||
max_bundle_size_mb: 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decisions & Risks
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|----------|-----------|
|
||||
| Separate `stella scan replay` from `stella replay` | `scan replay` takes explicit hashes; `replay` uses manifest files |
|
||||
| Generate both bash and PowerShell scripts | Cross-platform support |
|
||||
| Include README with hash table | Human-readable verification |
|
||||
| Content-addressable bundle manifest | Enables bundle integrity verification |
|
||||
|
||||
### Risks
|
||||
|
||||
| Risk | Impact | Mitigation | Owner |
|
||||
|------|--------|------------|-------|
|
||||
| Large evidence bundles | Slow downloads | Stream generation; size limits | Scanner Guild |
|
||||
| Missing input artifacts | Incomplete bundle | Graceful degradation; note in README | Scanner Guild |
|
||||
| Hash format changes | Command incompatibility | Version field in command info | Scanner Guild |
|
||||
| Offline replay fails | Cannot verify | Validate all inputs present before starting | CLI Guild |
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-24 | Sprint created from Quiet-by-Design Triage gap analysis. | Project Mgmt |
|
||||
1371
docs/implplan/SPRINT_9200_0001_0004_FE_quiet_triage_ui.md
Normal file
1371
docs/implplan/SPRINT_9200_0001_0004_FE_quiet_triage_ui.md
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user