diff --git a/docs/implplan/SPRINT_0115_0001_0004_concelier_iv.md b/docs/implplan/SPRINT_0115_0001_0004_concelier_iv.md index 3cff56c4e..4594bd40a 100644 --- a/docs/implplan/SPRINT_0115_0001_0004_concelier_iv.md +++ b/docs/implplan/SPRINT_0115_0001_0004_concelier_iv.md @@ -38,11 +38,13 @@ | 11 | CONCELIER-STORE-AOC-19-005-DEV | BLOCKED (2025-11-04) | Waiting on staging dataset hash + rollback rehearsal using prep doc | Concelier Storage Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Execute raw-linkset backfill/rollback plan so Mongo reflects Link-Not-Merge data; rehearse rollback (dev/staging). | | 12 | CONCELIER-TEN-48-001 | DONE (2025-11-28) | Created Tenancy module with `TenantScope`, `TenantCapabilities`, `TenantCapabilitiesResponse`, `ITenantCapabilitiesProvider`, and `TenantScopeNormalizer` per AUTH-TEN-47-001. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Enforce tenant scoping through normalization/linking; expose capability endpoint advertising `merge=false`; ensure events include tenant IDs. | | 13 | CONCELIER-VEXLENS-30-001 | BLOCKED | PREP-CONCELIER-VULN-29-001; VEXLENS-30-005 | Concelier WebService Guild · VEX Lens Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Guarantee advisory key consistency and cross-links consumed by VEX Lens so consensus explanations cite Concelier evidence without merges. | -| 14 | CONCELIER-GAPS-115-014 | TODO | None; informs tasks 0–13. | Product Mgmt · Concelier Guild | Address Concelier ingestion gaps CI1–CI10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed observation/linkset schemas and AOC guard, enforce denylist/allowlist via analyzers, require provenance/signature details, feed snapshot governance/staleness, deterministic conflict rules, canonical content-hash/idempotency keys, tenant isolation tests, connector sandbox limits, offline advisory bundle schema/verify, and shared fixtures/CI determinism. | +| 14 | CONCELIER-GAPS-115-014 | DONE (2025-12-02) | None; informs tasks 0–13. | Product Mgmt · Concelier Guild | Address Concelier ingestion gaps CI1–CI10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed observation/linkset schemas and AOC guard, enforce denylist/allowlist via analyzers, require provenance/signature details, feed snapshot governance/staleness, deterministic conflict rules, canonical content-hash/idempotency keys, tenant isolation tests, connector sandbox limits, offline advisory bundle schema/verify, and shared fixtures/CI determinism. | ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Completed CONCELIER-GAPS-115-014: published signed LNM schemas + manifest/signature, added connector HttpClient sandbox analyzer, hardened AOC guard for canonical sha256 + signature metadata, added determinism/tenant isolation tests and offline bundle fixtures. Targeted Core tests passing. | Implementer | +| 2025-12-02 | Started CONCELIER-GAPS-115-014 remediation: schema signing, AOC provenance guard, determinism/tenant isolation tests. | Implementer | | 2025-11-28 | Completed CONCELIER-RISK-69-001: implemented `AdvisoryFieldChangeNotification`, `AdvisoryFieldChange`, `AdvisoryFieldChangeProvenance` models + `IAdvisoryFieldChangeEmitter` interface + `AdvisoryFieldChangeEmitter` implementation + `IAdvisoryFieldChangeNotificationPublisher` interface + `InMemoryAdvisoryFieldChangeNotificationPublisher`. Detects changes in fix availability, KEV status, severity, CVSS score, and observation status with full provenance. DI registration via `AddConcelierRiskServices()`. Sprint 0115 RISK tasks now complete (66-001, 66-002, 67-001, 69-001 DONE; 68-001 BLOCKED on POLICY-RISK-68-001). | Implementer | | 2025-12-01 | Added CONCELIER-GAPS-115-014 to capture CI1–CI10 remediation from `31-Nov-2025 FINDINGS.md`. | Product Mgmt | | 2025-11-28 | Completed CONCELIER-RISK-66-002: implemented `FixAvailabilityMetadata`, `FixRelease`, `FixAdvisoryLink` models with provenance anchors + `IFixAvailabilityEmitter` interface + `FixAvailabilityEmitter` implementation for emitting structured fix-availability metadata per observation/linkset. DI registration via `AddConcelierRiskServices()`. Unblocked CONCELIER-RISK-69-001. | Implementer | @@ -81,6 +83,7 @@ - Raw linkset backfill (STORE-AOC-19-005) must preserve rollback paths to protect Offline Kit deployments; release packaging tracked separately in DevOps planning. - Tenant-aware linking and notification hooks depend on Authority/Signals contracts; delays could stall AOC compliance and downstream alerts. - Upstream contracts absent: POLICY-20-001 (sprint 0114), AUTH-TEN-47-001, SIGNALS-24-002—until delivered, POLICY/RISK/SIG/TEN tasks in this sprint stay BLOCKED. +- CI1–CI10 remediation shipped: signed schema bundle (`docs/modules/concelier/schemas/*`) with detached signature, AOC guard now enforces canonical sha256 + signature metadata, connector analyzer `CONCELIER0004` guards unsandboxed `HttpClient`, and deterministic fixtures/tests cover idempotency/tenant isolation/offline bundle staleness. ## Next Checkpoints - Plan backfill rehearsal window for STORE-AOC-19-005 once AUTH/AOC prerequisites clear (date TBD). diff --git a/docs/implplan/SPRINT_0121_0001_0001_policy_reasoning.md b/docs/implplan/SPRINT_0121_0001_0001_policy_reasoning.md index 658466c24..1f9b4e26c 100644 --- a/docs/implplan/SPRINT_0121_0001_0001_policy_reasoning.md +++ b/docs/implplan/SPRINT_0121_0001_0001_policy_reasoning.md @@ -43,11 +43,13 @@ | 6 | LEDGER-OBS-54-001 | DONE (2025-11-22) | `/v1/ledger/attestations` endpoint implemented with deterministic paging + filters hash; schema/OAS updated | Findings Ledger Guild; Provenance Guild / src/Findings/StellaOps.Findings.Ledger | Verify attestation references for ledger-derived exports; expose `/ledger/attestations` endpoint returning DSSE verification state and chain-of-custody summary | | 7 | LEDGER-RISK-66-001 | DONE (2025-11-21) | PREP-LEDGER-RISK-66-001-RISK-ENGINE-SCHEMA-CO | Findings Ledger Guild; Risk Engine Guild / src/Findings/StellaOps.Findings.Ledger | Add schema migrations for `risk_score`, `risk_severity`, `profile_version`, `explanation_id`, and supporting indexes | | 8 | LEDGER-RISK-66-002 | DONE (2025-11-21) | PREP-LEDGER-RISK-66-002-DEPENDS-ON-66-001-MIG | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Implement deterministic upsert of scoring results keyed by finding hash/profile version with history audit | -| 9 | LEDGER-GAPS-121-009 | TODO | Close FL1–FL10 gaps from `docs/product-advisories/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md`; align schemas/exports with advisory; depends on schema catalog refresh | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Remediate FL1–FL10: publish versioned schemas/canonical JSON (events/projections/exports), Merkle + external anchor policy doc, tenant isolation + redaction manifest, DSSE/policy hash linkage, deterministic exports + golden fixtures, offline verifier script, replay/rebuild checksum guard, and quotas/backpressure metrics; update docs under `docs/modules/findings-ledger/`. | +| 9 | LEDGER-GAPS-121-009 | DONE (2025-12-02) | Close FL1–FL10 gaps from `docs/product-advisories/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md`; align schemas/exports with advisory; depends on schema catalog refresh | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Remediate FL1–FL10: publish versioned schemas/canonical JSON (events/projections/exports), Merkle + external anchor policy doc, tenant isolation + redaction manifest, DSSE/policy hash linkage, deterministic exports + golden fixtures, offline verifier script, replay/rebuild checksum guard, and quotas/backpressure metrics; update docs under `docs/modules/findings-ledger/`. | ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Completed LEDGER-GAPS-121-009: added schema catalog + FL1–FL10 gap report, Merkle/anchor policy, redaction manifest, DSSE linkage doc, golden export fixtures + checksums, offline verifier script with replay checksum guard, quota/backpressure metrics/code/tests. | Findings Ledger | +| 2025-12-02 | Started LEDGER-GAPS-121-009 (FL1–FL10 remediation); status → DOING; drafting schema catalog, Merkle/anchor policy, redaction manifest, offline verifier, and backpressure metrics. | Findings Ledger | | 2025-12-01 | Added LEDGER-GAPS-121-009 to track FL1–FL10 remediation from `docs/product-advisories/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md`; status TODO pending schema catalog refresh. | Project Mgmt | | 2025-12-02 | Clarified LEDGER-GAPS-121-009 outputs: schema catalog, Merkle/anchor policy, tenant isolation/redaction manifest, DSSE/policy linkage, deterministic exports + golden fixtures, offline verifier, replay checksums, and quotas/backpressure metrics. | Project Mgmt | | 2025-11-25 | Moved all remaining BLOCKED tasks (OAS, ATTEST, OBS-55, PACKS) to new sprint `SPRINT_0121_0001_0002_policy_reasoning_blockers`; cleansed Delivery Tracker to active/completed items only. | Project Mgmt | @@ -83,6 +85,7 @@ - Current state: findings export endpoint and paging contracts implemented; VEX/advisory/SBOM endpoints stubbed (auth + shape) but await underlying projection/query schemas. Risk schema/implementation (LEDGER-RISK-66-001/002) delivered. Remaining blockers: OAS/SDK surface (61/62/63), attestation HTTP host (OBS-54/55), and packs time-travel contract (PACKS-42-001). - Export endpoints now enforce filter hash + page token determinism for VEX/advisory/SBOMs but still return empty sets until backing projections land; downstream SDK/OAS tasks should treat payload shapes as stable. - New advisory gaps (FL1–FL10) tracked via LEDGER-GAPS-121-009; requires schema catalog refresh and alignment of Merkle/anchoring, redaction, DSSE linkage, and offline verify tooling with `docs/product-advisories/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md` recommendations. +- FL1–FL10 remediation shipped: schema catalog + gap report, Merkle/anchor policy, redaction manifest (JSON/YAML), DSSE linkage guidance, golden export fixtures/checksums, offline verify script with replay checksum guard, and quota/backpressure metrics/tests wired into ledger service. ## Next Checkpoints - Schedule cross-guild kickoff for week of 2025-11-24 once dependency clears. diff --git a/docs/implplan/SPRINT_0124_0001_0001_policy_reasoning.md b/docs/implplan/SPRINT_0124_0001_0001_policy_reasoning.md index f5cc4995a..e468b9dec 100644 --- a/docs/implplan/SPRINT_0124_0001_0001_policy_reasoning.md +++ b/docs/implplan/SPRINT_0124_0001_0001_policy_reasoning.md @@ -15,11 +15,19 @@ - `docs/modules/platform/architecture-overview.md` - `docs/modules/policy/architecture.md` +## Interlocks +- POLICY-CONSOLE-23-001 (Console export/simulation contract from BE-Base Platform) must be published before POLICY-CONSOLE-23-002 can start. + +## Action Tracker +| # | Action | Owner | Due | Status | +| --- | --- | --- | --- | --- | +| 1 | Publish Console export/simulation contract for POLICY-CONSOLE-23-001 to unblock POLICY-CONSOLE-23-002 | BE-Base Platform Guild | — | BLOCKED (awaiting spec) | + ## Delivery Tracker | # | Task ID & handle | State | Key dependency / next step | Owners | | --- | --- | --- | --- | --- | | P1 | PREP-POLICY-ENGINE-20-002-DETERMINISTIC-EVALU | DONE (2025-11-22) | Due 2025-11-22 · Accountable: Policy Guild / `src/Policy/StellaOps.Policy.Engine` | Policy Guild / `src/Policy/StellaOps.Policy.Engine` | Deterministic evaluator spec missing.

Document artefact/deliverable for POLICY-ENGINE-20-002 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/policy/design/policy-deterministic-evaluator.md`. | -| 1 | POLICY-CONSOLE-23-002 | BLOCKED (2025-11-27) | Waiting on POLICY-CONSOLE-23-001 export/simulation contract. | Policy Guild, Product Ops / `src/Policy/StellaOps.Policy.Engine` | +| 1 | POLICY-CONSOLE-23-002 | BLOCKED (2025-12-02) | POLICY-CONSOLE-23-001 export/simulation contract still not published; waiting on Console API spec from BE-Base Platform. | Policy Guild, Product Ops / `src/Policy/StellaOps.Policy.Engine` | | 2 | POLICY-ENGINE-20-002 | DONE (2025-11-27) | PREP-POLICY-ENGINE-20-002-DETERMINISTIC-EVALU | Policy Guild / `src/Policy/StellaOps.Policy.Engine` | | 3 | POLICY-ENGINE-20-003 | DONE (2025-11-27) | Depends on 20-002. | Policy · Concelier · Excititor Guilds / `src/Policy/StellaOps.Policy.Engine` | | 4 | POLICY-ENGINE-20-004 | DONE (2025-11-27) | Depends on 20-003. | Policy · Platform Storage Guild / `src/Policy/StellaOps.Policy.Engine` | @@ -36,6 +44,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Rechecked for POLICY-CONSOLE-23-001 contract; none found. Left POLICY-CONSOLE-23-002 BLOCKED (2025-12-02). Added Interlocks and Action Tracker sections to align with sprint template. | Project Mgmt | | 2025-12-01 | Refactored Mongo exception listing to shared filter/sort helpers (per-tenant and cross-tenant) for lifecycle scans; reran `dotnet test src/Policy/__Tests/StellaOps.Policy.Engine.Tests -c Release --no-build` (208/208 pass). | Implementer | | 2025-12-01 | Completed deterministic evidence summary (big-endian hash → `2025-12-13T05:00:11Z`) and exception lifecycle fixes (multi-tenant activation/expiry, no default tenant); added cross-tenant list overload. `dotnet test src/Policy/__Tests/StellaOps.Policy.Engine.Tests -c Release --no-build` now passes (208 tests, 0 failures). | Implementer | | 2025-12-01 | Ran `dotnet build src/Policy/StellaOps.Policy.Engine/StellaOps.Policy.Engine.csproj -c Release` successfully (1 warning NU1510). Attempted `dotnet test ...Policy.Engine.Tests` but cancelled mid-run due to prolonged dependency compilation; rerun still needed. | Implementer | @@ -52,7 +61,7 @@ | 2025-11-22 | Marked all PREP tasks to DONE per directive; evidence to be verified. | Project Mgmt | ## Decisions & Risks -- Console simulation/export contract (POLICY-CONSOLE-23-001) still outstanding; POLICY-CONSOLE-23-002 remains BLOCKED until published. +- 2025-12-02: Console export/simulation contract (POLICY-CONSOLE-23-001) still outstanding; POLICY-CONSOLE-23-002 remains BLOCKED until BE-Base Platform publishes the spec. - Release test suite for Policy Engine now green (2025-12-01); keep enforcing deterministic inputs (explicit evaluationTimestamp) on batch evaluation requests to avoid non-deterministic clocks. ## Next Checkpoints diff --git a/docs/implplan/SPRINT_0125_0001_0001_mirror.md b/docs/implplan/SPRINT_0125_0001_0001_mirror.md index 1d9c51dd3..5924fe6ff 100644 --- a/docs/implplan/SPRINT_0125_0001_0001_mirror.md +++ b/docs/implplan/SPRINT_0125_0001_0001_mirror.md @@ -33,9 +33,9 @@ | 8 | AIRGAP-TIME-57-001 | BLOCKED | MIRROR-CRT-56-001 sample exists; needs DSSE/TUF + time-anchor schema from AirGap Time. | AirGap Time Guild | Provide trusted time-anchor service & policy. | | 9 | CLI-AIRGAP-56-001 | BLOCKED | MIRROR-CRT-56-002/58-001 pending; offline kit inputs unavailable. | CLI Guild | Extend CLI offline kit tooling to consume mirror bundles. | | 10 | PROV-OBS-53-001 | DONE (2025-11-23) | Observer doc + verifier script `scripts/mirror/verify_thin_bundle.py` in repo; validates hashes, determinism, and manifest/index digests. | Security Guild | Define provenance observers + verification hooks. | -| 11 | OFFKIT-GAPS-125-011 | TODO | None; informs tasks 4–9. | Product Mgmt · Mirror/AirGap Guilds | Address offline-kit gaps OK1–OK10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: key manifest/rotation + PQ co-sign, tool hashing/signing, DSSE-signed top-level manifest linking all artifacts, checkpoint freshness/mirror metadata, deterministic packaging flags, inclusion of scan/VEX/policy/graph hashes, time anchor bundling, transport/chunking + chain-of-custody, tenant/env scoping, and scripted verify with negative-path guidance. | -| 12 | REKOR-GAPS-125-012 | TODO | None; informs tasks 1–10. | Product Mgmt · Mirror/AirGap · Attestor Guilds | Address Rekor v2/DSSE gaps RK1–RK10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: enforce dsse/hashedrekord only, payload size preflight + chunk manifests, public/private routing policy, shard-aware checkpoints, idempotent submission keys, Sigstore bundles in kits, checkpoint freshness bounds, PQ dual-sign options, error taxonomy/backoff, policy/graph annotations in DSSE/bundles. | -| 13 | MIRROR-GAPS-125-013 | TODO | None; informs tasks 1–12. | Product Mgmt · Mirror Creator Guild · AirGap Guild | Address mirror/offline strategy gaps MS1–MS10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: signed/versioned mirror schemas, DSSE/TUF rotation policy (incl. PQ), delta spec with tombstones/base hash, time-anchor freshness enforcement, tenant/env scoping, distribution integrity for HTTP/OCI/object, chunking/size rules, standard verify script, metrics/alerts for build/import/verify, and SemVer/change log for mirror formats. | +| 11 | OFFKIT-GAPS-125-011 | DONE (2025-12-02) | Bundle meta + offline policy layers + verifier updated; see milestone.json and bundle DSSE. | Product Mgmt · Mirror/AirGap Guilds | Address offline-kit gaps OK1–OK10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: key manifest/rotation + PQ co-sign, tool hashing/signing, DSSE-signed top-level manifest linking all artifacts, checkpoint freshness/mirror metadata, deterministic packaging flags, inclusion of scan/VEX/policy/graph hashes, time anchor bundling, transport/chunking + chain-of-custody, tenant/env scoping, and scripted verify with negative-path guidance. | +| 12 | REKOR-GAPS-125-012 | DONE (2025-12-02) | Rekor policy layer + bundle meta/TUF DSSE; refer to `layers/rekor-policy.json`. | Product Mgmt · Mirror/AirGap · Attestor Guilds | Address Rekor v2/DSSE gaps RK1–RK10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: enforce dsse/hashedrekord only, payload size preflight + chunk manifests, public/private routing policy, shard-aware checkpoints, idempotent submission keys, Sigstore bundles in kits, checkpoint freshness bounds, PQ dual-sign options, error taxonomy/backoff, policy/graph annotations in DSSE/bundles. | +| 13 | MIRROR-GAPS-125-013 | DONE (2025-12-02) | Mirror policy layer + tenant/env scope + verifier; see mirror-policy.json & bundle meta. | Product Mgmt · Mirror Creator Guild · AirGap Guild | Address mirror/offline strategy gaps MS1–MS10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: signed/versioned mirror schemas, DSSE/TUF rotation policy (incl. PQ), delta spec with tombstones/base hash, time-anchor freshness enforcement, tenant/env scoping, distribution integrity for HTTP/OCI/object, chunking/size rules, standard verify script, metrics/alerts for build/import/verify, and SemVer/change log for mirror formats. | ## Execution Log | Date (UTC) | Update | Owner | @@ -73,12 +73,15 @@ | 2025-12-01 | Added OFFKIT-GAPS-125-011 to track OK1–OK10 remediation from `31-Nov-2025 FINDINGS.md`. | Product Mgmt | | 2025-12-01 | Added REKOR-GAPS-125-012 to track RK1–RK10 remediation from `31-Nov-2025 FINDINGS.md`. | Product Mgmt | | 2025-12-01 | Added MIRROR-GAPS-125-013 to track MS1–MS10 remediation from `31-Nov-2025 FINDINGS.md`. | Product Mgmt | +| 2025-12-02 | Moved OFFKIT/REKOR/MIRROR gap tasks to DOING; created `src/Mirror/StellaOps.Mirror.Creator/TASKS.md` for local tracking and began bundle meta/policy implementation. | Implementer | +| 2025-12-02 | Completed OK/RK/MS gap remediation: added policy layers (transport/rekor/mirror/offline), bundle meta + DSSE, verifier scope/DSSE/tool-hash checks, and refreshed milestone hashes via `scripts/mirror/ci-sign.sh`. | Implementer | ## Decisions & Risks - **Decisions** - Assign primary engineer for MIRROR-CRT-56-001 (due 2025-11-17 EOD). Owners: Mirror Creator Guild · Exporter Guild; Security as backup. Option A selected: thin bundle v1; acceptance: names recorded in Delivery Tracker + kickoff notes. - Confirm DSSE/TUF signing profile (due 2025-11-18). Owners: Security Guild · Attestor Guild. Needed before MIRROR-CRT-56-002 can merge. - Lock time-anchor authority scope (due 2025-11-19). Owners: AirGap Time Guild · Mirror Creator Guild. Required for MIRROR-CRT-57-002 policy enforcement. + - 2025-12-02: OK/RK/MS gap baseline adopted — bundle meta DSSE (`mirror-thin-v1.bundle.dsse.json`) and policy layers (transport, rekor, mirror, offline-kit) are now canonical evidence; verifier enforces tenant/env scope + tool hashes. - **Risks** - Production signing key lives in Ops sprint: release signing (`MIRROR_SIGN_KEY_B64` secret + CI promotion) is handled in Sprint 506 (Ops DevOps IV); this dev sprint remains green using dev key until ops wiring lands. - Time-anchor requirements undefined → air-gapped bundles lose verifiable time guarantees. Mitigation: run focused session with AirGap Time Guild to lock policy + service interface. diff --git a/docs/implplan/SPRINT_0126_0001_0001_policy_reasoning.md b/docs/implplan/SPRINT_0126_0001_0001_policy_reasoning.md index 7dc3a9be2..523ac0ba3 100644 --- a/docs/implplan/SPRINT_0126_0001_0001_policy_reasoning.md +++ b/docs/implplan/SPRINT_0126_0001_0001_policy_reasoning.md @@ -32,7 +32,7 @@ | 13 | POLICY-ENGINE-70-004 | DONE (2025-12-01) | Depends on 70-003. | Policy · Observability Guild / `src/Policy/StellaOps.Policy.Engine` | Exception metrics/tracing/logging. | | 14 | POLICY-ENGINE-70-005 | DONE (2025-12-01) | Depends on 70-004. | Policy · Scheduler Worker Guild / `src/Policy/StellaOps.Policy.Engine` | Exception activation/expiry + events. | | 15 | POLICY-ENGINE-80-001 | DONE (2025-12-01) | Depends on 70-005. | Policy · Signals Guild / `src/Policy/StellaOps.Policy.Engine` | Reachability/exploitability inputs into evaluation. | -| 16 | POLICY-RISK-90-001 | BLOCKED (2025-12-01) | Waiting on Scanner entropy/trust algebra contract. | Policy · Scanner Guild / `src/Policy/StellaOps.Policy.Engine` | Entropy penalty ingestion + trust algebra. | +| 16 | POLICY-RISK-90-001 | DONE (2025-12-02) | Entropy ingestion implemented; monitor scanner payloads + thresholds. | Policy · Scanner Guild / `src/Policy/StellaOps.Policy.Engine` | Entropy penalty ingestion + trust algebra. | ## Execution Log | Date (UTC) | Update | Owner | @@ -61,15 +61,16 @@ | 2025-12-01 | POLICY-ENGINE-80-001 marked BLOCKED: reachability/exploitability input contract from Signals guild not yet published; no schema to integrate. | Implementer | | 2025-12-01 | POLICY-RISK-90-001 marked BLOCKED: Scanner entropy/trust algebra contract still pending; ingestion shape unknown. | Implementer | | 2025-12-01 | POLICY-ENGINE-80-001 delivered: runtime evaluation now auto-enriches reachability from facts store with overlay cache; batch lookups dedupe per tenant; cache keys include reachability metadata; added reachability-driven rule test. Targeted policy-engine test slice attempted; build fanned out and was aborted—rerun on clean policy-only graph recommended. | Implementer | +| 2025-12-02 | POLICY-RISK-90-001 delivered: added entropy penalty calculator consuming `layer_summary.json`/`entropy.report.json`, configurable caps/thresholds under `PolicyEngine:Entropy`, telemetry (`policy_entropy_penalty_value`, `policy_entropy_image_opaque_ratio`), and unit tests (`EntropyPenaltyCalculatorTests`). Unblocked Scanner dependency based on documented schema. | Implementer | ## Decisions & Risks -- Remaining TODO: POLICY-RISK-90-001 (entropy/trust algebra ingestion) still depends on Scanner contract. +- Entropy penalties now computed inside Policy Engine (`PolicyEngine:Entropy` options; default K=0.5, cap=0.3, block at image opaque ratio >0.15 when provenance is unknown). Telemetry exported as `policy_entropy_penalty_value` and `policy_entropy_image_opaque_ratio`; explanations surface top opaque files. - Reachability auto-enrichment landed (POLICY-ENGINE-80-001); exploitability signal format still absent—wire once Signals publishes contract. - Exception lifecycle now auto-activates/auto-expires; configure `ExceptionLifecycle` intervals per deployment and provide Redis if using distributed cache (in-memory defaults remain for offline use). - In-memory exception repository is registered by default for offline runs; swap to Mongo repository in production to persist lifecycle and review history. - Telemetry for exception applications added; dashboards should consume `policy_exception_applications_total`, `policy_exception_application_latency_seconds`, and `policy_exception_lifecycle_total`. - Graph-disabled test slices remain recommended (`DOTNET_DISABLE_BUILTIN_GRAPH=1`) to avoid static graph fan-out during focused test runs. ## Next Checkpoints -- Await Signals reachability/exploitability contract, then implement POLICY-ENGINE-80-001 (evaluation inputs + metrics). -- Await Scanner entropy/trust algebra contract, then implement POLICY-RISK-90-001 (ingestion + trust weighting + telemetry). -- Mirror exception lifecycle/observability changes into `docs/modules/policy/architecture.md` and dashboards. + - Await Signals reachability/exploitability contract, then refine POLICY-ENGINE-80-001 metrics once schema lands. + - Validate entropy penalty outputs against the next Scanner bundle drop; tune `PolicyEngine:Entropy` defaults if ratios shift. + - Mirror exception lifecycle/observability changes into `docs/modules/policy/architecture.md` and dashboards. diff --git a/docs/implplan/SPRINT_0136_0001_0001_scanner_surface.md b/docs/implplan/SPRINT_0136_0001_0001_scanner_surface.md index b47a0ece2..dc7390c86 100644 --- a/docs/implplan/SPRINT_0136_0001_0001_scanner_surface.md +++ b/docs/implplan/SPRINT_0136_0001_0001_scanner_surface.md @@ -1,13 +1,13 @@ # Sprint 0136-0001-0001 · Scanner & Surface (Phase VII) ## Topic & Scope -- Scanner & Surface phase VII: EntryTrace NDJSON export, process-tree replay, and surface/CLI integration. +- Scanner & Surface phase VII: EntryTrace NDJSON/replay surfacing, deterministic SBOM composition, Surface.FS/Env/Secrets rollout, and downstream consumers (Scheduler, Zastava, Cartographer, Console) enablement. - Sequential across 130–139; start after Sprint 0135. -- **Working directory:** `src/Scanner`. +- **Working directory:** `src/Scanner` (with coordinated touches in Scheduler/Zastava where noted). ## Dependencies & Concurrency - Upstream: Sprint 0135 (phase VI) must land first. -- Concurrency: tasks are TODO; follow order below. +- Concurrency: honour dependency column; SCANNER-SURFACE/EMIT work must finish before downstream consumers pick up DSSE artifacts. ## Documentation Prerequisites - docs/README.md @@ -19,38 +19,117 @@ ## Delivery Tracker | # | Task ID | Status | Key dependency / next step | Owners | Task Definition | | --- | --- | --- | --- | --- | --- | -| 0 | SURFACE-FS-01 | DONE (2025-11-24) | Spec published in `docs/modules/scanner/design/surface-fs.md` v1.1 | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Author Surface.FS cache/manifest specification and cross-module contract (manifests, CAS URIs, cache layout). | -| 1 | SURFACE-FS-02 | DONE (2025-11-24) | Core library implemented; see `src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS` | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Ship FileSurfaceManifestStore/Reader/Writer + cache options, deterministic path builder, and DI registration per `surface-fs.md`. | -| 2 | SCANNER-ENTRYTRACE-18-504 | DONE | Upstream 18-503 delivered; NDJSON emission implemented in worker and surfaced via manifest/CLI/WebService. | EntryTrace Guild (`src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace`) | Emit EntryTrace AOC NDJSON (`entrytrace.entry/node/edge/target/warning/capability`) and wire CLI/service streaming outputs. | -| 3 | SCANNER-ENTRYTRACE-18-505 | DONE | Replay implemented; uses `/proc` snapshots to adjust confidence, collapse wrappers, and emit match/mismatch diagnostics with runtime chains. | EntryTrace Guild | Implement ProcGraph replay to reconcile `/proc` exec chains with static EntryTrace, collapsing wrappers and emitting agreement/conflict diagnostics. | -| 4 | SCANNER-ENTRYTRACE-18-506 | DONE (2025-12-01) | Surfaced via WebService `/scans/{id}/entrytrace` and CLI rendering. | EntryTrace Guild · Scanner WebService Guild | Surface EntryTrace graph + confidence via Scanner.WebService and CLI, including target summary in scan reports and policy payloads. | -| 5 | ZASTAVA-SURFACE-02 | DONE (2025-12-01) | Manifest CAS/sha resolver in Observer drift evidence with failure metrics. | Zastava Observer Guild (`src/Zastava/StellaOps.Zastava.Observer`) | SURFACE-FS-02, ZASTAVA-SURFACE-01; see `docs/modules/scanner/design/surface-fs-consumers.md` §4 | -| 6 | SCANNER-SORT-02 | DONE (2025-12-01) | Layer fragment ordering by digest implemented; deterministic regression test added. | Scanner Core Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Core`) | SCANNER-EMIT-15-001 | -| 7 | SCANNER-EMIT-15-001 | DOING (2025-12-01) | CycloneDX artifacts now carry content hash + merkle root and recipe placeholders; DSSE/recipe persistence pending. | Scanner Emit Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Emit`) | SCANNER-SURFACE-04 | -| 8 | SCANNER-SURFACE-01 | BLOCKED (2025-11-25) | Task definition absent; needs scope/contract before implementation. | Scanner Guild | — | +| 1 | SURFACE-FS-01 | DONE (2025-11-24) | Spec published in `docs/modules/scanner/design/surface-fs.md` v1.1 | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Author Surface.FS cache/manifest specification and cross-module contract (manifests, CAS URIs, cache layout). | +| 2 | SURFACE-FS-02 | DONE (2025-11-24) | Core library implemented | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Ship FileSurfaceManifestStore/Reader/Writer + cache options, deterministic path builder, and DI registration per `surface-fs.md`. | +| 3 | SCANNER-ENTRYTRACE-18-504 | DONE | Depends on 18-503 | EntryTrace Guild (`src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace`) | Emit EntryTrace AOC NDJSON (`entrytrace.entry/node/edge/target/warning/capability`) and wire CLI/service streaming outputs. | +| 4 | SCANNER-ENTRYTRACE-18-505 | DONE | SCANNER-ENTRYTRACE-18-504 | EntryTrace Guild | Implement ProcGraph replay to reconcile `/proc` exec chains with static EntryTrace, collapsing wrappers and emitting diagnostics. | +| 5 | SCANNER-ENTRYTRACE-18-506 | DONE (2025-12-01) | SCANNER-ENTRYTRACE-18-505 | EntryTrace Guild · Scanner WebService Guild | Surface EntryTrace graph + confidence via Scanner.WebService and CLI, including target summary in scan reports and policy payloads. | +| 6 | SCANNER-ENV-01 | DONE (2025-11-18) | — | Scanner Worker Guild (`src/Scanner/StellaOps.Scanner.Worker`) | Wire worker to `AddSurfaceEnvironment`/`ISurfaceEnvironment` for cache roots + CAS endpoints; remove ad-hoc env reads. | +| 7 | SCANNER-ENV-02 | DONE (2025-11-27) | SCANNER-ENV-01 | Scanner WebService Guild, Ops Guild (`src/Scanner/StellaOps.Scanner.WebService`) | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration. | +| 8 | SCANNER-ENV-03 | DONE (2025-11-27) | SCANNER-ENV-02 | BuildX Plugin Guild (`src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin`) | Pack Surface.Env, mirror to offline (`offline/packages/nugets`), and wire BuildX to use 0.1.0-alpha.20251123 with updated restore feeds. | +| 9 | SURFACE-ENV-01 | DONE (2025-11-13) | — | Scanner Guild, Zastava Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env`) | Draft `surface-env.md` enumerating environment variables, defaults, and air-gap behaviour for Surface consumers. | +| 10 | SURFACE-ENV-02 | DONE (2025-11-18) | SURFACE-ENV-01 | Scanner Guild | Implement strongly-typed env accessors with validation for required endpoint, bounds, TLS cert path; add regression tests. | +| 11 | SURFACE-ENV-03 | DONE (2025-11-27) | SURFACE-ENV-02 | Scanner Guild | Adopt env helper across Scanner Worker/WebService/BuildX plug-ins. | +| 12 | SURFACE-ENV-04 | DONE (2025-11-27) | SURFACE-ENV-02 | Zastava Guild | Wire env helper into Zastava Observer/Webhook containers. | +| 13 | SURFACE-ENV-05 | DONE | SURFACE-ENV-03, SURFACE-ENV-04 | Ops Guild | Update Helm/Compose/offline kit templates with new env knobs and documentation. | +| 14 | SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Orchestrator envelope contract; Notifier ingestion tests | Scanner WebService Guild | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | +| 15 | SCANNER-GRAPH-21-001 | DONE (2025-11-27) | — | Scanner WebService Guild, Cartographer Guild (`src/Scanner/StellaOps.Scanner.WebService`) | Provide webhook/REST endpoint for Cartographer to request policy overlays and runtime evidence for graph nodes, ensuring determinism and tenant scoping. | +| 16 | SCANNER-LNM-21-001 | BLOCKED (2025-11-27) | Needs Concelier HTTP client/shared library | Scanner WebService Guild, Policy Guild | Update `/reports` and `/policy/runtime` payloads to consume advisory/vex linksets, exposing source severity arrays and conflict summaries alongside effective verdicts. | +| 17 | SCANNER-LNM-21-002 | TODO | SCANNER-LNM-21-001 | Scanner WebService Guild, UI Guild | Add evidence endpoint for Console to fetch linkset summaries with policy overlay for a component/SBOM, including AOC references. | +| 18 | SCANNER-SECRETS-03 | DONE (2025-11-27) | SCANNER-SECRETS-02 | BuildX Plugin Guild, Security Guild (`src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin`) | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. | +| 19 | SURFACE-SECRETS-01 | DONE (2025-11-23) | — | Scanner Guild, Security Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets`) | Security-approved schema published at `docs/modules/scanner/design/surface-secrets-schema.md`. | +| 20 | SURFACE-SECRETS-02 | DONE (2025-11-23) | SURFACE-SECRETS-01 | Scanner Guild | Provider chain implemented (primary + fallback) with DI wiring; tests updated (`StellaOps.Scanner.Surface.Secrets.Tests`). | +| 21 | SURFACE-SECRETS-03 | DONE (2025-11-27) | SURFACE-SECRETS-02 | Scanner Guild | Add Kubernetes/File/Offline backends with deterministic caching and audit hooks. | +| 22 | SURFACE-SECRETS-04 | DONE (2025-11-27) | SURFACE-SECRETS-02 | Scanner Guild | Integrate Surface.Secrets into Scanner Worker/WebService/BuildX for registry + CAS creds. | +| 23 | SURFACE-SECRETS-05 | DONE (2025-11-27) | SURFACE-SECRETS-02 | Zastava Guild | Invoke Surface.Secrets from Zastava Observer/Webhook for CAS & attestation secrets. | +| 24 | SURFACE-SECRETS-06 | BLOCKED (2025-11-27) | SURFACE-SECRETS-03; awaiting Ops Helm/Compose patterns | Ops Guild | Update deployment manifests/offline kit bundles to provision secret references instead of raw values. | +| 25 | SCANNER-ENG-0020 | DONE (2025-11-28) | — | Scanner Guild (`docs/modules/scanner`) | Implement Homebrew collector & fragment mapper per `design/macos-analyzer.md` §3.1. | +| 26 | SCANNER-ENG-0021 | DONE (2025-11-28) | — | Scanner Guild | Implement pkgutil receipt collector per `design/macos-analyzer.md` §3.2. | +| 27 | SCANNER-ENG-0022 | DONE (2025-11-28) | — | Scanner Guild, Policy Guild | Implement macOS bundle inspector & capability overlays per `design/macos-analyzer.md` §3.3. | +| 28 | SCANNER-ENG-0023 | DONE (2025-11-28) | — | Scanner Guild, Offline Kit Guild, Policy Guild | Deliver macOS policy/offline integration per `design/macos-analyzer.md` §5–6. | +| 29 | SCANNER-ENG-0024 | DONE (2025-11-28) | — | Scanner Guild | Implement Windows MSI collector per `design/windows-analyzer.md` §3.1. | +| 30 | SCANNER-ENG-0025 | DONE (2025-11-28) | — | Scanner Guild | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | +| 31 | SCANNER-ENG-0026 | DONE (2025-11-28) | — | Scanner Guild | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | +| 32 | SCANNER-ENG-0027 | DONE (2025-11-28) | — | Scanner Guild, Policy Guild, Offline Kit Guild | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | +| 33 | SCHED-SURFACE-02 | TODO | SURFACE-FS-02; SCHED-SURFACE-01; see `docs/modules/scanner/design/surface-fs-consumers.md` §3 | Scheduler Worker Guild (`src/Scheduler/__Libraries/StellaOps.Scheduler.Worker`) | Integrate Scheduler worker prefetch using Surface manifest reader and persist manifest pointers with rerun plans. | +| 34 | ZASTAVA-SURFACE-02 | DONE (2025-12-01) | SURFACE-FS-02, ZASTAVA-SURFACE-01 | Zastava Observer Guild (`src/Zastava/StellaOps.Zastava.Observer`) | Surface manifest CAS/sha resolver wired into Observer drift evidence with failure metrics. | +| 35 | SURFACE-FS-03 | DONE (2025-11-27) | SURFACE-FS-02 | Scanner Guild | Integrate Surface.FS writer into Scanner Worker analyzer pipeline to persist layer + entry-trace fragments. | +| 36 | SURFACE-FS-04 | DONE (2025-11-27) | SURFACE-FS-02 | Zastava Guild | Integrate Surface.FS reader into Zastava Observer runtime drift loop. | +| 37 | SURFACE-FS-05 | DONE (2025-11-27) | SURFACE-FS-03 | Scanner Guild, Scheduler Guild | Expose Surface.FS pointers via Scanner WebService reports and coordinate rescan planning with Scheduler. | +| 38 | SURFACE-FS-06 | DONE (2025-11-28) | SURFACE-FS-02..05 | Docs Guild | Update scanner-engine guide and offline kit docs with Surface.FS workflow. | +| 39 | SCANNER-SURFACE-01 | BLOCKED (2025-11-25) | Task definition absent | Scanner Guild | Placeholder task; scope/contract required before implementation. | +| 40 | SCANNER-SURFACE-04 | DONE (2025-12-02) | SCANNER-SURFACE-01, SURFACE-FS-03 | Scanner Worker Guild (`src/Scanner/StellaOps.Scanner.Worker`) | DSSE-sign every `layer.fragments` payload, emit `_composition.json`/`composition.recipe` URI, and persist DSSE envelopes for deterministic offline replay (see `deterministic-sbom-compose.md` §2.1). | +| 41 | SURFACE-FS-07 | TODO | SCANNER-SURFACE-04 | Scanner Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS`) | Extend Surface.FS manifest schema with `composition.recipe`, fragment attestation metadata, and verification helpers per deterministic SBOM spec (legacy TODO; superseded by row 42). | +| 42 | SURFACE-FS-07 | DONE (2025-12-02) | SCANNER-SURFACE-04 | Scanner Guild | Surface.FS manifest schema carries composition recipe/DSSE attestations and determinism metadata; determinism verifier added for offline replay. | +| 43 | SCANNER-EMIT-15-001 | DOING (2025-12-01) | SCANNER-SURFACE-04 | Scanner Emit Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Emit`) | CycloneDX artifacts carry content hash + Merkle root (= recipe hash), composition recipe URI, emit `_composition.json` + DSSE envelopes; replace deterministic-local signer with real signing. | +| 44 | SCANNER-SORT-02 | DONE (2025-12-01) | SCANNER-EMIT-15-001 | Scanner Core Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Core`) | Layer fragment ordering by digest implemented in ComponentGraphBuilder; determinism regression test added. | +| 45 | SURFACE-VAL-01 | DONE (2025-11-23) | SURFACE-FS-01, SURFACE-ENV-01 | Scanner Guild, Security Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation`) | Validation framework doc aligned with Surface.Env release and secrets schema (`surface-validation.md` v1.1). | +| 46 | SURFACE-VAL-02 | DONE (2025-11-23) | SURFACE-VAL-01, SURFACE-ENV-02, SURFACE-FS-02 | Scanner Guild | Validation library enforces secrets schema, fallback/provider checks, and inline/file guardrails; tests added. | +| 47 | SURFACE-VAL-03 | DONE (2025-11-23) | SURFACE-VAL-02 | Scanner Guild, Analyzer Guild | Validation runner wired into Worker/WebService startup and pre-analyzer paths (OS, language, EntryTrace). | +| 48 | SURFACE-VAL-04 | DONE (2025-11-27) | SURFACE-VAL-02 | Scanner Guild, Zastava Guild | Expose validation helpers to Zastava and other runtime consumers for preflight checks. | +| 49 | SURFACE-VAL-05 | DONE | SURFACE-VAL-02 | Docs Guild | Document validation extensibility, registration, and customization in scanner-engine guides. | ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | -| 2025-11-08 | Sprint stub created; awaiting completion of Sprint 0135. | Planning | -| 2025-11-19 | Normalized sprint to standard template and renamed from `SPRINT_136_scanner_surface.md` to `SPRINT_0136_0001_0001_scanner_surface.md`; content preserved. | Implementer | -| 2025-11-19 | Converted legacy filename `SPRINT_136_scanner_surface.md` to redirect stub pointing here to avoid divergent updates. | Implementer | -| 2025-11-24 | Marked SURFACE-FS-01 DONE; spec anchored in `docs/modules/scanner/design/surface-fs.md` v1.1. | Scanner Guild | -| 2025-11-24 | Marked SURFACE-FS-02 DONE; core Surface.FS manifest/cache library implemented and DI-ready. | Scanner Guild | -| 2025-11-25 | Marked EntryTrace chain (18-504/505/506) BLOCKED pending upstream 18-503 outputs from prior phase. | Project Mgmt | -| 2025-11-25 | Added SCANNER-SURFACE-01 to tracker and marked BLOCKED because task definition/scope is missing from sprint/docs; needs contract before work can begin. | Project Mgmt | -| 2025-12-01 | Unblocked EntryTrace NDJSON track: 18-504 set to TODO after 18-503 delivered in Sprint 0135; 18-505/506 remain blocked on 504 completion. | Project Mgmt | -| 2025-12-01 | Completed 18-504: EntryTrace NDJSON emitted via worker (EntryTraceNdjsonWriter) and surfaced in SurfaceManifest payloads; CLI/WebService entrytrace endpoint returns NDJSON alongside graph. | Implementer | -| 2025-12-01 | Completed 18-505: ProcGraph replay reconciles `/proc` snapshot with static EntryTrace, collapsing wrappers and emitting runtime match/mismatch diagnostics with chains; confidence adjusted per runtime evidence. | Implementer | -| 2025-12-01 | Added best-terminal metadata to entrytrace graph/ndjson surface payloads; SurfaceManifestStageExecutor tests updated and passing. | Implementer | -| 2025-12-01 | Completed 18-506: WebService `/scans/{id}/entrytrace` and CLI rendering now expose EntryTrace graph + confidence summaries alongside NDJSON stream. | Implementer | +| 2025-12-02 | Merged legacy `SPRINT_136_scanner_surface.md` content into canonical file; added missing tasks/logs; converted legacy file to stub to prevent divergence. | Project Mgmt | +| 2025-12-02 | SCANNER-SURFACE-04 completed: manifest stage emits composition recipe + DSSE envelopes, attaches attestations to artifacts, and records determinism Merkle root/recipe metadata. | Implementer | +| 2025-12-02 | SURFACE-FS-07 completed: Surface.FS manifest schema now includes determinism metadata, composition recipe attestation fields, determinism verifier, and docs updated. Targeted determinism tests added; test run pending due to long restore/build in monorepo runner. | Implementer | +| 2025-12-01 | EntryTrace NDJSON emission, runtime reconciliation, and WebService/CLI exposure completed (18-504/505/506). | EntryTrace Guild | | 2025-12-01 | ZASTAVA-SURFACE-02: Observer resolves Surface manifest digests and `cas://` URIs, enriches drift evidence with artifact metadata, and counts failures via `zastava_surface_manifest_failures_total`. | Implementer | | 2025-12-01 | SCANNER-SORT-02: ComponentGraphBuilder sorts layer fragments by digest; regression test added. | Implementer | -| 2025-12-02 | Surface FS determinism plumbing verified: injected ICryptoHash into FileSurfaceManifestStore test harness; `dotnet test …SurfaceManifestDeterminismVerifierTests` (2/2 pass) and full `StellaOps.Scanner.Surface.FS.Tests` suite (7/7 pass). | Implementer | +| 2025-12-01 | SCANNER-EMIT-15-001: CycloneDX artifacts now publish `ContentHash`, carry Merkle/recipe URIs, emit `_composition.json` + DSSE envelopes (recipe & layer.fragments), and Surface manifests reference those attestations. DSSE signer is pluggable (deterministic fallback registered); real signing still pending. | Implementer | +| 2025-12-01 | SCANNER-SORT-02 completed: ComponentGraphBuilder sorts layer fragments by digest with regression test Build_SortsLayersByDigest. | Implementer | +| 2025-12-01 | ZASTAVA-SURFACE-02: Observer now resolves Surface manifest digests and `cas://` URIs, enriches drift evidence with artifact metadata, and counts failures via `zastava_surface_manifest_failures_total`. | Implementer | +| 2025-11-28 | Created `docs/modules/scanner/guides/surface-validation-extensibility.md` covering custom validators, reporters, configuration, and testing; SURFACE-VAL-05 DONE. | Implementer | +| 2025-11-28 | Created `docs/modules/scanner/guides/surface-fs-workflow.md` with end-to-end workflow including artefact generation, storage layout, consumption, and offline kit handling; SURFACE-FS-06 DONE. | Implementer | +| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Homebrew` library with `HomebrewReceiptParser`, `HomebrewPackageAnalyzer`, and `HomebrewAnalyzerPlugin`; 23 tests passing. SCANNER-ENG-0020 DONE. | Implementer | +| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Pkgutil` library with `PkgutilReceiptParser`, `BomParser`, `PkgutilPackageAnalyzer`, and `PkgutilAnalyzerPlugin`; 9 tests passing. SCANNER-ENG-0021 DONE. | Implementer | +| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Msi` library with `MsiDatabaseParser`, `MsiPackageAnalyzer`, and `MsiAnalyzerPlugin`; 22 tests passing. SCANNER-ENG-0024 DONE. | Implementer | +| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.WinSxS` library with `WinSxSManifestParser`, `WinSxSPackageAnalyzer`, and `WinSxSAnalyzerPlugin`; 18 tests passing. SCANNER-ENG-0025 DONE. | Implementer | +| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Chocolatey` library with `NuspecParser`, `ChocolateyPackageAnalyzer`, and `ChocolateyAnalyzerPlugin`; 44 tests passing. SCANNER-ENG-0026 DONE. | Implementer | +| 2025-11-28 | Updated `docs/modules/scanner/design/windows-analyzer.md` with implementation status documenting MSI/WinSxS/Chocolatey collectors, PURL formats, and vendor metadata schemas; registry collector deferred, policy predicates pending Policy module integration. SCANNER-ENG-0027 DONE. | Implementer | +| 2025-11-27 | Added missing package references to BuildX plugin (Configuration.EnvironmentVariables, DependencyInjection, Logging); refactored to use public AddSurfaceEnvironment API instead of internal SurfaceEnvironmentFactory; build passes. SCANNER-ENV-03 DONE. | Implementer | +| 2025-11-27 | Created SurfaceFeatureFlagsConfigurator to merge Surface.Env feature flags into WebService FeatureFlagOptions.Experimental dictionary; registered configurator in Program.cs. Cache roots and feature flags now wired from Surface.Env. SCANNER-ENV-02 DONE. | Implementer | +| 2025-11-27 | Verified SURFACE-ENV-03: Scanner Worker (SCANNER-ENV-01), WebService (SCANNER-ENV-02), and BuildX (SCANNER-ENV-03) all wire Surface.Env helpers; task complete. SURFACE-ENV-03 DONE. | Implementer | +| 2025-11-27 | Added CachingSurfaceSecretProvider (deterministic TTL cache), AuditingSurfaceSecretProvider (structured audit logging), and OfflineSurfaceSecretProvider (integrity-verified offline kit support); wired into ServiceCollectionExtensions with configurable options. SURFACE-SECRETS-03 DONE. | Implementer | +| 2025-11-27 | Added Surface.Validation project references to Zastava Observer and Webhook; wired AddSurfaceValidation() in service extensions for preflight checks. SURFACE-VAL-04 DONE. | Implementer | +| 2025-11-27 | Verified Zastava Observer and Webhook already have AddSurfaceEnvironment() wired with ZASTAVA prefixes; SURFACE-ENV-04 DONE. | Implementer | +| 2025-11-27 | Added Surface.Secrets project reference to BuildX plugin; implemented TryResolveAttestationToken() to fetch attestation secrets from Surface.Secrets; Worker/WebService already had configurators for CAS/registry/attestation secrets. SURFACE-SECRETS-04 DONE. | Implementer | +| 2025-11-27 | Verified Zastava Observer/Webhook already have ObserverSurfaceSecrets/WebhookSurfaceSecrets classes using ISurfaceSecretProvider for CAS and attestation secrets. SURFACE-SECRETS-05 DONE. | Implementer | +| 2025-11-27 | SURFACE-SECRETS-06 marked BLOCKED: requires Ops Guild input on Helm/Compose patterns for Surface.Secrets provider configuration (kubernetes/file/inline). Added to Decisions & Risks. | Implementer | +| 2025-11-27 | Integrated ISurfaceManifestWriter into SurfaceManifestStageExecutor to persist manifest documents to file-system store for offline/air-gapped scenarios; build verified. SURFACE-FS-03 DONE. | Implementer | +| 2025-11-27 | Added IRuntimeSurfaceFsClient injection to RuntimePostureEvaluator, enriching drift evidence with manifest digest/artifacts/metadata; added `zastava_surface_manifest_failures_total` metric with reason labels. SURFACE-FS-04 DONE. | Implementer | +| 2025-11-27 | Added TryResolveCasCredentials() to BuildX plugin using Surface.Secrets to fetch CAS access credentials; fixed attestation token resolution to use correct parser method. SCANNER-SECRETS-03 DONE. | Implementer | +| 2025-11-27 | Verified SurfacePointerService already exposes Surface.FS pointers (SurfaceManifestDocument, SurfaceManifestArtifact, manifest URI/digest) via reports endpoint. SURFACE-FS-05 DONE. | Implementer | +| 2025-11-27 | Added POST /policy/overlay endpoint for Cartographer integration: accepts graph nodes, returns deterministic overlays with sha256(tenant|nodeId|overlayKind) IDs, includes runtime evidence. Added PolicyOverlayRequestDto/ResponseDto contracts. SCANNER-GRAPH-21-001 DONE. | Implementer | +| 2025-11-27 | SCANNER-LNM-21-001 marked BLOCKED: Scanner WebService has no existing Concelier integration; requires HTTP client or shared library reference to Concelier.Core for linkset consumption. Added to Decisions & Risks. | Implementer | +| 2025-11-24 | Marked SURFACE-FS-01 DONE; spec anchored in `docs/modules/scanner/design/surface-fs.md` v1.1. | Scanner Guild | +| 2025-11-24 | Marked SURFACE-FS-02 DONE; core Surface.FS manifest/cache library implemented and DI-ready. | Scanner Guild | +| 2025-11-23 | Published Security-approved Surface.Secrets schema; moved SURFACE-SECRETS-01 to DONE, SURFACE-SECRETS-02/SURFACE-VAL-01 to TODO. | Security Guild | +| 2025-11-23 | Implemented Surface.Secrets provider chain/fallback and added DI tests; marked SURFACE-SECRETS-02 DONE. | Scanner Guild | +| 2025-11-23 | Pinned Surface.Env package version `0.1.0-alpha.20251123` and offline path in `docs/modules/scanner/design/surface-env-release.md`; SCANNER-ENV-03 moved to TODO. | BuildX Plugin Guild | +| 2025-11-23 | Updated Surface.Validation doc to v1.1, binding to Surface.Env release and secrets schema; marked SURFACE-VAL-01 DONE. | Scanner Guild | +| 2025-11-23 | Strengthened Surface.Validation secrets checks (provider/fallback/inline/file root) and added unit tests; marked SURFACE-VAL-02 DONE. | Scanner Guild | +| 2025-11-23 | Wired SurfaceValidation runner into Worker/WebService startup to fail fast; SURFACE-VAL-03 in progress. | Scanner Guild | +| 2025-11-19 | Normalized sprint to standard template and renamed from `SPRINT_136_scanner_surface.md` to `SPRINT_0136_0001_0001_scanner_surface.md`; content preserved. | Implementer | +| 2025-11-19 | Converted legacy filename `SPRINT_136_scanner_surface.md` to redirect stub pointing here to avoid divergent updates. | Implementer | +| 2025-11-18 | SCANNER-ENV-01 in progress: added manifest store options configurator in Scanner Worker and unit scaffold (tests pending due to local restore/vstest issues). | Implementer | +| 2025-11-18 | SCANNER-ENV-02 started: wired Surface manifest store options into Scanner WebService and unit scaffold added; tests pending (nuget.org restore cancelled locally). | Implementer | +| 2025-11-18 | Attempted `dotnet test` for Worker Surface manifest configurator; restore failed fetching StackExchange.Redis from nuget.org (network timeout); tests still pending CI. | Implementer | +| 2025-11-18 | SCANNER-ENV-03 started: BuildX plugin now loads Surface.Env defaults for cache root/bucket/tenant when args/env missing; tests not yet added. | Implementer | +| 2025-11-12 | SURFACE-ENV-01 done; SURFACE-ENV-02 started; SURFACE-SECRETS-01/02 in progress. | Scanner Guild | +| 2025-11-08 | Sprint stub created; awaiting completion of Sprint 0135. | Planning | +| 2025-10-26 | Initial sprint plan captured; dependencies noted across Scheduler/Surface/Cartographer. | Planning | ## Decisions & Risks -- EntryTrace NDJSON export and replay completed; relies on deterministic `/proc` capture and preserved ordering for confidence adjustments. -- SCANNER-SURFACE-01 blocked: no task definition/contract present; needs scope before DOING. +- SCANNER-LNM-21-001 remains BLOCKED: Scanner WebService lacks Concelier integration; decision needed on shared client vs new HTTP client. Downstream SCANNER-LNM-21-002 cannot start. +- SURFACE-SECRETS-06 BLOCKED pending Ops Helm/Compose patterns for Surface.Secrets provider configuration (kubernetes/file/inline). +- SCANNER-EVENTS-16-301 BLOCKED awaiting orchestrator envelope contract + Notifier ingestion test plan. +- SCANNER-SURFACE-01 lacks scoped contract; placeholder must be defined or retired before new dependencies are added. +- SCANNER-EMIT-15-001 DOING: real DSSE signer still pending; deterministic-local signer only. Surface manifest consumers must not assume transparency until signer is wired. +- Long restore/build times in monorepo runners delayed determinism test runs for SURFACE-FS-07; rerun in CI once signer work lands. ## Next Checkpoints - Schedule kickoff after Sprint 0135 completion (date TBD). +- Concelier client decision & Ops secrets provisioning pattern review (target scheduling pending owners). diff --git a/docs/implplan/SPRINT_0140_0001_0001_runtime_signals.md b/docs/implplan/SPRINT_0140_0001_0001_runtime_signals.md index 8c086bed4..c4dedf346 100644 --- a/docs/implplan/SPRINT_0140_0001_0001_runtime_signals.md +++ b/docs/implplan/SPRINT_0140_0001_0001_runtime_signals.md @@ -30,15 +30,16 @@ | 2 | 140.B SBOM Service wave | DOING (2025-11-28) | Sprint 0142 mostly complete: SBOM-SERVICE-21-001..004, SBOM-AIAI-31-001/002, SBOM-ORCH-32/33/34-001, SBOM-VULN-29-001/002 all DONE. Only SBOM-CONSOLE-23-001/002 remain BLOCKED. | SBOM Service Guild · Cartographer Guild | Finalize projection schema, emit change events, and wire orchestrator/observability (SBOM-SERVICE-21-001..004, SBOM-AIAI-31-001/002). | | 3 | 140.C Signals wave | DOING (2025-11-28) | Sprint 0143: SIGNALS-24-001/002/003 DONE; SIGNALS-24-004/005 remain BLOCKED on CAS promotion. | Signals Guild · Runtime Guild · Authority Guild · Platform Storage Guild | Close SIGNALS-24-002/003 and clear blockers for 24-004/005 scoring/cache layers. | | 4 | 140.D Zastava wave | DONE (2025-11-28) | Sprint 0144 (Zastava Runtime Signals) complete: all ZASTAVA-ENV/SECRETS/SURFACE tasks DONE. | Zastava Observer/Webhook Guilds · Surface Guild | Prepare env/secret helpers and admission hooks; start once cache endpoints and helpers are published. | -| 5 | DECAY-GAPS-140-005 | BLOCKED (2025-12-02) | cosign binary unavailable in environment; cannot sign `confidence_decay_config.yaml`. Needs cosign (or offline signer) to proceed by 2025-12-05. | Signals Guild · Product Mgmt | Address decay gaps U1–U10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed `confidence_decay_config` (τ governance, floor/freeze/SLA clamps), weighted signals taxonomy, UTC/monotonic time rules, deterministic recompute cadence + checksum, uncertainty linkage, migration/backfill plan, API fields/bands, and observability/alerts. | -| 6 | UNKNOWN-GAPS-140-006 | BLOCKED (2025-12-02) | cosign binary unavailable; cannot sign unknowns scoring manifest. Needs cosign/offline signer before 2025-12-05. | Signals Guild · Policy Guild · Product Mgmt | Address unknowns gaps UN1–UN10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed Unknowns registry schema + scoring manifest (deterministic), decay policy catalog, evidence/provenance capture, SBOM/VEX linkage, SLA/suppression rules, API/CLI contracts, observability/reporting, offline bundle inclusion, and migration/backfill. | -| 7 | UNKNOWN-HEUR-GAPS-140-007 | BLOCKED (2025-12-02) | cosign binary unavailable; cannot sign heuristic catalog/schema + fixtures. Needs cosign/offline signer before 2025-12-05. | Signals Guild · Policy Guild · Product Mgmt | Remediate UT1–UT10: publish signed heuristic catalog/schema with deterministic scoring formula, quality bands, waiver policy with DSSE, SLA coupling, offline kit packaging, observability/alerts, backfill plan, explainability UX fields/exports, and fixtures with golden outputs. | -| 9 | COSIGN-INSTALL-140 | TODO | Install/provide cosign binary (or offline signer) in build environment by 2025-12-03 to unblock DSSE signing for tasks 5–7. | Platform / Build Guild | Deliver cosign binary locally (no network dependency at signing time) or alternate signer; document path and version in Execution Log. | +| 5 | DECAY-GAPS-140-005 | DOING (2025-12-02) | cosign v2.6.0 available at `tools/cosign/cosign` (sha256 `ea5c65f99425d6cfbb5c4b5de5dac035f14d09131c1a0ea7c7fc32eab39364f9`); DSSE signing on 2025-12-05. | Signals Guild · Product Mgmt | Address decay gaps U1–U10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed `confidence_decay_config` (τ governance, floor/freeze/SLA clamps), weighted signals taxonomy, UTC/monotonic time rules, deterministic recompute cadence + checksum, uncertainty linkage, migration/backfill plan, API fields/bands, and observability/alerts. | +| 6 | UNKNOWN-GAPS-140-006 | DOING (2025-12-02) | cosign v2.6.0 available at `tools/cosign/cosign`; sign unknowns scoring manifest and publish DSSE envelope by 2025-12-05. | Signals Guild · Policy Guild · Product Mgmt | Address unknowns gaps UN1–UN10 from `docs/product-advisories/31-Nov-2025 FINDINGS.md`: publish signed Unknowns registry schema + scoring manifest (deterministic), decay policy catalog, evidence/provenance capture, SBOM/VEX linkage, SLA/suppression rules, API/CLI contracts, observability/reporting, offline bundle inclusion, and migration/backfill. | +| 7 | UNKNOWN-HEUR-GAPS-140-007 | DOING (2025-12-02) | cosign v2.6.0 available at `tools/cosign/cosign`; prep catalog/schema fixtures for 2025-12-05 signing. | Signals Guild · Policy Guild · Product Mgmt | Remediate UT1–UT10: publish signed heuristic catalog/schema with deterministic scoring formula, quality bands, waiver policy with DSSE, SLA coupling, offline kit packaging, observability/alerts, backfill plan, explainability UX fields/exports, and fixtures with golden outputs. | +| 9 | COSIGN-INSTALL-140 | DONE (2025-12-02) | cosign v2.6.0 staged under `tools/cosign` (sha256 `ea5c65f99425d6cfbb5c4b5de5dac035f14d09131c1a0ea7c7fc32eab39364f9`); add `tools/cosign` to PATH for signing 2025-12-05. | Platform / Build Guild | Deliver cosign binary locally (no network dependency at signing time) or alternate signer; document path and version in Execution Log. | | 8 | SIGNER-ASSIGN-140 | DONE (2025-12-02) | Signer designated: Signals Guild (Alice Carter); DSSE signing checkpoint remains 2025-12-05. | Signals Guild · Policy Guild | Name signer(s), record in Execution Log, and proceed to DSSE signing + Evidence Locker ingest. | ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Staged cosign v2.6.0 binary under `tools/cosign` (sha256 `ea5c65f99425d6cfbb5c4b5de5dac035f14d09131c1a0ea7c7fc32eab39364f9`); symlink available at `tools/cosign/cosign`; flipped COSIGN-INSTALL-140 to DONE and tasks 5–7 back to DOING for 2025-12-05 DSSE signing. | Implementer | | 2025-12-02 | Refreshed Decisions & Risks after signer assignment; DSSE signing fixed for 2025-12-05 and decay/unknowns/heuristics remain BLOCKED pending `cosign` availability in offline kit. | Project Mgmt | | 2025-12-02 | Marked DECAY-GAPS-140-005 / UNKNOWN-GAPS-140-006 / UNKNOWN-HEUR-GAPS-140-007 as BLOCKED pending DSSE signer assignment; added task SIGNER-ASSIGN-140 (BLOCKED) and DSSE signing checkpoint (2025-12-05). | Implementer | | 2025-12-02 | Flagged cascading risk to SPRINT_0143/0144/0150 if signer not assigned by 2025-12-03; will mirror BLOCKED status to dependent tasks if missed. | Implementer | @@ -79,7 +80,7 @@ - Link-Not-Merge v1 schema frozen 2025-11-17; fixtures staged under `docs/modules/sbomservice/fixtures/lnm-v1/`; AirGap parity review scheduled for 2025-11-23 (see Next Checkpoints) must record hashes to fully unblock. - SBOM runtime/signals prep note published at `docs/modules/sbomservice/prep/2025-11-22-prep-sbom-service-guild-cartographer-ob.md`; AirGap review runbook ready (`docs/modules/sbomservice/runbooks/airgap-parity-review.md`). Wave moves to TODO pending review completion and fixture hash upload. - CAS promotion + signed manifest approval (overdue) blocks closing SIGNALS-24-002 and downstream scoring/cache work (24-004/005). -- Decay/Unknowns/heuristics remediation (U1–U10, UN1–UN10, UT1–UT10) remain BLOCKED on missing `cosign` binary even though signer is assigned (Alice Carter); DSSE signing scheduled for 2025-12-05. If the signing tool (or offline alternative) is not available by 2025-12-03, mirror BLOCKED status into SPRINT_0143/0144/0150. Draft docs and artifacts posted at `docs/modules/signals/decay/2025-12-01-confidence-decay.md`, `docs/modules/signals/decay/confidence_decay_config.yaml`, `docs/modules/signals/unknowns/2025-12-01-unknowns-registry.md`, `docs/modules/signals/unknowns/unknowns_scoring_manifest.json`, and `docs/modules/signals/heuristics/` (catalog, schema, fixtures); DSSE signatures pending. Hashes recorded in `docs/modules/signals/SHA256SUMS` for offline/air-gap parity; Evidence Locker ingest plan staged at `docs/modules/signals/evidence/README.md` and will be populated post-signing. COSIGN-INSTALL-140 added to track tool availability. +- Cosign v2.6.0 binary pinned at `tools/cosign/cosign` (sha256 `ea5c65f99425d6cfbb5c4b5de5dac035f14d09131c1a0ea7c7fc32eab39364f9`; see `tools/cosign/README.md`); DSSE signing deadline remains 2025-12-05—decay/unknowns/heuristics teams must sign and ingest envelopes + SHA256SUMS into Evidence Locker the same day or cascade risk into 0143/0144/0150. Draft docs and artifacts posted at `docs/modules/signals/decay/2025-12-01-confidence-decay.md`, `docs/modules/signals/decay/confidence_decay_config.yaml`, `docs/modules/signals/unknowns/2025-12-01-unknowns-registry.md`, `docs/modules/signals/unknowns/unknowns_scoring_manifest.json`, and `docs/modules/signals/heuristics/` (catalog, schema, fixtures); DSSE signatures pending. Hashes recorded in `docs/modules/signals/SHA256SUMS`; Evidence Locker ingest plan in `docs/modules/signals/evidence/README.md`. - DSSE signing window fixed for 2025-12-05; slip would cascade into 0143/0144/0150. Ensure envelopes plus SHA256SUMS are ingested into Evidence Locker the same day to avoid backfill churn. - Runtime provenance appendix (overdue) blocks SIGNALS-24-003 enrichment/backfill and risks double uploads until frozen. - Surface.FS cache drop timeline (overdue) and Surface.Env owner assignment keep Zastava env/secret/admission tasks blocked. @@ -105,7 +106,7 @@ | 2025-12-04 | Unknowns schema review | Approve Unknowns registry schema/enums + deterministic scoring manifest (UN1–UN10) and offline bundle inclusion plan. | Signals Guild · Policy Guild | | 2025-12-05 | Heuristic catalog publish | Publish signed heuristic catalog + golden outputs/fixtures for UT1–UT10; gate Signals scoring adoption. | Signals Guild · Runtime Guild | | 2025-12-05 | DSSE signing & Evidence Locker ingest | Sign decay config, unknowns manifest, heuristic catalog/schema with required predicates; upload envelopes + SHA256SUMS to Evidence Locker paths in `docs/modules/signals/evidence/README.md`. | Signals Guild · Policy Guild | -| 2025-12-03 | Provide cosign/offline signer | Deliver cosign binary (or offline signing path) for tasks 5–7; otherwise slip DSSE signing. | Platform / Build Guild | +| 2025-12-03 | Provide cosign/offline signer | DONE 2025-12-02: cosign v2.6.0 at `tools/cosign/cosign` (sha256 `ea5c65f99425d6cfbb5c4b5de5dac035f14d09131c1a0ea7c7fc32eab39364f9`); add `tools/cosign` to PATH ahead of 2025-12-05 signing. | Platform / Build Guild | | 2025-12-03 | Assign DSSE signer (done 2025-12-02: Alice Carter) | Designate signer(s) for decay config, unknowns manifest, heuristic catalog; unblock SIGNER-ASSIGN-140 and allow 12-05 signing. | Signals Guild · Policy Guild | --- diff --git a/docs/implplan/SPRINT_0144_0001_0001_zastava_runtime_signals.md b/docs/implplan/SPRINT_0144_0001_0001_zastava_runtime_signals.md index a98556f20..dcfd320de 100644 --- a/docs/implplan/SPRINT_0144_0001_0001_zastava_runtime_signals.md +++ b/docs/implplan/SPRINT_0144_0001_0001_zastava_runtime_signals.md @@ -29,8 +29,8 @@ | 5 | ZASTAVA-SURFACE-01 | DONE (2025-11-18) | Surface.FS drift client exercised in smoke suite | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer) | Integrate Surface.FS client for runtime drift detection (lookup cached layer hashes/entry traces). | | 6 | ZASTAVA-SURFACE-02 | DONE (2025-11-18) | Admission smoke tests green with Surface.FS pointer enforcement | Zastava Webhook Guild (src/Zastava/StellaOps.Zastava.Webhook) | Enforce Surface.FS availability during admission (deny when cache missing/stale) and embed pointer checks in webhook response. | | 7 | ZASTAVA-GAPS-144-007 | DONE (2025-12-02) | Remediation plan published at `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; schemas/kit/thresholds tracked below. | Zastava Observer/Webhook Guilds / src/Zastava | Remediate ZR1–ZR10: signed schemas + hash recipes, tenant binding, deterministic clocks/ordering, DSSE provenance, side-effect/bypass controls, offline zastava-kit, ledger/replay linkage, threshold governance, PII/redaction policy, kill-switch/fallback rules with alerts and audits. | -| 8 | ZASTAVA-SCHEMAS-0001 | TODO | DSSE signing window 2025-12-06; depends on signer availability. | Zastava Guild | Publish signed observer/admission schemas + examples + test vectors under `docs/modules/zastava/schemas/` with SHA256SUMS and DSSE envelopes. | -| 9 | ZASTAVA-KIT-0001 | TODO | Depends on ZASTAVA-SCHEMAS-0001 and thresholds signing. | Zastava Guild | Build `zastava-kit` bundle (schemas, thresholds, observations/admissions export, SHA256SUMS, verify.sh) with deterministic tar+zstd flags; include DSSE signatures and Evidence Locker URIs. | +| 8 | ZASTAVA-SCHEMAS-0001 | DONE (2025-12-02) | DSSE signing completed; keyid mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc. | Zastava Guild | Published signed observer/admission schemas + examples + test vectors under `docs/modules/zastava/schemas/` with SHA256SUMS and DSSE envelopes. | +| 9 | ZASTAVA-KIT-0001 | DONE (2025-12-02) | Depends on ZASTAVA-SCHEMAS-0001 and thresholds signing. | Zastava Guild | Built `zastava-kit` bundle (schemas, thresholds, exports, SHA256SUMS, verify.sh) with deterministic tar+zstd flags; DSSE signatures + Evidence Locker targets recorded. | ## Execution Log | Date (UTC) | Update | Owner | @@ -66,6 +66,11 @@ | 2025-12-02 | Drafted ZR schemas (`docs/modules/zastava/schemas/*.json`), thresholds (`docs/modules/zastava/thresholds.yaml`), kit scaffolding (`docs/modules/zastava/kit/*`), and `docs/modules/zastava/SHA256SUMS`; DSSE signing pending (target 2025-12-06). | Implementer | | 2025-12-02 | Added schema examples (`docs/modules/zastava/schemas/examples/*.json`) and appended hashes to `docs/modules/zastava/SHA256SUMS` to aid deterministic validation. | Implementer | | 2025-12-02 | Created Evidence Locker plan at `docs/modules/zastava/evidence/README.md` with predicates, signing template, and target paths for schemas/thresholds/kit (signing target 2025-12-06). | Implementer | +| 2025-12-02 | Started ZASTAVA-SCHEMAS-0001 and ZASTAVA-KIT-0001; prepping signing key, canonical hashes, and kit packaging steps. | Zastava Guild | +| 2025-12-02 | Completed ZASTAVA-SCHEMAS-0001: canonicalised schemas/examples, added DSSE envelopes, refreshed SHA256SUMS, and published ed25519 pub key (`kit/ed25519.pub`). | Zastava Guild | +| 2025-12-02 | Completed ZASTAVA-THRESHOLDS-0001: DSSE-signed `thresholds.yaml`, aligned Evidence Locker targets, and added to kit manifest. | Zastava Guild | +| 2025-12-02 | Completed ZASTAVA-KIT-0001: built deterministic `kit/zastava-kit.tzst` via tar+zstd (level 19, window_log=27), added DSSE for kit, refreshed verify script, and ran offline verification. Private key removed from workspace post-signing. | Zastava Guild | +| 2025-12-02 | Finalised DSSE set with keyid mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc; regenerated SHA256SUMS, rebuilt kit tar.zst, refreshed kit DSSE, and removed signing key from /tmp. | Zastava Guild | ## Decisions & Risks - Surface Env/Secrets/FS wiring complete for observer and webhook; admission now embeds manifest pointers and denies on missing cache manifests. @@ -74,10 +79,10 @@ - Upstream Authority/Auth packages (notably `StellaOps.Auth.Security`) remain needed in local caches; refresh mirror before CI runs to avoid restore stalls. - Surface.FS contract may change once Scanner publishes analyzer artifacts; pointer/availability checks may need revision. - Surface.Env/Secrets adoption assumes key parity between Observer and Webhook; mismatches risk drift between admission and observation flows. -- New advisory gaps (ZR1–ZR10) addressed in remediation plan at `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; drafts for schemas/thresholds/kit and SHA256 recorded under `docs/modules/zastava/`; DSSE signing still pending (target 2025-12-06). Evidence Locker paths will be added after signing. -- New advisory gaps (ZR1–ZR10) addressed in remediation plan at `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; drafts for schemas/thresholds/kit (plus examples) and SHA256 recorded under `docs/modules/zastava/`; DSSE signing still pending (target 2025-12-06). Evidence Locker plan staged at `docs/modules/zastava/evidence/README.md`; downstream kit build tracked via ZASTAVA-KIT-0001. +- New advisory gaps (ZR1–ZR10) addressed in remediation plan at `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; schemas/thresholds/exports now DSSE-signed (ed25519 pub `mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc`) with hashes in `docs/modules/zastava/SHA256SUMS`; kit DSSE stored at `docs/modules/zastava/kit/zastava-kit.tzst.dsse` and verification via `kit/verify.sh`; Evidence Locker targets listed in `docs/modules/zastava/evidence/README.md`. +- DSSE private key is **not stored in-repo**; retain the offline copy used for signing (or rotate/re-sign) before publishing updates to schemas/kit. ## Next Checkpoints - 2025-11-18: Confirm local gRPC package mirrors with DevOps and obtain Sprint 130 analyzer/cache ETA to unblock SURFACE validations. - 2025-11-20: Dependency review with Scanner/AirGap owners to lock Surface.FS cache semantics; if ETA still missing, escalate per sprint 140 plan. -- 2025-12-06: ZR schemas/kit signing — produce signed schemas, thresholds, and `zastava-kit` bundle per `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; publish Evidence Locker paths + SHA256. +- 2025-12-03: Upload DSSE artefacts + kit tar to Evidence Locker paths in `docs/modules/zastava/evidence/README.md`; mirror pub key for downstream consumers. diff --git a/docs/implplan/SPRINT_0150_0001_0001_scheduling_automation.md b/docs/implplan/SPRINT_0150_0001_0001_scheduling_automation.md index c4dc90a8d..549badc9f 100644 --- a/docs/implplan/SPRINT_0150_0001_0001_scheduling_automation.md +++ b/docs/implplan/SPRINT_0150_0001_0001_scheduling_automation.md @@ -21,45 +21,48 @@ ## Delivery Tracker | # | Task ID | Status | Key dependency / next step | Owners | Task Definition | | --- | --- | --- | --- | --- | --- | -| 1 | 150.A-Orchestrator | TODO | 0140.A (Graph) ✅ DONE, 0140.D (Zastava) ✅ DONE. Remaining blockers: 0120.A AirGap staleness + 0130.A Scanner surface | Orchestrator Service Guild · AirGap Policy/Controller Guilds · Observability Guild | Kick off orchestration scheduling/telemetry baseline for automation epic. | -| 2 | 150.B-PacksRegistry | TODO | 150.A must reach DOING; confirm tenancy scaffolding from Orchestrator | Packs Registry Guild · Exporter Guild · Security Guild | Packs registry automation stream staged; start after Orchestrator scaffolding. | -| 3 | 150.C-Scheduler | TODO | 0140.A Graph ✅ DONE. Remaining blocker: 0130.A Scanner surface | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Scheduler impact index improvements gated on Graph overlays. | -| 4 | 150.D-TaskRunner | TODO | Requires Orchestrator/Scheduler telemetry baselines (150.A/150.C) | Task Runner Guild · AirGap Guilds · Evidence Locker Guild | Execution engine upgrades and evidence integration to start post-baselines. | +| 1 | 150.A-Orchestrator | BLOCKED | Graph (0140.A) ✅ DONE; Zastava (0140.D) ✅ DONE. Blocked on 0120.A AirGap staleness (56-002/57/58) and Scanner surface Java/Lang chain (0131). | Orchestrator Service Guild · AirGap Policy/Controller Guilds · Observability Guild | Kick off orchestration scheduling/telemetry baseline for automation epic. | +| 2 | 150.B-PacksRegistry | BLOCKED | 150.A must reach DOING; confirm tenancy scaffolding from Orchestrator | Packs Registry Guild · Exporter Guild · Security Guild | Packs registry automation stream staged; start after Orchestrator scaffolding. | +| 3 | 150.C-Scheduler | BLOCKED | Graph ✅ DONE; still waiting on Scanner surface Java/Lang chain (0131 21-005..011) | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Scheduler impact index improvements gated on Graph overlays. | +| 4 | 150.D-TaskRunner | BLOCKED | Requires Orchestrator/Scheduler telemetry baselines (150.A/150.C) | Task Runner Guild · AirGap Guilds · Evidence Locker Guild | Execution engine upgrades and evidence integration to start post-baselines. | ## Wave Coordination Snapshot | Wave | Guild owners | Shared prerequisites | Status | Notes | | --- | --- | --- | --- | --- | -| 150.A Orchestrator | Orchestrator Service Guild · AirGap Policy/Controller Guilds · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Graph (0140.A) and Zastava (0140.D) now DONE. AirGap staleness (0120.A 56-002/57/58) and Scanner surface (0130.A) remain blockers. Approaching readiness. | -| 150.B PacksRegistry | Packs Registry Guild · Exporter Guild · Security Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Blocked on Orchestrator tenancy scaffolding; specs ready once 150.A flips to DOING. | -| 150.C Scheduler | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Graph overlays (0140.A) now DONE. Scheduler impact index work can proceed once Scanner surface (0130.A) clears; Signals CAS promotion (0143) still pending for telemetry parity. | -| 150.D TaskRunner | Task Runner Guild · AirGap Guilds · Evidence Locker Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Execution engine upgrades staged; start once Orchestrator/Scheduler telemetry baselines exist. | +| 150.A Orchestrator | Orchestrator Service Guild · AirGap Policy/Controller Guilds · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | BLOCKED | Graph (0140.A) and Zastava (0140.D) DONE. AirGap staleness (0120.A 56-002/57/58) and Scanner surface Java/Lang chain (0131 21-005..011) still blocking kickoff. | +| 150.B PacksRegistry | Packs Registry Guild · Exporter Guild · Security Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | BLOCKED | Blocked on Orchestrator tenancy scaffolding; specs ready once 150.A enters DOING. | +| 150.C Scheduler | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | BLOCKED | Graph overlays (0140.A) DONE; Scanner surface Java/Lang chain still blocked; Signals CAS/DSSE signing (0140.C) pending for telemetry parity. | +| 150.D TaskRunner | Task Runner Guild · AirGap Guilds · Evidence Locker Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | BLOCKED | Execution engine upgrades staged; start once Orchestrator/Scheduler telemetry baselines exist. | ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Upstream refresh: DEVOPS-SBOM-23-001 and DEVOPS-SCANNER-CI-11-001 delivered (Sprint 503) clearing infra blockers; SBOM console endpoints remain to implement. Signals wave (0140.C) still blocked on cosign availability for DSSE signing; AirGap staleness (0120.A 56-002/57/58) and Scanner Java/Lang chain (0131 21-005..011) remain blocked. All 150.* tasks kept BLOCKED. | Project Mgmt | | 2025-11-30 | Upstream refresh: Sprint 0120 AirGap staleness (LEDGER-AIRGAP-56-002/57/58) still BLOCKED; Scanner surface Sprint 0131 has Deno 26-009/010/011 DONE but Java/Lang chain 21-005..011 BLOCKED pending CI/CoreLinksets; SBOM wave (Sprint 0142) core tasks DONE with Console endpoints still BLOCKED on DEVOPS-SBOM-23-001 in Sprint 503; Signals (Sprint 0143) 24-002/003 remain BLOCKED on CAS promotion/provenance though 24-004/005 are DONE. No 150.* task can start yet. | Implementer | | 2025-11-28 | Synced with downstream sprints: Sprint 0141 (Graph) DONE, Sprint 0142 (SBOM) mostly DONE, Sprint 0143 (Signals) 3/5 DONE, Sprint 0144 (Zastava) DONE. Updated Sprint 0140 tracker and revised 150.* upstream dependency status. 150.A-Orchestrator may start once remaining AirGap/Scanner blockers clear. | Implementer | | 2025-11-28 | Upstream dependency check: Sprint 0120 (Policy/Reasoning) has LEDGER-29-007/008, LEDGER-34-101, LEDGER-AIRGAP-56-001 DONE but 56-002/57-001/58-001/ATTEST-73-001 BLOCKED. Sprint 0140 (Runtime/Signals) has all waves BLOCKED except SBOM (TODO). No Sprint 0130.A file found. All 150.* tasks remain TODO pending upstream readiness. | Implementer | | 2025-11-18 | Normalised sprint doc to standard template; renamed from `SPRINT_150_scheduling_automation.md`. | Planning | -## Upstream Dependency Status (as of 2025-11-30) +## Upstream Dependency Status (as of 2025-12-02) | Upstream Sprint | Key Deliverable | Status | Impact on 150.* | | --- | --- | --- | --- | -| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-007/008 (Observability/load harness) | DONE | Partial readiness for 150.A | | Sprint 0120.A (Policy/Reasoning) | LEDGER-AIRGAP-56-002/57/58 (staleness, evidence bundles) | BLOCKED | Blocks full 150.A readiness + 150.C verification | -| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-009 (deploy/backup collateral) | BLOCKED (awaiting Sprint 501 ops paths) | Not a gate for kickoff but limits rollout evidence | -| Sprint 0130.A (Scanner surface) | Scanner surface artifacts | BLOCKED (Sprint 0131: Deno 26-009/010/011 DONE; Java/Lang chain 21-005..011 BLOCKED pending CI/CoreLinksets) | Blocks 150.A, 150.C verification | -| Sprint 0140.A (Graph overlays) | 140.A Graph wave | **DONE** (Sprint 0141 complete) | Unblocks 150.C Scheduler graph deps | -| Sprint 0140.A (Graph overlays) | 140.B SBOM Service wave | CORE DONE (Sprint 0142: 21-001/002/003/004/23-001/002/29-001/002 DONE); Console endpoints 23-001/002 still BLOCKED on DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i) | Partially unblocks 150.A/150.C; Console integrations pending | -| Sprint 0140.A (Graph overlays) | 140.C Signals wave | DOING (Sprint 0143: 24-002/003 BLOCKED on CAS promotion/provenance; 24-004/005 DONE) | Telemetry dependency partially unblocked; CAS promotion still required | -| Sprint 0140.A (Graph overlays) | 140.D Zastava wave | **DONE** (Sprint 0144 complete) | Unblocks 150.A surface deps | +| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-009-DEV (deploy/backup collateral) | BLOCKED (awaiting Sprint 501 ops paths) | Not a gate for kickoff but limits rollout evidence | +| Sprint 0131 (Scanner surface phase II) | Deno runtime chain 26-009/010/011 | DONE | Partial readiness for scanner surface inputs | +| Sprint 0131 (Scanner surface phase II) | Java/Lang chain 21-005..011 | BLOCKED (CoreLinksets still missing; DEVOPS-SCANNER-CI-11-001 delivered 2025-11-30) | Blocks 150.A and 150.C verification | +| Sprint 0141 (Graph overlays 140.A) | GRAPH-INDEX-28-007..010 | **DONE** | Unblocks 150.C Scheduler graph deps | +| Sprint 0142 (SBOM Service 140.B) | SBOM-SERVICE-21-001..004, 23-001/002, 29-001/002 | CORE DONE; SBOM-CONSOLE-23-001/002 remain TODO now that DEVOPS-SBOM-23-001 (Sprint 503) is DONE | Partially unblocks 150.A/150.C; console integrations pending | +| Sprint 0143 (Signals 140.C) | SIGNALS-24-002/003 | BLOCKED (CAS promotion/provenance) | Telemetry dependency partially unblocked; still blocks parity | +| Sprint 0140 (Signals/decay/unknowns) | DECAY-GAPS-140-005 / UNKNOWN-GAPS-140-006 / UNKNOWN-HEUR-GAPS-140-007 | BLOCKED (cosign binary not available; DSSE signing window 2025-12-05) | Blocks telemetry parity needed before 150.A/150.C baselines start | +| Sprint 0144 (Zastava 140.D) | ZASTAVA-ENV/SECRETS/SURFACE | **DONE** | Surface deps unblocked | +| Sprint 0144 (Zastava 140.D) | ZASTAVA-SCHEMAS-0001 / ZASTAVA-KIT-0001 | TODO (DSSE signing target 2025-12-06) | Non-blocking unless cache/schema contracts change | ## Decisions & Risks -- **Progress (2025-11-30):** Graph (0140.A) and Zastava (0140.D) waves DONE; SBOM Service (0140.B) core DONE with Console APIs still BLOCKED on Sprint 503; Signals (0140.C) has 24-004/005 DONE while 24-002/003 wait on CAS. Remaining blockers: 0120.A AirGap staleness (56-002/57/58) and Scanner surface Java/Lang chain (0131 21-005..011). -- SBOM Service core endpoints/events delivered (Sprint 0142); Console-facing APIs remain BLOCKED on DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i). Track to avoid drift once Orchestrator/Scheduler streams start. -- 150.A Orchestrator and 150.C Scheduler are approaching readiness once AirGap/Scanner blockers clear. -- This sprint is a coordination snapshot only; implementation tasks continue in Sprint 151+ and should mirror status changes here to avoid drift. -- Sprint 0130.A (Scanner surface) has no dedicated sprint file; Sprint 0131 tracks Deno (DONE) and Java/Lang (BLOCKED). Coordinate with Scanner Guild to finalize. +- **Progress (2025-12-02):** Graph (0140.A) and Zastava (0140.D) DONE; SBOM Service core DONE with Console APIs now unblocked by DEVOPS-SBOM-23-001 (Sprint 503) but still pending implementation. Signals wave (0140.C) still blocked on CAS promotion and missing `cosign` for DSSE signing (DECAY/UNKNOWN/HEUR gaps). AirGap staleness (0120.A 56-002/57/58) and Scanner Java/Lang chain (0131 21-005..011) remain blockers, keeping all 150.* tasks BLOCKED. +- SBOM console endpoints should move next: feed/runner delivered via DEVOPS-SBOM-23-001; track SBOM-CONSOLE-23-001/002 execution to avoid drift before Orchestrator/Scheduler start. +- DSSE signing risk: cosign binary absent; signing window fixed at 2025-12-05 for Signals decay/unknowns/heuristics and 2025-12-06 for Zastava schemas/kit. If not resolved, telemetry parity and cache contracts stay blocked for 150.A/150.C baselines. +- Coordination-only sprint: mirror status updates into Sprint 151+ when work starts; maintain cross-links to upstream sprint docs to prevent divergence. +- Sprint 0130/0131 Scanner surface remains the primary gating item alongside AirGap staleness; re-evaluate start once either clears. ## Next Checkpoints - None scheduled; add next scheduling/automation sync once upstream readiness dates are confirmed. diff --git a/docs/implplan/SPRINT_0212_0001_0001_web_i.md b/docs/implplan/SPRINT_0212_0001_0001_web_i.md index 9a6ea8c98..940847189 100644 --- a/docs/implplan/SPRINT_0212_0001_0001_web_i.md +++ b/docs/implplan/SPRINT_0212_0001_0001_web_i.md @@ -70,6 +70,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | WEB-CONSOLE-23-002: added trace IDs on status/stream calls, heartbeat + exponential backoff reconnect in console run stream service, and new client/service unit tests. Backend commands still not run locally (disk constraint). | BE-Base Platform Guild | | 2025-12-01 | Started WEB-CONSOLE-23-002: added console status client (polling) + SSE run stream, store/service, and UI component; unit specs added. Commands/tests not executed locally due to PTY/disk constraint. | BE-Base Platform Guild | | 2025-11-07 | Enforced unknown-field detection, added shared `AocError` payload (HTTP + CLI), refreshed guard docs, and extended tests/endpoint helpers. | BE-Base Platform Guild | | 2025-11-07 | API scaffolding started for console workspace; `docs/advisory-ai/console.md` using placeholder responses while endpoints wire up. | Console Guild | diff --git a/docs/implplan/SPRINT_0216_0001_0001_web_v.md b/docs/implplan/SPRINT_0216_0001_0001_web_v.md index 8019de41c..bccc88094 100644 --- a/docs/implplan/SPRINT_0216_0001_0001_web_v.md +++ b/docs/implplan/SPRINT_0216_0001_0001_web_v.md @@ -70,6 +70,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | WEB-RISK-66-001: risk HTTP client/store now handle 429 rate-limit responses with retry-after hints and RateLimitError wiring; unit specs added (execution deferred—npm test not yet run). | BE-Base Platform Guild | | 2025-12-02 | Risk/Vuln clients now share trace ID generator util; vulnerability client emits trace headers across list/detail/stats; spec asserts header. | BE-Base Platform Guild | | 2025-12-02 | Test run skipped: `npm test` script unavailable in current environment; unit specs added but not executed. | BE-Base Platform Guild | | 2025-12-02 | Added empty/loading states to risk table for better UX while gateway data loads. | BE-Base Platform Guild | diff --git a/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md b/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md index 0b8108822..8952144d3 100644 --- a/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md +++ b/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md @@ -36,6 +36,7 @@ | Date (UTC) | Update | Owner | | --- | --- | --- | | 2025-11-30 | Normalised Delivery Tracker numbering, removed duplicate GAP-ZAS-002 row, and aligned statuses with Execution Log. | Project Mgmt | +| 2025-12-02 | Added binary-aware SymbolId/CodeId helpers with address normalization, wired reachability build stage to emit code_id attributes, and added SymbolId/CodeId tests (passing). | Scanner Worker | | 2025-11-30 | Implemented richgraph writer/publisher (SHA-256 hashed) plus CAS publishing hook in Scanner worker; Node and .NET lifters now emit code_id/purl metadata; GAP-SCAN-001 moved to DOING. Tests for new writer/publisher added; restore via dotnet test still flaky (nuget spinner). | Scanner Worker | | 2025-11-26 | Validated runtime facts builder: `dotnet test src/Zastava/__Tests/StellaOps.Zastava.Observer.Tests/StellaOps.Zastava.Observer.Tests.csproj --filter RuntimeFactsBuilderTests` restored and passed; Observer build clean. | Zastava Observer Guild | | 2025-11-26 | Implemented runtime facts emitter in `StellaOps.Zastava.Observer` (callgraph-aware NDJSON publish + subject derivation); added reachability options and unit tests; set 201-001 and GAP-ZAS-002 to DONE. | Zastava Observer Guild | diff --git a/docs/implplan/SPRINT_0512_0001_0001_bench.md b/docs/implplan/SPRINT_0512_0001_0001_bench.md index c98c0b817..e978b3ed1 100644 --- a/docs/implplan/SPRINT_0512_0001_0001_bench.md +++ b/docs/implplan/SPRINT_0512_0001_0001_bench.md @@ -25,8 +25,8 @@ | P4 | PREP-BENCH-POLICY-20-002-POLICY-DELTA-SAMPLE | DONE (2025-11-20) | Due 2025-11-26 · Accountable: Bench Guild · Policy Guild · Scheduler Guild | Bench Guild · Policy Guild · Scheduler Guild | Prep artefact published at `docs/benchmarks/policy/bench-policy-20-002-prep.md` (baseline + delta datasets, deterministic harness plan, metrics). | | P5 | PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX | DONE (2025-11-20) | Prep doc at `docs/benchmarks/signals/bench-sig-26-001-prep.md`; awaits reachability schema hash. | Bench Guild · Signals Guild | Reachability schema/fixtures pending Sprint 0400/0401.

Document artefact/deliverable for BENCH-SIG-26-001 and publish location so downstream tasks can proceed. | | P6 | PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU | DONE (2025-11-20) | Prep doc at `docs/benchmarks/signals/bench-sig-26-002-prep.md`; depends on 26-001 datasets. | Bench Guild · Policy Guild | Blocked on 26-001 outputs.

Document artefact/deliverable for BENCH-SIG-26-002 and publish location so downstream tasks can proceed. | -| 1 | BENCH-GRAPH-21-001 | DOING (2025-12-01) | PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN | Bench Guild · Graph Platform Guild | Build graph viewport/path benchmark harness (50k/100k nodes) measuring Graph API/Indexer latency, memory, and tile cache hit rates. | -| 2 | BENCH-GRAPH-21-002 | DOING (2025-12-01) | PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR | Bench Guild · UI Guild | Add headless UI load benchmark (Playwright) for graph canvas interactions to track render times and FPS budgets. | +| 1 | BENCH-GRAPH-21-001 | DONE (2025-12-02) | PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN | Bench Guild · Graph Platform Guild | Build graph viewport/path benchmark harness (50k/100k nodes) measuring Graph API/Indexer latency, memory, and tile cache hit rates. | +| 2 | BENCH-GRAPH-21-002 | DONE (2025-12-02) | PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR | Bench Guild · UI Guild | Add headless UI load benchmark (Playwright) for graph canvas interactions to track render times and FPS budgets. | | 3 | BENCH-GRAPH-24-002 | BLOCKED | Waiting for 50k/100k graph fixture (SAMPLES-GRAPH-24-003) | Bench Guild · UI Guild | Implement UI interaction benchmarks (filter/zoom/table operations) citing p95 latency; integrate with perf dashboards. | | 4 | BENCH-IMPACT-16-001 | BLOCKED | PREP-BENCH-IMPACT-16-001-IMPACT-INDEX-DATASET | Bench Guild · Scheduler Team | ImpactIndex throughput bench (resolve 10k productKeys) + RAM profile. | | 5 | BENCH-POLICY-20-002 | BLOCKED | PREP-BENCH-POLICY-20-002-POLICY-DELTA-SAMPLE | Bench Guild · Policy Guild · Scheduler Guild | Add incremental run benchmark measuring delta evaluation vs full; capture SLA compliance. | @@ -76,6 +76,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | Marked BENCH-GRAPH-21-001/002 DONE after overlay-capable harness, SHA capture, UI driver metadata, and deterministic tests; runs still use synthetic fixtures until SAMPLES-GRAPH-24-003 arrives. | Implementer | | 2025-11-27 | Added offline runner `Determinism/offline_run.sh` with manifest verification toggle; updated bench doc offline workflow. | Bench Guild | | 2025-11-27 | Added feeds placement note (`Determinism/inputs/feeds/README.md`) and linked in bench offline workflow. | Bench Guild | | 2025-11-27 | Added sample manifest `inputs/inputs.sha256` for bundled demo SBOM/VEX/config; documented in bench README and offline workflow. | Bench Guild | @@ -91,6 +92,7 @@ | 2025-12-01 | Generated interim synthetic graph fixtures (50k/100k nodes with manifests) under `samples/graph/interim/` to unblock BENCH-GRAPH-21-001; task moved to DOING pending overlay schema for canonical fixture. | Implementer | | 2025-12-01 | Added Graph UI bench scaffold: scenarios JSON, driver (`ui_bench_driver.mjs`), and plan under `src/Bench/StellaOps.Bench/Graph/`; BENCH-GRAPH-21-002 moved to DOING using interim fixtures until overlay schema/UI target is available. | Implementer | | 2025-12-01 | Added graph bench runner `Graph/run_graph_bench.sh` and recorded sample results for graph-50k/100k fixtures; BENCH-GRAPH-21-001 progressing with interim fixtures. | Implementer | +| 2025-12-02 | Extended graph bench harness with optional overlay support + SHA capture, updated UI driver to emit trace/viewport metadata, and added deterministic tests (`graph/tests/test_graph_bench.py`, `ui_bench_driver.test.mjs`). | Implementer | | 2025-11-22 | Added ACT-0512-07 and corresponding risk entry to have UI bench harness skeleton ready once fixtures bind; no status changes. | Project Mgmt | | 2025-11-22 | Added ACT-0512-04 to build interim synthetic graph fixture so BENCH-GRAPH-21-001 can start while awaiting SAMPLES-GRAPH-24-003; no status changes. | Project Mgmt | | 2025-11-22 | Added ACT-0512-05 escalation path (due 2025-11-23) if SAMPLES-GRAPH-24-003 remains unavailable; updated Upcoming Checkpoints accordingly. | Project Mgmt | diff --git a/docs/implplan/SPRINT_0513_0001_0001_public_reachability_benchmark.md b/docs/implplan/SPRINT_0513_0001_0001_public_reachability_benchmark.md index 96784c2ef..b96cd1192 100644 --- a/docs/implplan/SPRINT_0513_0001_0001_public_reachability_benchmark.md +++ b/docs/implplan/SPRINT_0513_0001_0001_public_reachability_benchmark.md @@ -32,7 +32,7 @@ | 4 | BENCH-CASES-PY-513-004 | DONE (2025-11-30) | Depends on 513-002. | Bench Guild · Python Track (`bench/reachability-benchmark/cases/py`) | Create 5-8 Python cases: Flask, Django, FastAPI. Include requirements.txt pinned, pytest oracles, coverage.py output. Delivered 5 cases: unsafe-exec (reachable), guarded-exec (unreachable), flask-template (reachable), fastapi-guarded (unreachable), django-ssti (reachable). | | 5 | BENCH-CASES-JAVA-513-005 | BLOCKED (2025-11-30) | Depends on 513-002. | Bench Guild · Java Track (`bench/reachability-benchmark/cases/java`) | Create 5-8 Java cases: Spring Boot, Micronaut. Include pom.xml locked, JUnit oracles, JaCoCo coverage. Progress: 2/5 seeded (`spring-deserialize` reachable, `spring-guarded` unreachable); build/test blocked by missing JDK (`javac` not available in runner). | | 6 | BENCH-CASES-C-513-006 | DONE (2025-12-01) | Depends on 513-002. | Bench Guild · Native Track (`bench/reachability-benchmark/cases/c`) | Create 3-5 C/ELF cases: small HTTP servers, crypto utilities. Include Makefile, gcov/llvm-cov coverage, deterministic builds (SOURCE_DATE_EPOCH). | -| 7 | BENCH-BUILD-513-007 | DOING | Depends on 513-003 through 513-006. | Bench Guild · DevOps Guild | Implement `build_all.py` and `validate_builds.py`: deterministic Docker builds, hash verification, SBOM generation (syft), attestation stubs. Progress: scripts now auto-emit deterministic SBOM/attestation stubs from `case.yaml`; validate checks auxiliary artifact determinism; SBOM swap-in for syft still pending. | +| 7 | BENCH-BUILD-513-007 | DONE (2025-12-02) | Depends on 513-003 through 513-006. | Bench Guild · DevOps Guild | Implement `build_all.py` and `validate_builds.py`: deterministic Docker builds, hash verification, SBOM generation (syft), attestation stubs. Progress: scripts now auto-emit deterministic SBOM/attestation stubs from `case.yaml`; validate checks auxiliary artifact determinism; SBOM swap-in for syft still pending. | | 8 | BENCH-SCORER-513-008 | DONE (2025-11-30) | Depends on 513-002. | Bench Guild (`bench/reachability-benchmark/tools/scorer`) | Implement `rb-score` CLI: load cases/truth, validate submissions, compute precision/recall/F1, explainability score (0-3), runtime stats, determinism rate. | | 9 | BENCH-EXPLAIN-513-009 | DONE (2025-11-30) | Depends on 513-008. | Bench Guild | Implement explainability scoring rules: 0=no context, 1=path with ≥2 nodes, 2=entry+≥3 nodes, 3=guards/constraints included. Unit tests for each level. | | 10 | BENCH-BASELINE-SEMGREP-513-010 | DONE (2025-12-01) | Depends on 513-008 and cases. | Bench Guild | Semgrep baseline runner: added `baselines/semgrep/run_case.sh`, `run_all.sh`, rules, and `normalize.py` to emit benchmark submissions deterministically (telemetry off, schema-compliant). | @@ -93,6 +93,7 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-12-02 | BENCH-BUILD-513-007: added optional Syft SBOM path with deterministic fallback stub, attestation/SBOM stub tests, and verified via `python bench/reachability-benchmark/tools/build/test_build_tools.py`. Status set to DONE. | Bench Guild | | 2025-11-27 | Sprint created from product advisory `24-Nov-2025 - Designing a Deterministic Reachability Benchmark.md`; 17 tasks defined across 5 waves. | Product Mgmt | | 2025-11-29 | BENCH-REPO-513-001 DONE: scaffolded `bench/reachability-benchmark/` with LICENSE (Apache-2.0), NOTICE, README, CONTRIBUTING, .gitkeep, and directory layout (cases/, schemas/, tools/scorer/, baselines/, ci/, website/, benchmark/truth, benchmark/submissions). | Implementer | | 2025-11-29 | BENCH-SCHEMA-513-002 DONE: expanded schemas (case/entrypoints/truth/submission), added examples + offline validator `tools/validate.py`, and pinned requirements for deterministic validation. | Implementer | diff --git a/docs/implplan/SPRINT_136_scanner_surface.md b/docs/implplan/SPRINT_136_scanner_surface.md index 83f343a95..17ef0896b 100644 --- a/docs/implplan/SPRINT_136_scanner_surface.md +++ b/docs/implplan/SPRINT_136_scanner_surface.md @@ -1,105 +1,3 @@ -# Sprint 136 - Scanner & Surface +# Legacy sprint file (redirect) -Implementation order remains sequential across Sprint 130–139. Complete each sprint in order before pulling tasks from the next file. - -## 7. Scanner.VII — Scanner & Surface focus on Scanner (phase VII). -Dependency: Sprint 135 - 6. Scanner.VI — Scanner & Surface focus on Scanner (phase VI). - -| Task ID | State | Summary | Owner / Source | Depends On | -| --- | --- | --- | --- | --- | -| `SCANNER-ENTRYTRACE-18-504` | DONE | EntryTrace NDJSON (entry/node/edge/target/warning/capability) emitted via EntryTraceNdjsonWriter; Worker stores and WebService/CLI stream NDJSON payloads. | EntryTrace Guild (src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace) | SCANNER-ENTRYTRACE-18-503 | -| `SCANNER-ENTRYTRACE-18-505` | DONE | ProcGraph replay integrated: runtime snapshot reconciler matches terminals/wrappers, adjusts plan confidence, and emits diagnostics for agreements/mismatches. | EntryTrace Guild (src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace) | SCANNER-ENTRYTRACE-18-504 | -| `SCANNER-ENTRYTRACE-18-506` | DONE | EntryTrace graph and confidence exposed via WebService `/scans/{id}/entrytrace` and CLI (`stella scan entrytrace`, NDJSON option) with target summaries. | EntryTrace Guild, Scanner WebService Guild (src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace) | SCANNER-ENTRYTRACE-18-505 | -| `SCANNER-ENV-01` | DONE (2025-11-18) | Worker already wired to `AddSurfaceEnvironment`/`ISurfaceEnvironment` for cache roots + CAS endpoints; no remaining ad-hoc env reads. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker) | — | -| `SCANNER-ENV-02` | DONE (2025-11-27) | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration. | Scanner WebService Guild, Ops Guild (src/Scanner/StellaOps.Scanner.WebService) | SCANNER-ENV-01 | -| `SCANNER-ENV-03` | DONE (2025-11-27) | Surface.Env package packed and mirrored to offline (`offline/packages/nugets`); wire BuildX to use 0.1.0-alpha.20251123 and update restore feeds. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin) | SCANNER-ENV-02 | -| `SURFACE-ENV-01` | DONE (2025-11-13) | Draft `surface-env.md` enumerating environment variables, defaults, and air-gap behaviour for Surface consumers. | Scanner Guild, Zastava Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env) | — | -| `SURFACE-ENV-02` | DONE (2025-11-18) | Strongly-typed env accessors implemented; validation covers required endpoint, bounds, TLS cert path; regression tests passing. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env) | SURFACE-ENV-01 | -| `SURFACE-ENV-03` | DONE (2025-11-27) | Adopt the env helper across Scanner Worker/WebService/BuildX plug-ins. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env) | SURFACE-ENV-02 | -| `SURFACE-ENV-04` | DONE (2025-11-27) | Wire env helper into Zastava Observer/Webhook containers. | Zastava Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env) | SURFACE-ENV-02 | -| `SURFACE-ENV-05` | DONE | Update Helm/Compose/offline kit templates with new env knobs and documentation. | Ops Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Env) | SURFACE-ENV-03, SURFACE-ENV-04 | -| `SCANNER-EVENTS-16-301` | BLOCKED (2025-10-26) | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService) | — | -| `SCANNER-GRAPH-21-001` | DONE (2025-11-27) | Provide webhook/REST endpoint for Cartographer to request policy overlays and runtime evidence for graph nodes, ensuring determinism and tenant scoping. | Scanner WebService Guild, Cartographer Guild (src/Scanner/StellaOps.Scanner.WebService) | — | -| `SCANNER-LNM-21-001` | BLOCKED (2025-11-27) | Update `/reports` and `/policy/runtime` payloads to consume advisory/vex linksets, exposing source severity arrays and conflict summaries alongside effective verdicts. Blocked: requires Concelier HTTP client integration or shared library; no existing Concelier dependency in Scanner WebService. | Scanner WebService Guild, Policy Guild (src/Scanner/StellaOps.Scanner.WebService) | — | -| `SCANNER-LNM-21-002` | TODO | Add evidence endpoint for Console to fetch linkset summaries with policy overlay for a component/SBOM, including AOC references. | Scanner WebService Guild, UI Guild (src/Scanner/StellaOps.Scanner.WebService) | SCANNER-LNM-21-001 | -| `SCANNER-SECRETS-03` | DONE (2025-11-27) | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. | BuildX Plugin Guild, Security Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin) | SCANNER-SECRETS-02 | -| `SURFACE-SECRETS-01` | DONE (2025-11-23) | Security-approved schema published at `docs/modules/scanner/design/surface-secrets-schema.md`; proceed to provider wiring. | Scanner Guild, Security Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | — | -| `SURFACE-SECRETS-02` | DONE (2025-11-23) | Provider chain implemented (primary + fallback) with DI wiring; tests updated (`StellaOps.Scanner.Surface.Secrets.Tests`). | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | SURFACE-SECRETS-01 | -| `SURFACE-SECRETS-03` | DONE (2025-11-27) | Add Kubernetes/File/Offline backends with deterministic caching and audit hooks. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | SURFACE-SECRETS-02 | -| `SURFACE-SECRETS-04` | DONE (2025-11-27) | Integrate Surface.Secrets into Scanner Worker/WebService/BuildX for registry + CAS creds. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | SURFACE-SECRETS-02 | -| `SURFACE-SECRETS-05` | DONE (2025-11-27) | Invoke Surface.Secrets from Zastava Observer/Webhook for CAS & attestation secrets. | Zastava Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | SURFACE-SECRETS-02 | -| `SURFACE-SECRETS-06` | BLOCKED (2025-11-27) | Update deployment manifests/offline kit bundles to provision secret references instead of raw values. Requires Ops Guild input on Helm/Compose patterns for Surface.Secrets provider configuration. | Ops Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets) | SURFACE-SECRETS-03 | -| `SCANNER-ENG-0020` | DONE (2025-11-28) | Implement Homebrew collector & fragment mapper per `design/macos-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0021` | DONE (2025-11-28) | Implement pkgutil receipt collector per `design/macos-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0022` | DONE (2025-11-28) | Implement macOS bundle inspector & capability overlays per `design/macos-analyzer.md` §3.3. | Scanner Guild, Policy Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0023` | DONE (2025-11-28) | Deliver macOS policy/offline integration per `design/macos-analyzer.md` §5–6. | Scanner Guild, Offline Kit Guild, Policy Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0024` | DONE (2025-11-28) | Implement Windows MSI collector per `design/windows-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0025` | DONE (2025-11-28) | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0026` | DONE (2025-11-28) | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Scanner Guild (docs/modules/scanner) | — | -| `SCANNER-ENG-0027` | DONE (2025-11-28) | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Scanner Guild, Policy Guild, Offline Kit Guild (docs/modules/scanner) | — | -| `SCHED-SURFACE-02` | TODO | Integrate Scheduler worker prefetch using Surface manifest reader and persist manifest pointers with rerun plans. | Scheduler Worker Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.Worker) | SURFACE-FS-02, SCHED-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §3 for implementation checklist | -| `ZASTAVA-SURFACE-02` | DONE (2025-12-01) | Surface manifest CAS/sha resolver wired into Observer drift evidence with failure metrics. | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer) | SURFACE-FS-02, ZASTAVA-SURFACE-01. Reference `docs/modules/scanner/design/surface-fs-consumers.md` §4 for integration steps | -| `SURFACE-FS-03` | DONE (2025-11-27) | Integrate Surface.FS writer into Scanner Worker analyzer pipeline to persist layer + entry-trace fragments. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SURFACE-FS-02 | -| `SURFACE-FS-04` | DONE (2025-11-27) | Integrate Surface.FS reader into Zastava Observer runtime drift loop. | Zastava Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SURFACE-FS-02 | -| `SURFACE-FS-05` | DONE (2025-11-27) | Expose Surface.FS pointers via Scanner WebService reports and coordinate rescan planning with Scheduler. | Scanner Guild, Scheduler Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SURFACE-FS-03 | -| `SURFACE-FS-06` | DONE (2025-11-28) | Update scanner-engine guide and offline kit docs with Surface.FS workflow. | Docs Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SURFACE-FS-02..05 | -| `SCANNER-SURFACE-04` | DONE (2025-12-02) | DSSE-sign every `layer.fragments` payload, emit `_composition.json`/`composition.recipe` URI, and persist DSSE envelopes so offline kits can replay deterministically (see `docs/modules/scanner/deterministic-sbom-compose.md` §2.1). | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker) | SCANNER-SURFACE-01, SURFACE-FS-03 | -| `SURFACE-FS-07` | TODO | Extend Surface.FS manifest schema with `composition.recipe`, fragment attestation metadata, and verification helpers per deterministic SBOM spec. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SCANNER-SURFACE-04 | -| `SURFACE-FS-07` | DONE (2025-12-02) | Surface.FS manifest schema now carries composition recipe/DSSE attestations and determinism metadata; determinism verifier added for offline replay. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS) | SCANNER-SURFACE-04 | -| `SCANNER-EMIT-15-001` | DOING (2025-12-01) | CycloneDX artifacts now carry content hash, merkle root (= recipe hash), composition recipe URI, and emit `_composition.json` + DSSE envelopes for recipe and layer fragments. DSSE signing is still deterministic-local; replace with real signing. | Scanner Emit Guild (src/Scanner/__Libraries/StellaOps.Scanner.Emit) | SCANNER-SURFACE-04 | -| `SCANNER-SORT-02` | DONE (2025-12-01) | Layer fragment ordering by digest implemented in ComponentGraphBuilder; determinism regression test added. | Scanner Core Guild (src/Scanner/__Libraries/StellaOps.Scanner.Core) | SCANNER-EMIT-15-001 | -| `SURFACE-VAL-01` | DONE (2025-11-23) | Validation framework doc aligned with Surface.Env release and secrets schema (`docs/modules/scanner/design/surface-validation.md` v1.1). | Scanner Guild, Security Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation) | SURFACE-FS-01, SURFACE-ENV-01 | -| `SURFACE-VAL-02` | DONE (2025-11-23) | Validation library now enforces secrets schema, fallback/provider checks, and inline/file guardrails; tests added. | Scanner Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation) | SURFACE-VAL-01, SURFACE-ENV-02, SURFACE-FS-02 | -| `SURFACE-VAL-03` | DONE (2025-11-23) | Validation runner wired into Worker/WebService startup and pre-analyzer paths (OS, language, EntryTrace). | Scanner Guild, Analyzer Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation) | SURFACE-VAL-02 | -| `SURFACE-VAL-04` | DONE (2025-11-27) | Expose validation helpers to Zastava and other runtime consumers for preflight checks. | Scanner Guild, Zastava Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation) | SURFACE-VAL-02 | -| `SURFACE-VAL-05` | DONE | Document validation extensibility, registration, and customization in scanner-engine guides. | Docs Guild (src/Scanner/__Libraries/StellaOps.Scanner.Surface.Validation) | SURFACE-VAL-02 | - -## Execution Log -| Date (UTC) | Update | Owner | -| --- | --- | --- | -| 2025-12-02 | SCANNER-SURFACE-04 completed: manifest stage emits composition recipe + DSSE envelopes, attaches attestations to artifacts, and records determinism Merkle root/recipe metadata. | Implementer | -| 2025-12-02 | SURFACE-FS-07 completed: Surface.FS manifest schema now includes determinism metadata, composition recipe attestation fields, determinism verifier, and docs updated. Targeted determinism tests added; test run pending due to long restore/build in monorepo runner. | Implementer | -| 2025-11-27 | Added missing package references to BuildX plugin (Configuration.EnvironmentVariables, DependencyInjection, Logging); refactored to use public AddSurfaceEnvironment API instead of internal SurfaceEnvironmentFactory; build passes. SCANNER-ENV-03 DONE. | Implementer | -| 2025-11-27 | Created SurfaceFeatureFlagsConfigurator to merge Surface.Env feature flags into WebService FeatureFlagOptions.Experimental dictionary; registered configurator in Program.cs. Cache roots and feature flags now wired from Surface.Env. SCANNER-ENV-02 DONE. | Implementer | -| 2025-11-27 | Verified SURFACE-ENV-03: Scanner Worker (SCANNER-ENV-01), WebService (SCANNER-ENV-02), and BuildX (SCANNER-ENV-03) all wire Surface.Env helpers; task complete. SURFACE-ENV-03 DONE. | Implementer | -| 2025-11-27 | Added CachingSurfaceSecretProvider (deterministic TTL cache), AuditingSurfaceSecretProvider (structured audit logging), and OfflineSurfaceSecretProvider (integrity-verified offline kit support); wired into ServiceCollectionExtensions with configurable options. SURFACE-SECRETS-03 DONE. | Implementer | -| 2025-11-27 | Added Surface.Validation project references to Zastava Observer and Webhook; wired AddSurfaceValidation() in service extensions for preflight checks. SURFACE-VAL-04 DONE. | Implementer | -| 2025-11-27 | Verified Zastava Observer and Webhook already have AddSurfaceEnvironment() wired with ZASTAVA prefixes; SURFACE-ENV-04 DONE. | Implementer | -| 2025-11-27 | Added Surface.Secrets project reference to BuildX plugin; implemented TryResolveAttestationToken() to fetch attestation secrets from Surface.Secrets; Worker/WebService already had configurators for CAS/registry/attestation secrets. SURFACE-SECRETS-04 DONE. | Implementer | -| 2025-11-27 | Verified Zastava Observer/Webhook already have ObserverSurfaceSecrets/WebhookSurfaceSecrets classes using ISurfaceSecretProvider for CAS and attestation secrets. SURFACE-SECRETS-05 DONE. | Implementer | -| 2025-11-27 | SURFACE-SECRETS-06 marked BLOCKED: requires Ops Guild input on Helm/Compose patterns for Surface.Secrets provider configuration (kubernetes/file/inline). Added to Decisions & Risks. | Implementer | -| 2025-11-27 | Integrated ISurfaceManifestWriter into SurfaceManifestStageExecutor to persist manifest documents to file-system store for offline/air-gapped scenarios; build verified. SURFACE-FS-03 DONE. | Implementer | -| 2025-11-27 | Added IRuntimeSurfaceFsClient injection to RuntimePostureEvaluator, enriching drift evidence with manifest digest/artifacts/metadata; added `zastava_surface_manifest_failures_total` metric with reason labels. SURFACE-FS-04 DONE. | Implementer | -| 2025-11-27 | Added TryResolveCasCredentials() to BuildX plugin using Surface.Secrets to fetch CAS access credentials; fixed attestation token resolution to use correct parser method. SCANNER-SECRETS-03 DONE. | Implementer | -| 2025-11-27 | Verified SurfacePointerService already exposes Surface.FS pointers (SurfaceManifestDocument, SurfaceManifestArtifact, manifest URI/digest) via reports endpoint. SURFACE-FS-05 DONE. | Implementer | -| 2025-11-27 | Added POST /policy/overlay endpoint for Cartographer integration: accepts graph nodes, returns deterministic overlays with sha256(tenant\|nodeId\|overlayKind) IDs, includes runtime evidence. Added PolicyOverlayRequestDto/ResponseDto contracts. SCANNER-GRAPH-21-001 DONE. | Implementer | -| 2025-11-27 | SCANNER-LNM-21-001 marked BLOCKED: Scanner WebService has no existing Concelier integration; requires HTTP client or shared library reference to Concelier.Core for linkset consumption. Added to Decisions & Risks. | Implementer | -| 2025-12-01 | EntryTrace NDJSON emission, runtime reconciliation, and WebService/CLI exposure completed (18-504/505/506). | EntryTrace Guild | -| 2025-12-01 | ZASTAVA-SURFACE-02: Observer resolves Surface manifest digests and `cas://` URIs, enriches drift evidence with artifact metadata, and counts failures via `zastava_surface_manifest_failures_total`. | Implementer | -| 2025-12-01 | SCANNER-SORT-02: ComponentGraphBuilder sorts layer fragments by digest; regression test added. | Implementer | -| 2025-12-01 | SCANNER-EMIT-15-001: CycloneDX artifacts now publish `ContentHash`, carry Merkle/recipe URIs, emit `_composition.json` + DSSE envelopes (recipe & layer.fragments), and Surface manifests reference those attestations. DSSE signer is pluggable (deterministic fallback registered); real signing still pending. | Implementer | -| 2025-12-01 | SCANNER-SORT-02 completed: ComponentGraphBuilder sorts layer fragments by digest with regression test Build_SortsLayersByDigest. | Implementer | -| 2025-12-01 | ZASTAVA-SURFACE-02: Observer now resolves Surface manifest digests and `cas://` URIs, enriches drift evidence with artifact metadata, and counts failures via `zastava_surface_manifest_failures_total`. | Implementer | -| 2025-11-23 | Published Security-approved Surface.Secrets schema (`docs/modules/scanner/design/surface-secrets-schema.md`); moved SURFACE-SECRETS-01 to DONE, SURFACE-SECRETS-02/SURFACE-VAL-01 to TODO. | Security Guild | -| 2025-11-23 | Implemented Surface.Secrets provider chain/fallback and added DI tests; marked SURFACE-SECRETS-02 DONE. | Scanner Guild | -| 2025-11-23 | Pinned Surface.Env package version `0.1.0-alpha.20251123` and offline path in `docs/modules/scanner/design/surface-env-release.md`; SCANNER-ENV-03 moved to TODO. | BuildX Plugin Guild | -| 2025-11-23 | Updated Surface.Validation doc to v1.1, binding to Surface.Env release and secrets schema; marked SURFACE-VAL-01 DONE. | Scanner Guild | -| 2025-11-23 | Strengthened Surface.Validation secrets checks (provider/fallback/inline/file root) and added unit tests; marked SURFACE-VAL-02 DONE. | Scanner Guild | -| 2025-11-23 | Added runtime validation gates to Worker/WebService startup and OS/Language/EntryTrace analyzer pipelines; marked SURFACE-VAL-03 DONE. | Scanner Guild | -| 2025-11-23 | Packed Surface.Env 0.1.0-alpha.20251123 and mirrored to `offline/packages/nugets`; SCANNER-ENV-03 now DOING for BuildX wiring. | BuildX Plugin Guild | -| 2025-11-23 | Wired SurfaceValidation runner into Worker/WebService startup to fail fast; SURFACE-VAL-03 in progress. | Scanner Guild | -| 2025-10-26 | Initial sprint plan captured; dependencies noted across Scheduler/Surface/Cartographer. | Planning | -| 2025-11-12 | SURFACE-ENV-01 done; SURFACE-ENV-02 started; SURFACE-SECRETS-01/02 in progress. | Scanner Guild | -| 2025-11-18 | SCANNER-ENV-01 in progress: added manifest store options configurator in Scanner Worker and unit scaffold (tests pending due to local restore/vstest issues). | Implementer | -| 2025-11-18 | SCANNER-ENV-02 started: wired Surface manifest store options into Scanner WebService and unit scaffold added; tests pending (nuget.org restore cancelled locally). | Implementer | -| 2025-11-18 | Attempted `dotnet test` for Worker Surface manifest configurator; restore failed fetching StackExchange.Redis from nuget.org (network timeout); tests still pending CI. | Implementer | -| 2025-11-18 | SCANNER-ENV-03 started: BuildX plugin now loads Surface.Env defaults (SCANNER/SURFACE prefixes) for cache root/bucket/tenant when args/env missing; tests not yet added. | Implementer | -| 2025-11-19 | Marked SCANNER-ENV-03, SURFACE-SECRETS-01/02, and SURFACE-VAL-01 BLOCKED pending Security/Surface schema approvals and published env/secrets artifacts; move back to TODO once upstream contracts land. | Implementer | -| 2025-11-28 | Created `docs/modules/scanner/guides/surface-validation-extensibility.md` covering custom validators, reporters, configuration, and testing; SURFACE-VAL-05 DONE. | Implementer | -| 2025-11-28 | Created `docs/modules/scanner/guides/surface-fs-workflow.md` with end-to-end workflow including artefact generation, storage layout, consumption, and offline kit handling; SURFACE-FS-06 DONE. | Implementer | -| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Homebrew` library with `HomebrewReceiptParser` (INSTALL_RECEIPT.json parsing), `HomebrewPackageAnalyzer` (Cellar discovery for Intel/Apple Silicon), and `HomebrewAnalyzerPlugin`; added `BuildHomebrew` PURL builder, `HomebrewCellar` evidence source; 23 tests passing. SCANNER-ENG-0020 DONE. | Implementer | -| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Pkgutil` library with `PkgutilReceiptParser` (plist parsing), `BomParser` (BOM file enumeration), `PkgutilPackageAnalyzer` (receipt discovery from /var/db/receipts), and `PkgutilAnalyzerPlugin`; added `BuildPkgutil` PURL builder, `PkgutilReceipt` evidence source; 9 tests passing. SCANNER-ENG-0021 DONE. | Implementer | -| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Msi` library with `MsiDatabaseParser` (OLE compound document parser), `MsiPackageAnalyzer` (Windows/Installer/*.msi discovery), and `MsiAnalyzerPlugin`; added `BuildWindowsMsi` PURL builder, `WindowsMsi` evidence source; 22 tests passing. SCANNER-ENG-0024 DONE. | Implementer | -| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.WinSxS` library with `WinSxSManifestParser` (XML assembly identity parser), `WinSxSPackageAnalyzer` (WinSxS/Manifests/*.manifest discovery), and `WinSxSAnalyzerPlugin`; added `BuildWindowsWinSxS` PURL builder, `WindowsWinSxS` evidence source; 18 tests passing. SCANNER-ENG-0025 DONE. | Implementer | -| 2025-11-28 | Created `StellaOps.Scanner.Analyzers.OS.Windows.Chocolatey` library with `NuspecParser` (nuspec + directory name fallback), `ChocolateyPackageAnalyzer` (ProgramData/Chocolatey/lib discovery), and `ChocolateyAnalyzerPlugin`; added `BuildChocolatey` PURL builder, `WindowsChocolatey` evidence source; 44 tests passing. SCANNER-ENG-0026 DONE. | Implementer | -| 2025-11-28 | Updated `docs/modules/scanner/design/windows-analyzer.md` with implementation status section documenting MSI/WinSxS/Chocolatey collector details, PURL formats, and vendor metadata schemas; registry collector deferred, policy predicates pending Policy module integration. SCANNER-ENG-0027 DONE. | Implementer | +This sprint was renamed to `SPRINT_0136_0001_0001_scanner_surface.md` on 2025-11-19 to comply with the standard filename template. Please update and read the canonical file instead. diff --git a/docs/implplan/SPRINT_165_timelineindexer.md b/docs/implplan/SPRINT_165_timelineindexer.md index 79cc4b024..6a2f1a993 100644 --- a/docs/implplan/SPRINT_165_timelineindexer.md +++ b/docs/implplan/SPRINT_165_timelineindexer.md @@ -1,31 +1,3 @@ -# Sprint 165 - Export & Evidence · 160.C) TimelineIndexer +# Legacy sprint file (redirect) -Active items only. Completed/historic work now resides in docs/implplan/archived/tasks.md (updated 2025-11-08). - -[Export & Evidence] 160.C) TimelineIndexer -Depends on: Sprint 110.A - AdvisoryAI, Sprint 120.A - AirGap, Sprint 130.A - Scanner, Sprint 150.A - Orchestrator -Summary: Export & Evidence focus on TimelineIndexer). -Task ID | State | Task description | Owners (Source) ---- | --- | --- | --- -TIMELINE-OBS-52-001 | TODO | Bootstrap `StellaOps.Timeline.Indexer` service with Postgres migrations for `timeline_events`, `timeline_event_details`, `timeline_event_digests`; enable RLS scaffolding and deterministic migration scripts. | Timeline Indexer Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) -TIMELINE-OBS-52-002 | TODO | Implement event ingestion pipeline (NATS/Redis consumers) with ordering guarantees, dedupe on `(event_id, tenant_id)`, correlation to trace IDs, and backpressure metrics. Dependencies: TIMELINE-OBS-52-001. | Timeline Indexer Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) -TIMELINE-OBS-52-003 | TODO | Expose REST/gRPC APIs for timeline queries (`GET /timeline`, `/timeline/{id}`) with filters, pagination, and tenant enforcement. Provide OpenAPI + contract tests. Dependencies: TIMELINE-OBS-52-002. | Timeline Indexer Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) -TIMELINE-OBS-52-004 | TODO | Finalize RLS policies, scope checks (`timeline:read`), and audit logging for query access. Include integration tests for cross-tenant isolation and legal hold markers. Dependencies: TIMELINE-OBS-52-003. | Timeline Indexer Guild, Security Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) -TIMELINE-OBS-53-001 | TODO | Link timeline events to evidence bundle digests + attestation subjects; expose `/timeline/{id}/evidence` endpoint returning signed manifest references. Dependencies: TIMELINE-OBS-52-004. | Timeline Indexer Guild, Evidence Locker Guild (src/TimelineIndexer/StellaOps.TimelineIndexer) - -## Task snapshot (2025-11-12) -- Core service: `TIMELINE-OBS-52-001/002` cover Postgres migrations/RLS scaffolding and NATS/Redis ingestion with deterministic ordering + metrics. -- API surface: `TIMELINE-OBS-52-003/004` expose REST/gRPC query endpoints, RLS policies, audit logging, and legal-hold tests. -- Evidence linkage: `TIMELINE-OBS-53-001` joins timeline events to EvidenceLocker digests for `/timeline/{id}/evidence`. - -## Dependencies & blockers -- Waiting on orchestrator + notifications schema (Wave 150/140) to finalize ingestion payload and event IDs. -- Requires EvidenceLocker bundle digest schema to link timeline entries to sealed manifests. -- Needs Scheduler/Orchestrator queue readiness for ingestion ordering semantics (impacting 52-002). -- Security/Compliance review required for Postgres RLS migrations before coding begins. - -## Ready-to-start checklist -1. Obtain sample orchestrator capsule events + notifications once schema drops; attach to this doc for reference. -2. Draft Postgres migration + RLS design and share with Security/Compliance for approval. -3. Define ingestion ordering tests (NATS to Postgres) and expected metrics/alerts. -4. Align evidence linkage contract with EvidenceLocker (bundle IDs, DSSE references) prior to implementing `TIMELINE-OBS-53-001`. +This sprint was renamed to `SPRINT_0165_0001_0001_timelineindexer.md` on 2025-11-19 to meet the standard filename template. Please consult the canonical file for all updates. diff --git a/docs/modules/concelier/link-not-merge-schema.md b/docs/modules/concelier/link-not-merge-schema.md index fa29ffca5..fb3376f44 100644 --- a/docs/modules/concelier/link-not-merge-schema.md +++ b/docs/modules/concelier/link-not-merge-schema.md @@ -9,6 +9,7 @@ _Frozen v1 (add-only) — approved 2025-11-17 for CONCELIER-LNM-21-001/002/101._ ## Status - Frozen v1 as of 2025-11-17; further schema changes must go through ADR + sprint gating (CONCELIER-LNM-22x+). +- Canonical JSON Schemas + signed manifest live in `docs/modules/concelier/schemas/` (advisory observation, linkset, offline bundle). Verify with `openssl dgst -sha256 -verify schema-signing-pub.pem -signature schema.manifest.sig schema.manifest.json`. ## Observation document (Mongo JSON Schema excerpt) ```json diff --git a/docs/modules/concelier/schemas/README.md b/docs/modules/concelier/schemas/README.md new file mode 100644 index 000000000..f8bbc5b0a --- /dev/null +++ b/docs/modules/concelier/schemas/README.md @@ -0,0 +1,32 @@ +# Concelier schema bundle (CI1–CI10 remediation) + +This folder publishes the signed JSON Schemas for Link-Not-Merge ingestion artifacts and the offline bundle manifest used by Offline Kit builds. + +- `advisory-observation.schema.json` — canonical observation shape (provenance + content hash enforced). +- `advisory-linkset.schema.json` — linkset materialization with conflict reasons and deterministic IDs. +- `offline-advisory-bundle.schema.json` — manifest for air-gapped advisory bundles, including staleness and signature metadata. +- `schema.manifest.json` — digest manifest over all schemas. +- `schema.manifest.sig` — detached ECDSA (P-256) signature over the manifest (public key: `schema-signing-pub.pem`). +- `schema.manifest.sig.b64` — base64 view of the signature for air-gapped copy/paste. +- `samples/` — deterministic sample payloads for CI fixtures (see `tests` notes below). + +## Verify locally (deterministic, offline) + +```bash +# 1) Validate schemas are unchanged +sha256sum -c schema.manifest.json + +# 2) Verify detached signature with the published public key +openssl dgst -sha256 -verify schema-signing-pub.pem \ + -signature schema.manifest.sig \ + schema.manifest.json +``` + +## Test coverage + +The fixtures in `samples/` are consumed by `StellaOps.Concelier.Core.Tests` to assert: +- deterministic idempotency keys and conflict ordering (`Linksets/AdvisoryLinksetIdempotencyTests`), +- tenant normalization and signature requirements for observations (`Aoc/AdvisoryObservationWriteGuardTests`), +- offline bundle manifest validation (`Schemas/OfflineBundleSchemaTests`). + +Keep the manifest and signature updated whenever schema files change. Keys are dev/test-only; production signing happens in the release pipeline. diff --git a/docs/modules/concelier/schemas/advisory-linkset.schema.json b/docs/modules/concelier/schemas/advisory-linkset.schema.json new file mode 100644 index 000000000..80db64d1b --- /dev/null +++ b/docs/modules/concelier/schemas/advisory-linkset.schema.json @@ -0,0 +1,85 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://stellaops.local/concelier/schemas/advisory-linkset.schema.json", + "title": "Concelier Advisory Linkset", + "type": "object", + "additionalProperties": false, + "required": [ + "linksetId", + "tenantId", + "advisoryId", + "source", + "observationIds", + "createdAt" + ], + "properties": { + "linksetId": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }, + "tenantId": { "type": "string", "minLength": 1 }, + "source": { "type": "string", "minLength": 1 }, + "advisoryId": { "type": "string", "minLength": 1 }, + "observationIds": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "uniqueItems": true, + "minItems": 1 + }, + "normalized": { + "type": ["object", "null"], + "additionalProperties": true + }, + "provenance": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "observationHashes": { + "type": "array", + "items": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }, + "uniqueItems": true + }, + "toolVersion": { "type": "string" }, + "policyHash": { "type": "string" } + } + }, + "confidence": { "type": ["number", "null"], "minimum": 0, "maximum": 1 }, + "conflicts": { + "type": ["array", "null"], + "items": { + "type": "object", + "additionalProperties": false, + "required": ["field", "reason"], + "properties": { + "field": { "type": "string" }, + "reason": { + "type": "string", + "enum": [ + "severity-mismatch", + "affected-range-divergence", + "reference-clash", + "alias-inconsistency", + "metadata-gap", + "statement-conflict" + ] + }, + "sourceIds": { + "type": ["array", "null"], + "items": { "type": "string" }, + "uniqueItems": true + } + } + } + }, + "aliases": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "primary": { "type": "string" }, + "others": { "type": "array", "items": { "type": "string" }, "uniqueItems": true } + } + }, + "purls": { "type": ["array", "null"], "items": { "type": "string" }, "uniqueItems": true }, + "cpes": { "type": ["array", "null"], "items": { "type": "string" }, "uniqueItems": true }, + "createdAt": { "type": "string", "format": "date-time" }, + "updatedAt": { "type": ["string", "null"], "format": "date-time" }, + "builtByJobId": { "type": ["string", "null"] } + } +} diff --git a/docs/modules/concelier/schemas/advisory-observation.schema.json b/docs/modules/concelier/schemas/advisory-observation.schema.json new file mode 100644 index 000000000..648025dc4 --- /dev/null +++ b/docs/modules/concelier/schemas/advisory-observation.schema.json @@ -0,0 +1,163 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://stellaops.local/concelier/schemas/advisory-observation.schema.json", + "title": "Concelier Advisory Observation (Link-Not-Merge)", + "type": "object", + "additionalProperties": false, + "required": [ + "observationId", + "tenant", + "source", + "upstream", + "content", + "linkset", + "rawLinkset", + "createdAt" + ], + "properties": { + "observationId": { "type": "string", "minLength": 1 }, + "tenant": { "type": "string", "minLength": 1, "pattern": "^[a-z0-9:-]+$" }, + "source": { + "type": "object", + "additionalProperties": false, + "required": ["vendor", "stream", "api"], + "properties": { + "vendor": { "type": "string", "minLength": 1 }, + "stream": { "type": "string", "minLength": 1 }, + "api": { "type": "string", "format": "uri" }, + "collectorVersion": { "type": "string" } + } + }, + "upstream": { + "type": "object", + "additionalProperties": false, + "required": [ + "upstreamId", + "fetchedAt", + "receivedAt", + "contentHash", + "signature" + ], + "properties": { + "upstreamId": { "type": "string", "minLength": 1 }, + "documentVersion": { "type": "string" }, + "fetchedAt": { "type": "string", "format": "date-time" }, + "receivedAt": { "type": "string", "format": "date-time" }, + "contentHash": { + "type": "string", + "pattern": "^sha256:[A-Fa-f0-9]{64}$" + }, + "signature": { + "type": "object", + "additionalProperties": false, + "required": ["present"], + "properties": { + "present": { "type": "boolean" }, + "format": { "type": "string" }, + "keyId": { "type": "string" }, + "signature": { "type": "string" } + }, + "allOf": [ + { + "if": { "properties": { "present": { "const": true } } }, + "then": { + "required": ["format", "keyId", "signature"], + "properties": { + "format": { "minLength": 1 }, + "keyId": { "minLength": 1 }, + "signature": { "minLength": 1 } + } + } + }, + { + "if": { "properties": { "present": { "const": false } } }, + "then": { + "properties": { + "format": { "maxLength": 0 }, + "keyId": { "maxLength": 0 }, + "signature": { "maxLength": 0 } + } + } + } + ] + }, + "metadata": { + "type": "object", + "additionalProperties": { "type": "string" }, + "propertyNames": { "pattern": "^[A-Za-z0-9_.:-]+$" } + } + } + }, + "content": { + "type": "object", + "additionalProperties": false, + "required": ["format", "raw"], + "properties": { + "format": { "type": "string", "minLength": 1 }, + "specVersion": { "type": "string" }, + "raw": { "type": ["object", "array"] }, + "metadata": { + "type": "object", + "additionalProperties": { "type": "string" }, + "propertyNames": { "pattern": "^[A-Za-z0-9_.:-]+$" } + } + } + }, + "linkset": { + "type": "object", + "additionalProperties": false, + "properties": { + "aliases": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }, + "purls": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }, + "cpes": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }, + "references": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["type", "url"], + "properties": { + "type": { "type": "string" }, + "url": { "type": "string", "format": "uri" } + } + } + }, + "reconciledFrom": { "type": "array", "items": { "type": "string" }, "uniqueItems": true } + } + }, + "rawLinkset": { + "type": "object", + "additionalProperties": false, + "properties": { + "aliases": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }, + "packageUrls": { "type": "array", "items": { "type": "string" } }, + "cpes": { "type": "array", "items": { "type": "string" } }, + "references": { + "type": "array", + "items": { + "type": "object", + "properties": { + "type": { "type": "string" }, + "url": { "type": "string" } + }, + "required": ["type", "url"], + "additionalProperties": false + } + }, + "relationships": { "type": "array", "items": { "type": "object" } }, + "reconciledFrom": { "type": "array", "items": { "type": "string" } }, + "scopes": { "type": "array", "items": { "type": "string" } }, + "notes": { + "type": "object", + "additionalProperties": { "type": "string" } + } + } + }, + "attributes": { + "type": "object", + "additionalProperties": { "type": "string" }, + "propertyNames": { "pattern": "^[A-Za-z0-9_.:-]+$" } + }, + "createdAt": { "type": "string", "format": "date-time" } + } +} diff --git a/docs/modules/concelier/schemas/offline-advisory-bundle.schema.json b/docs/modules/concelier/schemas/offline-advisory-bundle.schema.json new file mode 100644 index 000000000..94c986ddd --- /dev/null +++ b/docs/modules/concelier/schemas/offline-advisory-bundle.schema.json @@ -0,0 +1,102 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://stellaops.local/concelier/schemas/offline-advisory-bundle.schema.json", + "title": "Concelier Offline Advisory Bundle", + "type": "object", + "additionalProperties": false, + "required": [ + "bundleId", + "tenant", + "exportKind", + "snapshot", + "manifest", + "hashes", + "signatures", + "createdAt" + ], + "properties": { + "bundleId": { "type": "string", "pattern": "^bundle:[A-Za-z0-9._:-]+$" }, + "tenant": { "type": "string", "minLength": 1 }, + "exportKind": { "type": "string", "enum": ["json", "trivydb"] }, + "createdAt": { "type": "string", "format": "date-time" }, + "snapshot": { + "type": "object", + "additionalProperties": false, + "required": ["windowStart", "windowEnd", "sources"], + "properties": { + "windowStart": { "type": "string", "format": "date-time" }, + "windowEnd": { "type": "string", "format": "date-time" }, + "stalenessHours": { "type": "integer", "minimum": 0 }, + "sources": { + "type": "array", + "items": { + "type": "object", + "required": ["name", "cursor", "hash"], + "additionalProperties": false, + "properties": { + "name": { "type": "string" }, + "cursor": { "type": "string" }, + "hash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }, + "snapshotUri": { "type": "string", "format": "uri" } + } + }, + "uniqueItems": true + } + } + }, + "manifest": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["path", "sha256", "size"], + "properties": { + "path": { "type": "string" }, + "sha256": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" }, + "size": { "type": "integer", "minimum": 0 }, + "contentType": { "type": "string" } + } + }, + "uniqueItems": true + }, + "hashes": { + "type": "object", + "additionalProperties": false, + "patternProperties": { + "^sha256$": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" } + } + }, + "signatures": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["type", "keyId", "signature"], + "properties": { + "type": { "type": "string", "enum": ["dsse-inline", "detached"] }, + "keyId": { "type": "string" }, + "signature": { "type": "string" }, + "envelopeDigest": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }, + "rekor": { + "type": "object", + "additionalProperties": false, + "properties": { + "logIndex": { "type": "integer", "minimum": 0 }, + "uuid": { "type": "string" }, + "integratedTime": { "type": "integer", "minimum": 0 } + } + } + } + } + }, + "determinism": { + "type": "object", + "additionalProperties": false, + "properties": { + "contentHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }, + "idempotencyKey": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "canonVersion": { "type": "string", "default": "1" } + } + } + } +} diff --git a/docs/modules/concelier/schemas/samples/offline-advisory-bundle.sample.json b/docs/modules/concelier/schemas/samples/offline-advisory-bundle.sample.json new file mode 100644 index 000000000..5393ab442 --- /dev/null +++ b/docs/modules/concelier/schemas/samples/offline-advisory-bundle.sample.json @@ -0,0 +1,55 @@ +{ + "$schema": "../offline-advisory-bundle.schema.json", + "bundleId": "bundle:concelier:offline:2025-12-02T00-00Z", + "tenant": "default", + "exportKind": "json", + "createdAt": "2025-12-02T00:00:00Z", + "snapshot": { + "windowStart": "2025-11-25T00:00:00Z", + "windowEnd": "2025-12-01T23:59:59Z", + "stalenessHours": 168, + "sources": [ + { + "name": "osv", + "cursor": "2025-12-01T23:50:00Z", + "hash": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd", + "snapshotUri": "https://mirror.example/offline/osv-2025-12-01.zip" + }, + { + "name": "redhat", + "cursor": "2025-12-01T23:45:00Z", + "hash": "sha256:abcd456789abcdef0123456789abcdef0123456789abcdef0123456789abcd" + } + ] + }, + "manifest": [ + { + "path": "export/index.json", + "sha256": "89abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567", + "size": 482192, + "contentType": "application/json" + }, + { + "path": "export/db/trivy.db", + "sha256": "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210", + "size": 1289932, + "contentType": "application/octet-stream" + } + ], + "hashes": { + "sha256": "0f0e0d0c0b0a09080706050403020100ffeeddccbbaa99887766554433221100" + }, + "signatures": [ + { + "type": "dsse-inline", + "keyId": "schema-offline-pub", + "signature": "MEUCIQDkexampleSignedDigestx+deterministicSig==", + "envelopeDigest": "sha256:aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55" + } + ], + "determinism": { + "contentHash": "sha256:d3c3f6c75c6a3f0906bcee457cc77a2d6d7c0f9d1a1d7da78c0d2ab8e0dba111", + "idempotencyKey": "29d58b9fdc5c4e65b26c03f3bd9f442ff0c7f8514b8a9225f8b6417ffabc0101", + "canonVersion": "1" + } +} diff --git a/docs/modules/concelier/schemas/schema-signing-pub.pem b/docs/modules/concelier/schemas/schema-signing-pub.pem new file mode 100644 index 000000000..76fa856eb --- /dev/null +++ b/docs/modules/concelier/schemas/schema-signing-pub.pem @@ -0,0 +1,4 @@ +-----BEGIN PUBLIC KEY----- +MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyi7gVscxgRXQzX5ErNuQFN3dPjVw +YzU0JE3PGhjSinBwpODxtweLfP6zw2N6f0H9z25t8HwTpFeuk1PWqTX7Gg== +-----END PUBLIC KEY----- diff --git a/docs/modules/concelier/schemas/schema.manifest.json b/docs/modules/concelier/schemas/schema.manifest.json new file mode 100644 index 000000000..ce2e37350 --- /dev/null +++ b/docs/modules/concelier/schemas/schema.manifest.json @@ -0,0 +1,22 @@ +{ + "version": 1, + "generatedAt": "2025-12-02T00:00:00Z", + "files": [ + { + "path": "advisory-observation.schema.json", + "sha256": "e3f40aea09794f72f2722c46657377518489e2ca7e3122cfbb65655c3296c083" + }, + { + "path": "advisory-linkset.schema.json", + "sha256": "e3b40a0cca5aff85be2fbc5af9a96f00f5b7a20f6740a3f32947fae56bd599e5" + }, + { + "path": "offline-advisory-bundle.schema.json", + "sha256": "9b64af7c2e5fa0c071af7dc04b7984fd1787b4f9e2082cb47174610097e2dc51" + }, + { + "path": "samples/offline-advisory-bundle.sample.json", + "sha256": "15874bbafe5b2ead5ec9a853c32d715a4b48d41107ff2887d6ccdc222e462f45" + } + ] +} diff --git a/docs/modules/concelier/schemas/schema.manifest.sig b/docs/modules/concelier/schemas/schema.manifest.sig new file mode 100644 index 000000000..2b3398024 Binary files /dev/null and b/docs/modules/concelier/schemas/schema.manifest.sig differ diff --git a/docs/modules/concelier/schemas/schema.manifest.sig.b64 b/docs/modules/concelier/schemas/schema.manifest.sig.b64 new file mode 100644 index 000000000..05950e384 --- /dev/null +++ b/docs/modules/concelier/schemas/schema.manifest.sig.b64 @@ -0,0 +1,2 @@ +MEUCIBDcyrpqWmYQUrkWLTwMs6QyG2YWCFTxte10/7TobThlAiEAvqOSESmIxNFQ6pDtHlhpfL1K +1SZrDM+PhdAMSOMwoU4= diff --git a/docs/modules/findings-ledger/README.md b/docs/modules/findings-ledger/README.md index 07bc1b940..bc022a066 100644 --- a/docs/modules/findings-ledger/README.md +++ b/docs/modules/findings-ledger/README.md @@ -2,3 +2,9 @@ # Findings Ledger Start here for ledger docs. + +## Quick links +- FL1–FL10 remediation tracker: `gaps-FL1-FL10.md` +- Schema catalog (events/projections/exports): `schema-catalog.md` +- Merkle & external anchor policy: `merkle-anchor-policy.md` +- Tenant isolation & redaction manifest: `tenant-isolation-redaction.md` diff --git a/docs/modules/findings-ledger/dsse-policy-linkage.md b/docs/modules/findings-ledger/dsse-policy-linkage.md new file mode 100644 index 000000000..9fc1637a2 --- /dev/null +++ b/docs/modules/findings-ledger/dsse-policy-linkage.md @@ -0,0 +1,26 @@ +# DSSE & Policy Hash Linkage (FL6) + +**Goal:** Every export, replay report, and anchor manifest is tied to the exact policy digest that produced it and is verifiable offline via DSSE. + +## Binding rules +1. **Policy digest:** `policyVersion` (SHA-256 over policy bundle) is mandatory in ledger events, projections, exports, and replay reports. +2. **DSSE payload types** + - `application/vnd.stella-ledger-export+json` — export manifests (hashlist + filtersHash). + - `application/vnd.stella-ledger-anchor+json` — Merkle anchors (see `merkle-anchor-policy.md`). + - `application/vnd.stella-ledger-harness+json` — replay harness report. +3. **Hashlists:** export manifests contain `sha256` for each emitted NDJSON line (`lineDigest`), plus a dataset digest (`datasetSha256`) over concatenated line digests. Replay harness exposes `eventStreamChecksum` and `projectionChecksum`. +4. **Policy linkage:** DSSE payload must include `policyHash` and `schemaVersion` to prevent replay under mismatched policy versions. + +## Offline verification flow +1. Verify DSSE signature (local key or Rekor transparency log if online). +2. Recompute dataset checksum with `tools/LedgerReplayHarness/scripts/verify_export.py --input --expected `. +3. Cross-check `policyHash` in payload matches policy bundle in use; mismatch → block import/export. + +## File locations +- Harness DSSE placeholder now embeds `policyHash` when `LEDGER_POLICY_HASH` env var is set. +- Export manifests and checksums: `docs/modules/findings-ledger/golden-checksums.json`. +- External anchors: `docs/modules/findings-ledger/merkle-anchor-policy.md` (DSSE template). +- Set `LEDGER_POLICY_HASH` before running `tools/LedgerReplayHarness` to imprint the policy digest into the generated `.sig` file. + +## Change management +- Any change to payloadType or hash recipe bumps schema version in `schema-catalog.md` and requires new DSSE key roll announcement. diff --git a/docs/modules/findings-ledger/gaps-FL1-FL10.md b/docs/modules/findings-ledger/gaps-FL1-FL10.md new file mode 100644 index 000000000..a9099c163 --- /dev/null +++ b/docs/modules/findings-ledger/gaps-FL1-FL10.md @@ -0,0 +1,28 @@ +# Findings Ledger — FL1–FL10 Remediation (LEDGER-GAPS-121-009) + +**Source advisory:** `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md` +**Created:** 2025-12-02 · **Owner:** Findings Ledger Guild + +## Gap closure map + +| ID | Gap summary | Remediation artefact(s) | Evidence / notes | +| --- | ----------- | ----------------------- | ---------------- | +| FL1 | Versioned ledger event schema (canonical JSON + hashes) | `docs/modules/findings-ledger/schema-catalog.md` §1; updated `docs/modules/findings-ledger/schema.md` canonical rules | Canonical envelope `v1.0.0` stamped; hash derivation pinned to `sha256(canonicalJson)` + `sha256(eventHash-sequence)`. | +| FL2 | Projection schema versions + cycle hash determinism | `schema-catalog.md` §2; `schema.md` §4 | Projection `v1.0.0` with cycle-hash recipe and required fields; rebuild checksum guard in harness. | +| FL3 | Export schema (canonical/compact) + filter hash versioning | `schema-catalog.md` §3; golden fixtures under `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/` | Canonical export shape tagged `export.v1.canonical`; compact tagged `export.v1.compact`; fixtures hashed. | +| FL4 | Merkle + external anchor policy (Rekor/offline) | `docs/modules/findings-ledger/merkle-anchor-policy.md` | Anchoring cadence (1k/15m), Rekor/air-gap policy, anchor ref format, DSSE anchoring manifest. | +| FL5 | Tenant isolation + redaction manifest for exports/logs | `docs/modules/findings-ledger/tenant-isolation-redaction.md`; manifest: `docs/modules/findings-ledger/redaction-manifest.yaml` | Per-tenant partitions, export field redaction (comments, actor ids), signed manifest checksum. | +| FL6 | DSSE + policy hash linkage for exports and attestations | `docs/modules/findings-ledger/dsse-policy-linkage.md`; harness DSSE placeholder includes `policyHash` | Describes payloadType + bindings to policy digest and export hashlist. | +| FL7 | Deterministic export fixtures (golden) | `fixtures/golden/*.ndjson` (findings, vex, advisories, sboms) | Each includes `filtersHash`, `cycleHash`, `policyVersion`; hashes logged in manifest. | +| FL8 | Offline verifier script for bundles/exports | `tools/LedgerReplayHarness/scripts/verify_export.py` | Pure-Python, no deps; validates ordering, recomputes SHA-256 and optional expected hash file. | +| FL9 | Replay/rebuild checksum guard | Harness update: `tools/LedgerReplayHarness/Program.cs` (`--expected-checksum`) | Computes event-stream and projection checksums; fails on mismatch; emitted in report. | +| FL10 | Quotas/backpressure metrics and alerts | Metrics update: `Observability/LedgerMetrics.cs`; doc: `observability.md` §2/§4 | New counters `ledger_backpressure_applied_total`, gauge `ledger_quota_remaining`, alert guidance. | + +## How to verify +- Run `dotnet run --project tools/LedgerReplayHarness -- --fixture --connection --tenant --report out/report.json --metrics out/metrics.json --expected-checksum ` (use a file produced by a known-good run; template: `docs/modules/findings-ledger/replay-checksums.sample.json`). +- Validate exports: `python tools/LedgerReplayHarness/scripts/verify_export.py --input fixtures/golden/findings-canonical.ndjson --schema export.v1.canonical`. +- Check manifest hashes: `sha256sum docs/modules/findings-ledger/redaction-manifest.yaml fixtures/golden/*.ndjson`. + +## Follow-ons +- Integrate Rekor anchor publishing toggle into Helm/Compose overlays (tracked separately). +- Mirror golden fixtures into Offline Kit once export pipeline emits real data. diff --git a/docs/modules/findings-ledger/golden-checksums.json b/docs/modules/findings-ledger/golden-checksums.json new file mode 100644 index 000000000..058de0809 --- /dev/null +++ b/docs/modules/findings-ledger/golden-checksums.json @@ -0,0 +1,53 @@ +{ + "generatedAt": "2025-12-02T00:00:00Z", + "policyHash": "sha256:policy-v1", + "datasets": { + "findings-canonical.ndjson": { + "path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/findings-canonical.ndjson", + "schema": "export.v1.canonical", + "records": 2, + "filtersHash": "a81d6c6d2bcf9c0e7cbb1fcd292e4b7cc21f6d5c4e3f2b1a0c9d8e7f6c5b4a3e", + "sha256": "cd270235484748f2f4c871e9d574796e6f61b48df9cc65e009dab4ba0769dfa4" + }, + "vex-compact.ndjson": { + "path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/vex-compact.ndjson", + "schema": "export.v1.compact", + "records": 1, + "filtersHash": "b5c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb998877665544", + "sha256": "e786a12b4ee08776df73f7f2a97907280b5f8bb76cc7a901e2a680d3fe69e85e" + }, + "advisories-canonical.ndjson": { + "path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/advisories-canonical.ndjson", + "schema": "export.v1.canonical", + "records": 1, + "filtersHash": "c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb99887766554433", + "sha256": "6d5a2d522179b616c112c255c7dd06b3434ae0a4992009d25ea82f50144425ab" + }, + "sboms-compact.ndjson": { + "path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/sboms-compact.ndjson", + "schema": "export.v1.compact", + "records": 1, + "filtersHash": "d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb9988776655443322", + "sha256": "c89be7fcc511c4ef5a4a291c45061da1a7f4592506150e5b9bce92ba2bb5bbe2" + } + }, + "manifests": { + "redaction-manifest.yaml": { + "path": "docs/modules/findings-ledger/redaction-manifest.yaml", + "schema": "redaction.v1", + "sha256": "7c2f437a47c6514ad4688072b8b5e33b2e0cd0f9f289f15b49bf2f7def54a730" + }, + "redaction-manifest.json": { + "path": "docs/modules/findings-ledger/redaction-manifest.json", + "schema": "redaction.v1", + "sha256": "6965ea311f65482e6f51da0fd26cae1995997fcd456cea6dac84ab7b3354990a" + } + }, + "replay": { + "sample": { + "path": "docs/modules/findings-ledger/replay-checksums.sample.json", + "schema": "ledger.harness.v1", + "note": "replace with harness-produced checksums before enforcement" + } + } +} diff --git a/docs/modules/findings-ledger/merkle-anchor-policy.md b/docs/modules/findings-ledger/merkle-anchor-policy.md new file mode 100644 index 000000000..09d9695bd --- /dev/null +++ b/docs/modules/findings-ledger/merkle-anchor-policy.md @@ -0,0 +1,50 @@ +# Merkle & External Anchor Policy (FL4) + +**Audience:** Findings Ledger Guild · DevOps · Compliance +**Applies to:** `src/Findings/StellaOps.Findings.Ledger` (Merkle worker, anchoring jobs) + +## Anchoring cadence +- **Batch size:** 1,000 events or **15 minutes**, whichever is first (`LedgerServiceOptions:Merkle.BatchSize/WindowDuration`). +- **Tree:** flat Merkle over `merkle_leaf_hash` (see `schema-catalog.md` §1). Root hashed with SHA-256; no salt. +- **Partitions:** per-tenant batching only; no cross-tenant mixing. +- **Ordering:** leaves ordered by `(sequence_no, recorded_at)`. Any deviation is a failure. + +## Anchor references +- `ledger_merkle_roots.anchor_reference` formats: + - `rekor::` when pushed to Rekor. + - `airgap::` when sealed in offline bundle. + - `none` (empty) for internal-only anchors. +- External publication is optional but **must** include DSSE envelope with payload: + +```json +{ + "payloadType": "application/vnd.stella-ledger-anchor+json", + "payload": { + "tenant": "", + "rootHash": "", + "leafCount": 1000, + "windowStart": "2025-12-02T00:00:00Z", + "windowEnd": "2025-12-02T00:15:00Z", + "policyHash": "", + "schemaVersion": "ledger.event.v1" + }, + "signatures": [...] +} +``` + +## Determinism & recovery +- Anchor worker enforces stable ordering; replay harness recomputes Merkle roots and fails when root mismatch (FL9 guard). +- Root hash + DSSE signature are stored alongside export bundles for offline verification. +- External anchors **never** include tenant-identifying data beyond tenant id already present in ledger tables. + +## Air-gap posture +- Rekor publication optional; when disabled, anchors are sealed inside offline bundles with `anchor_reference=airgap::`. +- Anchor manifest is bundled in Offline Kit under `offline/ledger/anchors//.json`. +- No outbound network calls when `ExternalAnchoring:Enabled=false`. + +## Monitoring +- Metrics: `ledger_merkle_anchor_duration_seconds`, `ledger_merkle_anchor_failures_total`, `ledger_backpressure_applied_total{reason="anchoring"}`, `ledger_quota_remaining{kind="ingest"}`. +- Alerts: see `observability.md` (AnchorFailure + new Backpressure alert). + +## Change control +- Any change to batch size/window or hash recipe requires bumping `ledger.event` schema minor version and updating `schema-catalog.md`. diff --git a/docs/modules/findings-ledger/observability.md b/docs/modules/findings-ledger/observability.md index 91c8323cf..e96ceb12b 100644 --- a/docs/modules/findings-ledger/observability.md +++ b/docs/modules/findings-ledger/observability.md @@ -14,7 +14,10 @@ | --- | --- | --- | --- | | `ledger_write_duration_seconds` | Histogram | `tenant`, `event_type`, `source` | End-to-end append latency (API ingress → persisted). P95 ≤ 120 ms. | | `ledger_events_total` | Counter | `tenant`, `event_type`, `source` (`policy`, `workflow`, `orchestrator`) | Incremented per committed event. Mirrors Merkle leaf count. | -| `ledger_ingest_backlog_events` | Gauge | — | Number of events buffered in the writer/anchor queues. Alert when >5 000 for 5 min. | +| `ledger_ingest_backlog_events` | Gauge | `tenant` | Number of events buffered in the writer/anchor queues. Alert when >5 000 for 5 min. | +| `ledger_quota_remaining` | Gauge | `tenant` | Remaining ingest capacity before backpressure applies (defaults to 5 000 events). | +| `ledger_backpressure_applied_total` | Counter | `tenant`, `reason`, `limit` | Incremented whenever backlog crosses quota threshold. | +| `ledger_quota_rejections_total` | Counter | `tenant`, `reason` | Incremented when requests are actively rejected due to quotas. | | `ledger_projection_lag_seconds` | Gauge | `tenant` | Wall-clock difference between latest ledger event and projection tail. Target <30 s. | | `ledger_projection_rebuild_seconds` | Histogram | `tenant` | Duration of replay/rebuild operations triggered by LEDGER-29-008 harness. | | `ledger_projection_apply_seconds` | Histogram | `tenant`, `event_type`, `policy_version`, `evaluation_status` | Time to apply a single ledger event to projection. Target P95 <1 s. | @@ -43,6 +46,7 @@ | --- | --- | --- | | **LedgerWriteSLA** | `ledger_write_latency_seconds` P95 > 1 s for 3 intervals | Check DB contention, review queue backlog, scale writer. | | **LedgerBacklogGrowing** | `ledger_ingest_backlog_events` > 5 000 for 5 min | Inspect upstream policy runs, ensure projector keeping up. | +| **LedgerBackpressure** | `ledger_backpressure_applied_total` increases while `ledger_quota_remaining` < 0 | Throttle callers, raise quota or scale anchor worker. | | **ProjectionLag** | `ledger_projection_lag_seconds` > 30 s | Trigger rebuild, verify change streams. | | **AnchorFailure** | `ledger_merkle_anchor_failures_total` increase > 0 | Collect logs, rerun anchor, verify signing service. | | **AttachmentSecurityError** | `ledger_attachments_encryption_failures_total` increase > 0 | Audit attachments pipeline; check key material and storage endpoints. | diff --git a/docs/modules/findings-ledger/redaction-manifest.json b/docs/modules/findings-ledger/redaction-manifest.json new file mode 100644 index 000000000..4cf7c1ff8 --- /dev/null +++ b/docs/modules/findings-ledger/redaction-manifest.json @@ -0,0 +1,29 @@ +{ + "schemaVersion": "redaction.v1", + "generatedAt": "2025-12-02T00:00:00Z", + "owner": "findings-ledger-guild", + "rules": { + "ledger.event": [ + { "path": "$.actor.id", "action": "mask", "maskWith": "user:" }, + { "path": "$.payload.comment", "action": "drop" }, + { "path": "$.payload.ticket.url", "action": "drop" }, + { "path": "$.payload.attachments[*].downloadUrl", "action": "drop" } + ], + "export.canonical": [ + { "path": "$.actorId", "action": "mask", "maskWith": "user:" }, + { "path": "$.comment", "action": "drop" }, + { "path": "$.attachments", "action": "drop" } + ], + "export.compact": [ + { "path": "$.actorId", "action": "drop" }, + { "path": "$.comment", "action": "drop" }, + { "path": "$.policyRationale", "action": "drop" }, + { "path": "$.attachments", "action": "drop" }, + { "path": "$.labels", "action": "drop" } + ], + "observability": [ + { "path": "$.event_body", "action": "drop" }, + { "path": "$.actor_id", "action": "hash", "hashWith": "sha256" } + ] + } +} diff --git a/docs/modules/findings-ledger/redaction-manifest.yaml b/docs/modules/findings-ledger/redaction-manifest.yaml new file mode 100644 index 000000000..37cc0b9c7 --- /dev/null +++ b/docs/modules/findings-ledger/redaction-manifest.yaml @@ -0,0 +1,39 @@ +schemaVersion: redaction.v1 +generatedAt: 2025-12-02T00:00:00Z +owner: findings-ledger-guild +rules: + ledger.event: + - path: $.actor.id + action: mask + maskWith: "user:" + - path: $.payload.comment + action: drop + - path: $.payload.ticket.url + action: drop + - path: $.payload.attachments[*].downloadUrl + action: drop + export.canonical: + - path: $.actorId + action: mask + maskWith: "user:" + - path: $.comment + action: drop + - path: $.attachments + action: drop + export.compact: + - path: $.actorId + action: drop + - path: $.comment + action: drop + - path: $.policyRationale + action: drop + - path: $.attachments + action: drop + - path: $.labels + action: drop + observability: + - path: $.event_body + action: drop + - path: $.actor_id + action: hash + hashWith: sha256 diff --git a/docs/modules/findings-ledger/replay-checksums.sample.json b/docs/modules/findings-ledger/replay-checksums.sample.json new file mode 100644 index 000000000..7bef2feec --- /dev/null +++ b/docs/modules/findings-ledger/replay-checksums.sample.json @@ -0,0 +1,5 @@ +{ + "eventStream": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "projection": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "notes": "Replace with real values from harness output before enforcing checksum guard." +} diff --git a/docs/modules/findings-ledger/schema-catalog.md b/docs/modules/findings-ledger/schema-catalog.md new file mode 100644 index 000000000..119a4f4e7 --- /dev/null +++ b/docs/modules/findings-ledger/schema-catalog.md @@ -0,0 +1,75 @@ +# Findings Ledger Schema Catalog (FL1–FL3) + +**Scope:** Versioned canonical schemas for ledger events, projections, and exports. +**Status:** v1.0.0 sealed (2025-12-02) — breaking changes require new minor/major version tags. + +## 1) Ledger event envelope — `ledger.event.v1` + +| Field | Type | Notes | +| --- | --- | --- | +| `event.id` | `uuid` | V7 GUID allowed. | +| `event.type` | `string` (`ledger_event_type`) | See `schema.md` §2.2. | +| `event.tenant` | `string` | Partition key. | +| `event.chainId` | `uuid` | Derived when absent (`tenant :: policyVersion`), see `workflow-inference.md`. | +| `event.sequence` | `long` | Gapless per chain, starts at 1. | +| `event.policyVersion` | `string` | SHA-256 digest of policy bundle; propagated into exports and DSSE. | +| `event.finding` | object | `id`, `artifactId`, `vulnId`. | +| `event.actor` | object | `id`, `type` (`system|operator|integration`). | +| `event.occurredAt` | `string` (UTC ISO-8601 ms) | Domain clock. | +| `event.recordedAt` | `string` (UTC ISO-8601 ms) | Service `TimeProvider`. | +| `event.payload` | object | Mutation-specific body. | +| `event.evidenceBundleRef` | `string?` | DSSE/capsule id (optional). | +| `event.airgap.bundle` | object? | See `airgap-provenance.md`. | +| `event_hash` | `char(64)` | `sha256(canonicalJson)` lower-hex. | +| `previous_hash` | `char(64)` | All-zero for chain genesis. | +| `merkle_leaf_hash` | `char(64)` | `sha256(event_hash || "-" || sequence)`. | + +Canonicalisation: UTF-8, sorted keys, lower-case enums, ISO-8601 UTC with millisecond precision, arrays stable-order. Any field addition bumps minor version. + +## 2) Finding projection — `ledger.projection.v1` + +| Field | Type | Notes | +| --- | --- | --- | +| `tenantId` | `string` | Partition key. | +| `findingId` | `string` | Stable identity. | +| `policyVersion` | `string` | Hash of active policy bundle. | +| `status` | `string` | `affected|triaged|accepted_risk|resolved|unknown`. | +| `severity` | `number` | 0–10, 3 decimal places. | +| `riskScore` | `number` | 0–10, 3 decimal places. | +| `riskSeverity` | `string` | `low|medium|high|critical|unknown`. | +| `riskProfileVersion` | `string` | Version/hash from Risk Engine. | +| `riskExplanationId` | `uuid?` | Links to explain bundle. | +| `labels` | `json` | KEV/runtime flags, sorted keys. | +| `currentEventId` | `uuid` | Source ledger event. | +| `explainRef` | `string?` | Object storage / DSSE reference. | +| `policyRationale` | `json` | Array of rationale refs. | +| `updatedAt` | `string` UTC | Projection timestamp. | +| `cycleHash` | `char(64)` | `sha256(canonicalProjectionJson)`; used in exports. | + +Projection deterministic hash recipe: serialize projection record with sorted keys (excluding `updatedAt` jitter) and hash via SHA-256. The replay harness recomputes and compares. + +## 3) Export payloads — `export.v1` + +Shapes share headers: `policyVersion`, `projectionVersion` (cycle hash), `filtersHash`, `pageToken`, `observedAt`, `provenance` (`ledgerRoot`, `projectorVersion`, `policyHash`, optional `dsseDigest`). + +### Canonical vs compact +- **Canonical (`export.v1.canonical`)** — full provenance fields, evidence refs, DSSE linkage. +- **Compact (`export.v1.compact`)** — drops verbose fields (`policyRationale`, comments, actor ids), keeps `cycleHash` + `filtersHash` for determinism; redaction manifest enforced. + +### Record fields +- Findings: `findingId`, `eventSequence`, `status`, `severity`, `risk`, `advisories[]`, `evidenceBundleRef`, `cycleHash`. +- VEX: `vexStatementId`, `product`, `status`, `justification`, `knownExploited`, `cycleHash`. +- Advisories: `advisoryId`, `source`, `cvss{version,vector,baseScore}`, `epss`, `kev`, `cycleHash`. +- SBOMs: `sbomId`, `subject{digest,mediaType}`, `sbomFormat`, `componentsCount`, `materials[]`, `cycleHash`. + +Filters hash: `sha256(sortedQueryString)`; stored alongside fixtures for replayability. + +## 4) Versioning rules +- Patch: backward-compatible field additions (new optional key) — bump patch digit. +- Minor: additive required fields or canonical rule tweaks — bump minor. +- Major: breaking change (field removal/rename, hash recipe) — bump major and keep prior schema frozen. + +## 5) Reference artefacts +- Golden fixtures: `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/*.ndjson`. +- Checksum manifest: `docs/modules/findings-ledger/golden-checksums.json`. +- Offline verifier: `tools/LedgerReplayHarness/scripts/verify_export.py`. diff --git a/docs/modules/findings-ledger/schema.md b/docs/modules/findings-ledger/schema.md index 2dbfdae7c..53f994234 100644 --- a/docs/modules/findings-ledger/schema.md +++ b/docs/modules/findings-ledger/schema.md @@ -119,6 +119,11 @@ Canonicalisation rules: 5. Numbers use decimal notation; omit trailing zeros. 6. Arrays maintain supplied order. +### 2.4 Versioning & DSSE linkage (FL1, FL6) +- Canonical schema identifiers are catalogued in `schema-catalog.md` (`ledger.event.v1`, `ledger.projection.v1`, `export.v1.*`). +- Any change to the envelope, hash recipe, or required fields bumps the catalog version; legacy versions remain frozen. +- DSSE artefacts (anchors, exports, replay reports) **must** embed `policyVersion` and `schemaVersion` (see `dsse-policy-linkage.md`). + Hash pipeline: ``` @@ -270,7 +275,7 @@ Ordering and pagination: `ORDER BY recorded_at ASC, attestation_id ASC` with cur 1. Canonical serialize the envelope (§2.3). 2. Compute `event_hash` and store along with `previous_hash`. 3. Build Merkle tree per anchoring window using leaf hash `SHA256(event_hash || '-' || sequence_no)`. -4. Persist root in `ledger_merkle_roots` and, when configured, submit to external transparency log (Rekor v2). Store receipt/UUID in `anchor_reference`. +4. Persist root in `ledger_merkle_roots` and, when configured, submit to external transparency log (Rekor v2). Store receipt/UUID in `anchor_reference` (see `merkle-anchor-policy.md`). 5. Projection rows compute `cycle_hash = SHA256(canonical_projection_json)` where canonical projection includes fields `{tenant_id, finding_id, policy_version, status, severity, labels, current_event_id}` with sorted keys. Verification flow for auditors: @@ -284,6 +289,8 @@ Verification flow for auditors: - Initial migration script: `src/Findings/StellaOps.Findings.Ledger/migrations/001_initial.sql`. - Sample canonical event: `seed-data/findings-ledger/fixtures/ledger-event.sample.json` (includes pre-computed `eventHash`, `previousHash`, and `merkleLeafHash` values). - Sample projection row: `seed-data/findings-ledger/fixtures/finding-projection.sample.json` (includes canonical `cycleHash` for replay validation). +- Golden export fixtures (FL7): `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/*.ndjson` with checksums in `docs/modules/findings-ledger/golden-checksums.json`. +- Redaction manifest (FL5): `docs/modules/findings-ledger/redaction-manifest.yaml` governs mask/drop rules for canonical vs compact exports. Fixtures follow canonical key ordering and include precomputed hashes to validate tooling. diff --git a/docs/modules/findings-ledger/tenant-isolation-redaction.md b/docs/modules/findings-ledger/tenant-isolation-redaction.md new file mode 100644 index 000000000..ad041e80b --- /dev/null +++ b/docs/modules/findings-ledger/tenant-isolation-redaction.md @@ -0,0 +1,28 @@ +# Tenant Isolation & Redaction Manifest (FL5) + +**Purpose:** Document how Findings Ledger enforces tenant boundaries and which fields are redacted in deterministic exports. + +## Isolation controls +- Storage: all ledger, projection, history, and merkle tables are **LIST-partitioned by `tenant_id`** (PostgreSQL). Cross-tenant queries are disallowed at repo level. +- Queueing: Merkle batches and projector pipelines are keyed by `(tenant_id, chain_id)`; no mixing. +- Exports: `/ledger/export/*` requires `X-Stella-Tenant`; service rejects multi-tenant requests. +- Hashing: event/projection hashes include `tenant_id` as part of canonical envelope, preventing replay across tenants. + +## Redaction policy +- User-generated content (comments, attachments metadata) is excluded from compact exports and masked in canonical exports per manifest. +- Actor identifiers are truncated to realm (`user:`); emails/PII never emitted. +- Evidence bundle references are retained, but inline evidence payloads are not stored in ledger. + +## Manifest +- Path: `docs/modules/findings-ledger/redaction-manifest.yaml` (JSON twin: `redaction-manifest.json` for offline tooling). +- Content: declarative list of fields redacted or truncated for each export shape. +- The manifest is signed in checksum list `docs/modules/findings-ledger/golden-checksums.json`; sha256 must match before release. + +### Applying the manifest +- Canonical exports apply `redact: mask` rules only to PII (`actorId`, `comment`); compact exports drop (`drop: true`) the same fields plus verbose rationale arrays. +- Log pipelines ensure `event_body` is never written to logs; only metadata/hashes appear (see `observability.md`). + +## Validation steps +1. `sha256sum docs/modules/findings-ledger/redaction-manifest.yaml` matches `golden-checksums.json`. +2. Run `python tools/LedgerReplayHarness/scripts/verify_export.py --input fixtures/golden/findings-canonical.ndjson --schema export.v1.canonical --manifest docs/modules/findings-ledger/redaction-manifest.json` (script enforces mask/drop rules offline). +3. Confirm export responses in staging omit masked fields for the requesting tenant. diff --git a/docs/modules/mirror/dsse-tuf-profile.md b/docs/modules/mirror/dsse-tuf-profile.md index 5461466e9..2c57a464f 100644 --- a/docs/modules/mirror/dsse-tuf-profile.md +++ b/docs/modules/mirror/dsse-tuf-profile.md @@ -12,6 +12,7 @@ Applies to `mirror-thin-v1.*` artefacts in `out/mirror/thin/`. - Payload: `mirror-thin-v1.manifest.json` - Signature: ed25519 over base64url(payload) - Envelope path: `out/mirror/thin/mirror-thin-v1.manifest.dsse.json` +- Bundle meta DSSE (OK1/OK3/MS8): payload type `application/vnd.stellaops.mirror.bundle+json`, payload `mirror-thin-v1.bundle.json`, envelope path `mirror-thin-v1.bundle.dsse.json`. ## TUF metadata layout ``` @@ -23,9 +24,10 @@ out/mirror/thin/tuf/ keys/mirror-ed25519-test-1.pub ``` -### Targets mapping -- `mirror-thin-v1.tar.gz` → targets entry with sha256 `210dc49e8d3e25509298770a94da277aa2c9d4c387d3c24505a61fe1d7695a49` -- `mirror-thin-v1.manifest.json` → sha256 `0ae51fa87648dae0a54fab950181a3600a8363182d89ad46d70f3a56b997b504` +### Targets mapping (latest dev build 2025-12-02) +- `mirror-thin-v1.tar.gz` → targets entry with sha256 `fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d` +- `mirror-thin-v1.manifest.json` → sha256 `1affb0b796ff037117b46aa1f1d8056a9c80755e925af058ea72132ba158becf` +- `mirror-thin-v1.bundle.json` (top-level kit manifest) → sha256 `a3b16f5d1b74ffdf9aedbbfe9282d368dc3dcf70676c8ac7e8cdd984162e7f90` ### Determinism rules - Sort keys in JSON; indent=2; trailing newline. diff --git a/docs/modules/mirror/signing-runbook.md b/docs/modules/mirror/signing-runbook.md index 69b5dfbd9..38180dac2 100644 --- a/docs/modules/mirror/signing-runbook.md +++ b/docs/modules/mirror/signing-runbook.md @@ -12,6 +12,8 @@ MIRROR_SIGN_KEY_B64: ${{ secrets.MIRROR_SIGN_KEY_B64 }} REQUIRE_PROD_SIGNING: 1 OCI: 1 + TENANT_SCOPE: tenant-demo + ENV_SCOPE: lab run: | scripts/mirror/check_signing_prereqs.sh scripts/mirror/ci-sign.sh @@ -40,7 +42,9 @@ MIRROR_SIGN_KEY_B64=LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1DNENBUUF3QlFZREsyVndC **Do not ship with this key.** Set `REQUIRE_PROD_SIGNING=1` for release/tag builds so they fail without the real key. Add the production key as a Gitea secret (`MIRROR_SIGN_KEY_B64`) and rerun the workflow; remove this temporary key block once rotated. ## Verification -The CI step already runs `scripts/mirror/verify_thin_bundle.py`. For OCI, ensure `out/mirror/thin/oci/index.json` references the manifest digest. +The CI step already runs `scripts/mirror/verify_thin_bundle.py --bundle-meta mirror-thin-v1.bundle.json --tenant $TENANT_SCOPE --environment $ENV_SCOPE --pubkey out/mirror/thin/tuf/keys/ci-ed25519.pub` so offline-kit policies (OK1–OK10), Rekor policy (RK1–RK10), and mirror-format policy (MS1–MS10) are validated alongside the tarball. For OCI, ensure `out/mirror/thin/oci/index.json` references the manifest digest. + +`milestone.json` now carries manifest/tar/bundle/bundle-dsse hashes plus policy layer hashes to allow air-gapped import verification. ## Fallback (if secret absent) - CI can fall back to an embedded test Ed25519 key when `MIRROR_SIGN_KEY_B64` is unset **only when `REQUIRE_PROD_SIGNING` is not set**. This is for dev smoke runs; release/tag jobs must set `REQUIRE_PROD_SIGNING=1` to forbid fallback. diff --git a/docs/modules/policy/architecture.md b/docs/modules/policy/architecture.md index 3ec85c1b0..9eb097c65 100644 --- a/docs/modules/policy/architecture.md +++ b/docs/modules/policy/architecture.md @@ -112,7 +112,7 @@ Key notes: | **DSL Compiler** (`Dsl/`) | Parse, canonicalise, IR generation, checksum caching. | Uses Roslyn-like pipeline; caches by `policyId+version+hash`. | | **Selection Layer** (`Selection/`) | Batch SBOM ↔ advisory ↔ VEX joiners; apply equivalence tables; support incremental cursors. | Deterministic ordering (SBOM → advisory → VEX). | | **Evaluator** (`Evaluation/`) | Execute IR with first-match semantics, compute severity/trust/reachability weights, record rule hits. | Stateless; all inputs provided by selection layer. | -| **Signals** (`Signals/`) | Normalizes reachability, trust, entropy, uncertainty, runtime hits into a single dictionary passed to Evaluator; supplies default `unknown` values when signals missing. | Aligns with `signals.*` namespace in DSL. | +| **Signals** (`Signals/`) | Normalizes reachability, trust, entropy, uncertainty, runtime hits into a single dictionary passed to Evaluator; supplies default `unknown` values when signals missing. Entropy penalties are derived from Scanner `layer_summary.json`/`entropy.report.json` (K=0.5, cap=0.3, block at image opaque ratio > 0.15 w/ unknown provenance) and exported via `policy_entropy_penalty_value` / `policy_entropy_image_opaque_ratio`. | Aligns with `signals.*` namespace in DSL. | | **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | Mongo transactions per SBOM chunk. | | **Orchestrator** (`Runs/`) | Change-stream ingestion, fairness, retry/backoff, queue writer. | Works with Scheduler Models DTOs. | | **API** (`Api/`) | Minimal API endpoints, DTO validation, problem responses, idempotency. | Generated clients for CLI/UI. | diff --git a/docs/modules/zastava/README.md b/docs/modules/zastava/README.md index ccfaddb0e..71d9516b1 100644 --- a/docs/modules/zastava/README.md +++ b/docs/modules/zastava/README.md @@ -2,7 +2,8 @@ Zastava monitors running workloads, verifies supply chain posture, and enforces runtime policy via Kubernetes admission webhooks. -## Latest updates (2025-11-30) +## Latest updates (2025-12-02) +- DSSE-signed schemas, thresholds, exports, and deterministic `zastava-kit` bundle published under `docs/modules/zastava`; verification via `kit/verify.sh` and hashes in `SHA256SUMS`. - Sprint tracker `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md` and module `TASKS.md` added to mirror status. - Observability runbook stub + dashboard placeholder added under `operations/` (offline import). - Surface.Env/Surface.Secrets adoption remains pending platform contracts; align with platform docs before enabling sealed mode. diff --git a/docs/modules/zastava/SHA256SUMS b/docs/modules/zastava/SHA256SUMS index 7dc795656..a4da92ea7 100644 --- a/docs/modules/zastava/SHA256SUMS +++ b/docs/modules/zastava/SHA256SUMS @@ -1,7 +1,17 @@ -e65d4b68c9bdaa569c6d4c5a9b0a8bc1dc41876f948983011ff6f9d3466565d0 schemas/observer_event.schema.json -f466bf2b399f065558867eaf3c961cff8803f4a1506bae5539c9ce62e9ab005d schemas/webhook_admission.schema.json +1b05f31ab9486f9a03ecf872fa5c681e9acbad2adb71a776c271dbcf997ca2a8 schemas/observer_event.schema.json +99382de0e6a2b9c21146c03640c2e08b0e5e41be11fdbc213f0f071357da5a99 schemas/observer_event.schema.json.dsse +222db5258f5ba1ee720f8df03858263363b8636ff8ec9370f5ad390e8def0b3c schemas/webhook_admission.schema.json +19f108da1a512a488536bc2cd9d9cb1cf9824d748d8fc6a32d0e31c89be9a897 schemas/webhook_admission.schema.json.dsse +da065beabf8e038298a54f04ffa3e140cc149e0d64c301f6fd4c3925f2d64ee6 schemas/examples/observer_event.example.json +7e3cd0c18c9dfaf9001a16a99be7f9ff01e2d21b14eca9fb97c332342ac53c94 schemas/examples/webhook_admission.example.json +e17d36a2a39d748b76994ad3e3e4f3fa8db1b9298a3ce5eaaafb575791c01da3 schemas/README.md +f88bdebaa9858ffe3cd0fbb46e914c933e18709165bfc59f976136097fa8493d exports/observer_events.ndjson +de9b24675a0a758e40647844a31a13a1be1667750a39fe59465b0353fd0dddd9 exports/observer_events.ndjson.dsse +232809cf6a1cc7ba5fa34e0daf00fab9b6f970a613bc822457eef0d841fb2229 exports/webhook_admissions.ndjson +0edf6cabd636c7bb1f210af2aecaf83de3cc21c82113a646429242ae72618b17 exports/webhook_admissions.ndjson.dsse 40fabd4d7bc75c35ae063b2e931e79838c79b447528440456f5f4846951ff59d thresholds.yaml -652fce7d7b622ae762c8fb65a1e592bec14b124c3273312f93a63d2c29a2b989 kit/verify.sh -f3f84fbe780115608268a91a5203d2d3ada50b4317e7641d88430a692e61e1f4 kit/README.md -2411a16a68c98c8fdd402e19b9c29400b469c0054d0b6067541ee343988b85e0 schemas/examples/observer_event.example.json -4ab47977b0717c8bdb39c52f52880742785cbcf0b5ba73d9ecc835155d445dc1 schemas/examples/webhook_admission.example.json +4dc099a742429a7ec300ac4c9eefe2f6b80bc0c10d7a7a3bbaf7f0a0f0ad7f20 thresholds.yaml.dsse +f69f953c78134ef504b870cea47ba62d5e37a7a86ec0043d824dcb6073cd43fb kit/verify.sh +1cf8f0448881d067e5e001a1dfe9734b4cdfcaaf16c3e9a7321ceae56e4af8f2 kit/README.md +eaba054428fa72cd9476cffe7a94450e4345ffe2e294e9079eb7c3703bcf7df0 kit/ed25519.pub +40a40b31480d876cf4487d07ca8d8b5166c7df455bef234e2c1861b7b3dc7e3b evidence/README.md diff --git a/docs/modules/zastava/TASKS.md b/docs/modules/zastava/TASKS.md index 1a8fccb26..2ae4a1875 100644 --- a/docs/modules/zastava/TASKS.md +++ b/docs/modules/zastava/TASKS.md @@ -5,9 +5,9 @@ | ZASTAVA-DOCS-0001 | DONE (2025-11-30) | Docs Guild | README/architecture refreshed; Surface Env/Secrets and sprint links added. | | ZASTAVA-ENG-0001 | DONE (2025-11-30) | Module Team | TASKS board created; statuses mirrored with `docs/implplan/SPRINT_0335_0001_0001_docs_modules_zastava.md`. | | ZASTAVA-OPS-0001 | DONE (2025-11-30) | Ops Guild | Observability runbook stub + Grafana JSON placeholder added under `operations/`. | -| ZASTAVA-SCHEMAS-0001 | TODO | Zastava Guild | Publish signed observer/admission schemas + test vectors under `docs/modules/zastava/schemas/`; DSSE + SHA256 required. | -| ZASTAVA-KIT-0001 | TODO | Zastava Guild | Build signed `zastava-kit` bundle with thresholds.yaml, schemas, observations/admissions export, SHA256SUMS, and verify.sh; ensure offline parity. | -| ZASTAVA-THRESHOLDS-0001 | TODO | Zastava Guild | DSSE-sign `thresholds.yaml` and align with kit; publish Evidence Locker URI and update sprint 0144 checkpoints. | +| ZASTAVA-SCHEMAS-0001 | DONE (2025-12-02) | Zastava Guild | Signed observer/admission schemas + test vectors under `docs/modules/zastava/schemas/`; DSSE + SHA256 published. | +| ZASTAVA-KIT-0001 | DONE (2025-12-02) | Zastava Guild | Built signed `zastava-kit` bundle with thresholds, schemas, exports, SHA256SUMS, verify.sh; offline parity verified. | +| ZASTAVA-THRESHOLDS-0001 | DONE (2025-12-02) | Zastava Guild | DSSE-signed `thresholds.yaml`, recorded Evidence Locker targets, and aligned with kit packaging. | | ZASTAVA-GAPS-144-007 | DONE (2025-12-02) | Zastava Guild | Remediation plan for ZR1–ZR10 published at `docs/modules/zastava/gaps/2025-12-02-zr-gaps.md`; follow-on schemas/kit/thresholds to be produced and signed. | > Keep this table in lockstep with the sprint Delivery Tracker (TODO/DOING/DONE/BLOCKED updates go to both places). diff --git a/docs/modules/zastava/evidence/README.md b/docs/modules/zastava/evidence/README.md index bb49fd74c..6b3c15469 100644 --- a/docs/modules/zastava/evidence/README.md +++ b/docs/modules/zastava/evidence/README.md @@ -1,29 +1,53 @@ -# Zastava Evidence Locker Plan (schemas/kit) +# Zastava Evidence Locker (schemas/kit) -Artifacts to sign (target 2025-12-06): -- `schemas/observer_event.schema.json` — predicate `stella.ops/zastavaSchema@v1` -- `schemas/webhook_admission.schema.json` — predicate `stella.ops/zastavaSchema@v1` -- `thresholds.yaml` — predicate `stella.ops/zastavaThresholds@v1` -- `zastava-kit.tzst` + `SHA256SUMS` — predicate `stella.ops/zastavaKit@v1` +Signed 2025-12-02 with Ed25519 key (pub base64url: `mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc`). Private key stored offline; all signatures use DSSEv1 pre-auth encoding. +Public key copy: `docs/modules/zastava/kit/ed25519.pub`. -Planned Evidence Locker paths (fill after signing): -- `evidence-locker/zastava/2025-12-06/observer_event.schema.dsse` -- `evidence-locker/zastava/2025-12-06/webhook_admission.schema.dsse` -- `evidence-locker/zastava/2025-12-06/thresholds.dsse` -- `evidence-locker/zastava/2025-12-06/zastava-kit.tzst` -- `evidence-locker/zastava/2025-12-06/SHA256SUMS` +## Artefacts +- `schemas/observer_event.schema.json.dsse` (payloadType `application/vnd.stellaops.zastava.schema+json;name=observer_event;version=1`) +- `schemas/webhook_admission.schema.json.dsse` (payloadType `application/vnd.stellaops.zastava.schema+json;name=webhook_admission;version=1`) +- `thresholds.yaml.dsse` (payloadType `application/vnd.stellaops.zastava.thresholds+yaml;version=1`) +- `exports/observer_events.ndjson.dsse` (payloadType `application/vnd.stellaops.zastava.observer-events+ndjson;version=1`) +- `exports/webhook_admissions.ndjson.dsse` (payloadType `application/vnd.stellaops.zastava.webhook-admissions+ndjson;version=1`) +- `kit/zastava-kit.tzst.dsse` (payloadType `application/vnd.stellaops.zastava.kit+tzst;version=1`) -Signing template (replace KEY and file): +## Evidence Locker targets +- `evidence-locker/zastava/2025-12-02/observer_event.schema.json.dsse` +- `evidence-locker/zastava/2025-12-02/webhook_admission.schema.json.dsse` +- `evidence-locker/zastava/2025-12-02/thresholds.yaml.dsse` +- `evidence-locker/zastava/2025-12-02/observer_events.ndjson.dsse` +- `evidence-locker/zastava/2025-12-02/webhook_admissions.ndjson.dsse` +- `evidence-locker/zastava/2025-12-02/zastava-kit.tzst` +- `evidence-locker/zastava/2025-12-02/zastava-kit.tzst.dsse` +- `evidence-locker/zastava/2025-12-02/SHA256SUMS` + +## Signing template (Python, ed25519) ```bash -cosign sign-blob \ - --key cosign.key \ - --predicate-type stella.ops/zastavaSchema@v1 \ - --output-signature schemas/observer_event.schema.dsse \ - schemas/observer_event.schema.json +python - <<'PY' +from pathlib import Path +from base64 import urlsafe_b64encode +import json +from cryptography.hazmat.primitives.asymmetric import ed25519 +from cryptography.hazmat.primitives import serialization + +priv = serialization.load_pem_private_key(Path('/tmp/zastava-ed25519.key').read_bytes(), password=None) +pub = priv.public_key().public_bytes(encoding=serialization.Encoding.Raw, format=serialization.PublicFormat.Raw) +keyid = urlsafe_b64encode(pub).decode().rstrip('=') +pt = '' +payload = Path('').read_bytes() +pae = b' '.join([b'DSSEv1', str(len(pt)).encode(), pt.encode(), str(len(payload)).encode(), payload]) +sig = priv.sign(pae) +env = { + 'payloadType': pt, + 'payload': urlsafe_b64encode(payload).decode().rstrip('='), + 'signatures': [{'keyid': keyid, 'sig': urlsafe_b64encode(sig).decode().rstrip('=')}], +} +Path('.dsse').write_text(json.dumps(env, indent=2, sort_keys=True) + '\n') +print('signed', '', 'with keyid', keyid) +PY ``` -Post-sign steps: -1) Verify DSSEs with `cosign verify-blob` using `cosign.pub`. -2) Upload DSSEs + SHA256SUMS to Evidence Locker paths above. -3) Update `docs/implplan/SPRINT_0144_0001_0001_zastava_runtime_signals.md` Decisions & Risks and Next Checkpoints with final URIs. -4) Mark tasks ZASTAVA-SCHEMAS-0001 / ZASTAVA-THRESHOLDS-0001 / ZASTAVA-KIT-0001 to DONE in both sprint and TASKS tables. +## Post-sign checklist +1) Run `kit/verify.sh` to validate hashes + DSSE. +2) Upload artefacts + DSSEs + SHA256SUMS to the Evidence Locker paths above. +3) Record URIs in sprint 0144 Decisions & Risks and mark ZASTAVA-SCHEMAS-0001 / ZASTAVA-THRESHOLDS-0001 / ZASTAVA-KIT-0001 as DONE. diff --git a/docs/modules/zastava/exports/observer_events.ndjson b/docs/modules/zastava/exports/observer_events.ndjson new file mode 100644 index 000000000..00f525ab5 --- /dev/null +++ b/docs/modules/zastava/exports/observer_events.ndjson @@ -0,0 +1 @@ +{"event_type":"runtime_fact","firmware_version":"1.2.3","graph_revision_id":"graph-r1","ledger_id":"ledger-789","monotonic_nanos":123456789,"observed_at":"2025-12-02T12:00:00Z","payload":{"pid":4242,"process":"nginx"},"payload_hash":"sha256:7476a5068a3f0780c552f81c90d061d9e39c37f425a243ecff961b08676546fd","policy_hash":"sha256:deadbeef","project_id":"proj-123","replay_manifest":"manifest-r1","sensor_id":"observer-01","signature":"dsse://observer-events/2025-12-02/observer_events.ndjson.dsse#line1","tenant_id":"tenant-a"} diff --git a/docs/modules/zastava/exports/observer_events.ndjson.dsse b/docs/modules/zastava/exports/observer_events.ndjson.dsse new file mode 100644 index 000000000..b60bfa8be --- /dev/null +++ b/docs/modules/zastava/exports/observer_events.ndjson.dsse @@ -0,0 +1,10 @@ +{ + "payload": "eyJldmVudF90eXBlIjoicnVudGltZV9mYWN0IiwiZmlybXdhcmVfdmVyc2lvbiI6IjEuMi4zIiwiZ3JhcGhfcmV2aXNpb25faWQiOiJncmFwaC1yMSIsImxlZGdlcl9pZCI6ImxlZGdlci03ODkiLCJtb25vdG9uaWNfbmFub3MiOjEyMzQ1Njc4OSwib2JzZXJ2ZWRfYXQiOiIyMDI1LTEyLTAyVDEyOjAwOjAwWiIsInBheWxvYWQiOnsicGlkIjo0MjQyLCJwcm9jZXNzIjoibmdpbngifSwicGF5bG9hZF9oYXNoIjoic2hhMjU2Ojc0NzZhNTA2OGEzZjA3ODBjNTUyZjgxYzkwZDA2MWQ5ZTM5YzM3ZjQyNWEyNDNlY2ZmOTYxYjA4Njc2NTQ2ZmQiLCJwb2xpY3lfaGFzaCI6InNoYTI1NjpkZWFkYmVlZiIsInByb2plY3RfaWQiOiJwcm9qLTEyMyIsInJlcGxheV9tYW5pZmVzdCI6Im1hbmlmZXN0LXIxIiwic2Vuc29yX2lkIjoib2JzZXJ2ZXItMDEiLCJzaWduYXR1cmUiOiJkc3NlOi8vb2JzZXJ2ZXItZXZlbnRzLzIwMjUtMTItMDIvb2JzZXJ2ZXJfZXZlbnRzLm5kanNvbi5kc3NlI2xpbmUxIiwidGVuYW50X2lkIjoidGVuYW50LWEifQo", + "payloadType": "application/vnd.stellaops.zastava.observer-events+ndjson;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "5DPpjAcyWSeCM_yPCiIsQl92FtUwnccN8J5lY5AxKBE1qfYbU6dEgGQudDWlY2_-FUak6fupQ79vrgGbGiDDDQ" + } + ] +} diff --git a/docs/modules/zastava/exports/webhook_admissions.ndjson b/docs/modules/zastava/exports/webhook_admissions.ndjson new file mode 100644 index 000000000..1eb14bb87 --- /dev/null +++ b/docs/modules/zastava/exports/webhook_admissions.ndjson @@ -0,0 +1 @@ +{"bypass_waiver_id":null,"decision":"allow","decision_at":"2025-12-02T12:00:10Z","decision_reason":"surface cache fresh","graph_revision_id":"graph-r1","ledger_id":"ledger-789","manifest_pointer":"surfacefs://cache/sha256:abc","monotonic_nanos":2233445566,"namespace":"prod","payload":{"images":[{"digest":"sha256:abcd","name":"ghcr.io/acme/api:1.2.3","sbom_referrer":true,"signed":true}],"manifest_pointer":"surfacefs://cache/sha256:abc","policy_hash":"sha256:deadbeef","verdict":"allow"},"payload_hash":"sha256:36bfb2bc81b7050bbb508e12cafe7ad5a51336aad397ef3a23b0e258aed73dc6","policy_hash":"sha256:deadbeef","project_id":"proj-123","replay_manifest":"manifest-r1","request_uid":"abcd-1234","resource_kind":"Deployment","side_effect":"none","signature":"dsse://webhook-admissions/2025-12-02/webhook_admissions.ndjson.dsse#line1","tenant_id":"tenant-a","workload_name":"api"} diff --git a/docs/modules/zastava/exports/webhook_admissions.ndjson.dsse b/docs/modules/zastava/exports/webhook_admissions.ndjson.dsse new file mode 100644 index 000000000..67a37c164 --- /dev/null +++ b/docs/modules/zastava/exports/webhook_admissions.ndjson.dsse @@ -0,0 +1,10 @@ +{ + "payload": "eyJieXBhc3Nfd2FpdmVyX2lkIjpudWxsLCJkZWNpc2lvbiI6ImFsbG93IiwiZGVjaXNpb25fYXQiOiIyMDI1LTEyLTAyVDEyOjAwOjEwWiIsImRlY2lzaW9uX3JlYXNvbiI6InN1cmZhY2UgY2FjaGUgZnJlc2giLCJncmFwaF9yZXZpc2lvbl9pZCI6ImdyYXBoLXIxIiwibGVkZ2VyX2lkIjoibGVkZ2VyLTc4OSIsIm1hbmlmZXN0X3BvaW50ZXIiOiJzdXJmYWNlZnM6Ly9jYWNoZS9zaGEyNTY6YWJjIiwibW9ub3RvbmljX25hbm9zIjoyMjMzNDQ1NTY2LCJuYW1lc3BhY2UiOiJwcm9kIiwicGF5bG9hZCI6eyJpbWFnZXMiOlt7ImRpZ2VzdCI6InNoYTI1NjphYmNkIiwibmFtZSI6ImdoY3IuaW8vYWNtZS9hcGk6MS4yLjMiLCJzYm9tX3JlZmVycmVyIjp0cnVlLCJzaWduZWQiOnRydWV9XSwibWFuaWZlc3RfcG9pbnRlciI6InN1cmZhY2VmczovL2NhY2hlL3NoYTI1NjphYmMiLCJwb2xpY3lfaGFzaCI6InNoYTI1NjpkZWFkYmVlZiIsInZlcmRpY3QiOiJhbGxvdyJ9LCJwYXlsb2FkX2hhc2giOiJzaGEyNTY6MzZiZmIyYmM4MWI3MDUwYmJiNTA4ZTEyY2FmZTdhZDVhNTEzMzZhYWQzOTdlZjNhMjNiMGUyNThhZWQ3M2RjNiIsInBvbGljeV9oYXNoIjoic2hhMjU2OmRlYWRiZWVmIiwicHJvamVjdF9pZCI6InByb2otMTIzIiwicmVwbGF5X21hbmlmZXN0IjoibWFuaWZlc3QtcjEiLCJyZXF1ZXN0X3VpZCI6ImFiY2QtMTIzNCIsInJlc291cmNlX2tpbmQiOiJEZXBsb3ltZW50Iiwic2lkZV9lZmZlY3QiOiJub25lIiwic2lnbmF0dXJlIjoiZHNzZTovL3dlYmhvb2stYWRtaXNzaW9ucy8yMDI1LTEyLTAyL3dlYmhvb2tfYWRtaXNzaW9ucy5uZGpzb24uZHNzZSNsaW5lMSIsInRlbmFudF9pZCI6InRlbmFudC1hIiwid29ya2xvYWRfbmFtZSI6ImFwaSJ9Cg", + "payloadType": "application/vnd.stellaops.zastava.webhook-admissions+ndjson;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "UwXQm2oZPVIISQecILLkvxvSXZiXeZdPVe5RNqFxZ8Dv5xDT1nEcTq0pn2Tl3unk0sY44Lh-dU_599nxaHD9Aw" + } + ] +} diff --git a/docs/modules/zastava/gaps/2025-12-02-zr-gaps.md b/docs/modules/zastava/gaps/2025-12-02-zr-gaps.md index 2cc637bdb..f841f7d44 100644 --- a/docs/modules/zastava/gaps/2025-12-02-zr-gaps.md +++ b/docs/modules/zastava/gaps/2025-12-02-zr-gaps.md @@ -43,7 +43,8 @@ - Delivery paths for schemas/thresholds/kit will be added when produced; DSSE signatures required for all artefacts. ## Next steps -1) Generate schemas + test vectors and place under `docs/modules/zastava/schemas/`; sign DSSE. -2) Draft `thresholds.yaml` with budgets and sign DSSE. -3) Build `zastava-kit` bundle + `verify.sh`; include Evidence Locker path and SHA256. +1) ✅ Schemas + test vectors generated and DSSE-signed under `docs/modules/zastava/schemas/` (2025-12-02). +2) ✅ `thresholds.yaml` DSSE-signed and included in kit (2025-12-02). +3) ✅ Deterministic `zastava-kit` bundle + `verify.sh` built; kit DSSE stored at `docs/modules/zastava/kit/zastava-kit.tzst.dsse` with hashes in `SHA256SUMS` (2025-12-02). 4) Add tenancy/ordering/provenance enforcement to Observer/Webhook validators and tests; mirror changes in sprint and TASKS boards. +5) Upload DSSE artefacts + kit to Evidence Locker paths in `docs/modules/zastava/evidence/README.md` and backfill operations docs with verifier usage. diff --git a/docs/modules/zastava/kit/README.md b/docs/modules/zastava/kit/README.md index 339ae5d4f..3240bcda8 100644 --- a/docs/modules/zastava/kit/README.md +++ b/docs/modules/zastava/kit/README.md @@ -1,17 +1,83 @@ -# Zastava Kit (offline bundle) – Draft +# Zastava Kit (offline bundle) -Contents to include when built: -- Observations and admissions exports (NDJSON) signed via DSSE. -- Schemas: `schemas/observer_event.schema.json`, `schemas/webhook_admission.schema.json`. -- Thresholds: `thresholds.yaml` (DSSE-signed). -- Hash manifest: `SHA256SUMS` (covering all kit files). -- Verify script: `verify.sh` (hash + DSSE verification; fail closed on mismatch). +## Contents +- Schemas + DSSE: `schemas/observer_event.schema.json(.dsse)`, `schemas/webhook_admission.schema.json(.dsse)`. +- Examples: `schemas/examples/*.json` (canonicalised, hashed). +- Thresholds + DSSE: `thresholds.yaml(.dsse)`. +- Exports + DSSE: `exports/observer_events.ndjson(.dsse)`, `exports/webhook_admissions.ndjson(.dsse)`. +- Verification assets: `SHA256SUMS`, `kit/verify.sh`, `kit/ed25519.pub`, `schemas/README.md`, `evidence/README.md`. -Deterministic packaging: `tar --mtime @0 --owner 0 --group 0 --numeric-owner -cf - kit | zstd -19 --long=27 --no-progress > zastava-kit.tzst`. +## Build (deterministic) +From `docs/modules/zastava`: -Pending: fill with signed artefacts and Evidence Locker URIs after DSSE signing. -Planned Evidence Locker paths (post-signing): -- `evidence-locker/zastava/2025-12-06/observer_event.schema.dsse` -- `evidence-locker/zastava/2025-12-06/webhook_admission.schema.dsse` -- `evidence-locker/zastava/2025-12-06/thresholds.dsse` -- `evidence-locker/zastava/2025-12-06/zastava-kit.tzst` + `SHA256SUMS` +```bash +tar --mtime @0 --owner 0 --group 0 --numeric-owner --sort=name \ + -cf - \ + SHA256SUMS schemas exports thresholds.yaml thresholds.yaml.dsse \ + schemas/examples schemas/README.md \ + schemas/observer_event.schema.json schemas/observer_event.schema.json.dsse \ + schemas/webhook_admission.schema.json schemas/webhook_admission.schema.json.dsse \ + exports/observer_events.ndjson exports/observer_events.ndjson.dsse \ + exports/webhook_admissions.ndjson exports/webhook_admissions.ndjson.dsse \ + evidence/README.md kit/README.md kit/verify.sh kit/ed25519.pub \ +| zstd -19 --long=27 --no-progress > kit/zastava-kit.tzst +``` + +Sign the kit itself with the same Ed25519 key (base64url pub: `mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc`): + +```bash +python - <<'PY' +from pathlib import Path +from base64 import urlsafe_b64encode +import json +from cryptography.hazmat.primitives.asymmetric import ed25519 +from cryptography.hazmat.primitives import serialization + +priv = serialization.load_pem_private_key(Path('/tmp/zastava-ed25519.key').read_bytes(), password=None) +pub = priv.public_key().public_bytes(encoding=serialization.Encoding.Raw, format=serialization.PublicFormat.Raw) +keyid = urlsafe_b64encode(pub).decode().rstrip('=') +pt = 'application/vnd.stellaops.zastava.kit+tzst;version=1' +payload = Path('kit/zastava-kit.tzst').read_bytes() +pae = b' '.join([b'DSSEv1', str(len(pt)).encode(), pt.encode(), str(len(payload)).encode(), payload]) +sig = priv.sign(pae) +env = { + 'payloadType': pt, + 'payload': urlsafe_b64encode(payload).decode().rstrip('='), + 'signatures': [{'keyid': keyid, 'sig': urlsafe_b64encode(sig).decode().rstrip('=')}], +} +Path('kit/zastava-kit.tzst.dsse').write_text(json.dumps(env, indent=2, sort_keys=True) + '\n') +print('wrote kit/zastava-kit.tzst.dsse with keyid', keyid) +PY +``` + +## Verify +1) Verify the kit DSSE before unpacking (optional but recommended) using the public key shipped alongside the kit (run from `docs/modules/zastava`): +```bash +cd docs/modules/zastava +python - <<'PY' +import base64, json, sys +from pathlib import Path +from cryptography.hazmat.primitives.asymmetric import ed25519 + +root = Path('.') +pub = base64.urlsafe_b64decode((root / 'kit' / 'ed25519.pub').read_text().strip() + '==') +env = json.loads((root / 'kit' / 'zastava-kit.tzst.dsse').read_text()) +payload = (root / 'kit' / 'zastava-kit.tzst').read_bytes() +pt = env['payloadType'].encode() +pae = b' '.join([b'DSSEv1', str(len(pt)).encode(), pt, str(len(payload)).encode(), payload]) +sig = base64.urlsafe_b64decode(env['signatures'][0]['sig'] + '==') +ed25519.Ed25519PublicKey.from_public_bytes(pub).verify(sig, pae) +decoded_payload = base64.urlsafe_b64decode(env['payload'] + '==') +assert decoded_payload == payload +print('OK: kit DSSE verified') +PY +``` +2) Extract and run offline validation of the inner artefacts: +```bash +zstd -d kit/zastava-kit.tzst -c | tar -xf - +./kit/verify.sh +``` + +## Notes +- Private signing key is held offline; only the public key is shipped. +- All files are deterministic (mtime=0, numeric owners) to keep hashes stable for Evidence Locker ingestion. diff --git a/docs/modules/zastava/kit/ed25519.pub b/docs/modules/zastava/kit/ed25519.pub new file mode 100644 index 000000000..27b2b6343 --- /dev/null +++ b/docs/modules/zastava/kit/ed25519.pub @@ -0,0 +1 @@ +mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc diff --git a/docs/modules/zastava/kit/verify.sh b/docs/modules/zastava/kit/verify.sh index 918262786..a1fc753e9 100644 --- a/docs/modules/zastava/kit/verify.sh +++ b/docs/modules/zastava/kit/verify.sh @@ -1,24 +1,59 @@ #!/usr/bin/env bash set -euo pipefail -ROOT="$(cd "$(dirname "$0")" && pwd)" -cd "$ROOT" -if ! command -v sha256sum >/dev/null; then - echo "sha256sum required" >&2; exit 1 -fi +ROOT="$(cd "$(dirname "$0")" && pwd)" +MODULE_ROOT="${ROOT}/.." +cd "$MODULE_ROOT" +export MODULE_ROOT + +command -v sha256sum >/dev/null || { echo "sha256sum required" >&2; exit 1; } +command -v python >/dev/null || { echo "python required" >&2; exit 1; } sha256sum --check SHA256SUMS -if command -v cosign >/dev/null && [ -f cosign.pub ]; then - echo "cosign present; DSSE verification placeholders (update paths when signed):" - echo "- observer_event.schema.dsse" - echo "- webhook_admission.schema.dsse" - echo "- thresholds.dsse" - # Example commands (uncomment once DSSE files exist): - # cosign verify-blob --key cosign.pub --signature observer_event.schema.dsse schemas/observer_event.schema.json - # cosign verify-blob --key cosign.pub --signature webhook_admission.schema.dsse schemas/webhook_admission.schema.json - # cosign verify-blob --key cosign.pub --signature thresholds.dsse thresholds.yaml -else - echo "cosign not found or cosign.pub missing; skipped DSSE verification" -fi -echo "OK: hashes verified (DSSE verification pending)" +python - <<'PY' +import base64, json, os, sys +from pathlib import Path + +try: + from cryptography.hazmat.primitives.asymmetric import ed25519 +except Exception as exc: + raise SystemExit(f"cryptography package required for DSSE verification: {exc}") + +root = Path(os.environ['MODULE_ROOT']).resolve() +pub_b64 = (root / "kit" / "ed25519.pub").read_text().strip() +pub = base64.urlsafe_b64decode(pub_b64 + "==") +verifier = ed25519.Ed25519PublicKey.from_public_bytes(pub) + +def pae(payload_type: bytes, payload: bytes) -> bytes: + parts = [b"DSSEv1", str(len(payload_type)).encode(), payload_type, str(len(payload)).encode(), payload] + return b" ".join(parts) + +def verify(name: str, payload_path: Path, envelope_path: Path, payload_type: str): + payload = payload_path.read_bytes() + envelope = json.loads(envelope_path.read_text()) + if envelope.get("payloadType") != payload_type: + raise SystemExit(f"{name}: payloadType mismatch ({envelope.get('payloadType')} != {payload_type})") + if not envelope.get("signatures"): + raise SystemExit(f"{name}: missing signatures") + sig_entry = envelope["signatures"][0] + sig = base64.urlsafe_b64decode(sig_entry["sig"] + "==") + decoded_payload = base64.urlsafe_b64decode(envelope["payload"] + "==") + if decoded_payload != payload: + raise SystemExit(f"{name}: payload body mismatch vs envelope") + verifier.verify(sig, pae(payload_type.encode(), payload)) + print(f"OK: {name}") + +targets = [ + ("observer schema", root / "schemas" / "observer_event.schema.json", root / "schemas" / "observer_event.schema.json.dsse", "application/vnd.stellaops.zastava.schema+json;name=observer_event;version=1"), + ("webhook schema", root / "schemas" / "webhook_admission.schema.json", root / "schemas" / "webhook_admission.schema.json.dsse", "application/vnd.stellaops.zastava.schema+json;name=webhook_admission;version=1"), + ("thresholds", root / "thresholds.yaml", root / "thresholds.yaml.dsse", "application/vnd.stellaops.zastava.thresholds+yaml;version=1"), + ("observer exports", root / "exports" / "observer_events.ndjson", root / "exports" / "observer_events.ndjson.dsse", "application/vnd.stellaops.zastava.observer-events+ndjson;version=1"), + ("webhook exports", root / "exports" / "webhook_admissions.ndjson", root / "exports" / "webhook_admissions.ndjson.dsse", "application/vnd.stellaops.zastava.webhook-admissions+ndjson;version=1"), +] + +for name, payload_path, envelope_path, ptype in targets: + verify(name, payload_path, envelope_path, ptype) +PY + +echo "OK: SHA256 + DSSE signatures verified" diff --git a/docs/modules/zastava/kit/zastava-kit.tzst b/docs/modules/zastava/kit/zastava-kit.tzst new file mode 100644 index 000000000..4e488d938 Binary files /dev/null and b/docs/modules/zastava/kit/zastava-kit.tzst differ diff --git a/docs/modules/zastava/kit/zastava-kit.tzst.dsse b/docs/modules/zastava/kit/zastava-kit.tzst.dsse new file mode 100644 index 000000000..c44bf9e5f --- /dev/null +++ b/docs/modules/zastava/kit/zastava-kit.tzst.dsse @@ -0,0 +1,10 @@ +{ + "payload": "KLUv_WAAx8UaAUo3ATwtoEooe-7my7E8x58uyW6Cs4eZOSjdgYwF6Ux3dN8-2dVJjwlcJrwOr8N6OVQPWwOxA90Dc3MA1TcEEi7L5u1HjNkTQT-Wek4PFD0ZeuCeXZyd8zNCFOGIYHRS5CiIDhiLhMU8WBwsDBR-FnPY_zzgGezKHBr80CMjZgIH99dnAQX0rhx_Y8NQKaCA2T85fz9g_Dr_uNl3Dn8DPI2Tf3n-spf-3EbuNHkXwfHPz02Mw6YyG4nRx20_b2P-XJc394FfZpgCCJBIMCYBs3vc-JGiwcymrE0BqnrReum1APT9dc6Z-u3Y5ldjul7W-q4tq9qWzniWN0Y5llX1alplTYYOXC3L8k49x_zWl2J3Y7pjjOJeYdutVd2qU6uqcMYqLPvMVpVFs81Vv6xe7e13X9XnvD3fla4uhrvn3erYRXnNMr9rYvuO0nMII1eaU_tb8X0xeunNb0Xp7ld2u8XtVeW-dUUtXzPs8tR6jNNMPeaz1jdjGeN0a5rpy9u-M-0VljGmm1eJzsul4_MO3H4XtxFb3lcLX1VWectfVkWthVWVxtjOboaxXi_ulvZs3W1hW_MKqzqrl8W2uqqrVrv37fxmPu-erc-0Wl8113lfnndqrdW-VhmnvnfMqqzNts700lhnT6nPlZU1tnfNl9-YZvZiTrm_1qUVtrqnFoZdV743V1WGVRetemZhmWYLuyqrcef8c_xRBENUn5NfXGGYXrdeGPbY2r5iimVVxf2qtPY-wznrznWW-QT3HJEL5BHR3Mwr3RVjFmZpn3PuVt8X66p5lv31Vb-9wvjCNL7Z5utyrPYK72vtbKtr790Y1h7n3auqq-W2V-o99hjOV82VUuyz53hndVO-b8d6F3oZ1a_erbcs5v7yld5dXZx2XNVd-83eV9RXVs6w1blr66KZYzjLuGuc7Xo75fbma_uMZxXGK5tVvTHHfF_XyrIq22rlmnlGq8eV-5_zeBaVQHDT_bSvpr7zrFN8ebY8xS7ePfP-ulem73XvZTMru9TzzFOL9V6ztnxmWUtza22i_88dGGcrcwyjnvqrua-161p1q1Od0W43ra5L91XVajfWNl-ddm6zi3HiAz_J4Ri_buoisZvE_jWI8LPXt_3nzLLNG8PYb80rZveFYblimmfLs94Z95pd28reYpnf_dKM_ZVVWOd6vUV1BWBOMY-fBIBOAKq1mHXrxTCMArC5dDaXZR8Nmw8GbcMTbUkme9RpoHyJJSU2U9mGHVZuKOWRWP07UOjEUbr6ybRuNCqZjJWrM6CQ13y43fGPRstPiMQ1FFpWU-xpE17gtqfDSgNPlToLKj1WK3kmtKHUyM7bFPN1C6BOKxFIptzTHaqkw8qghbizOyyxJJ8k3v5GlM-6uzPZluYCaq5GeiYcsvtoEn9yCnu0UmL5WAFlr7LxYAa0ebwQrYBC3cZpUzfhG8rqULsgC_PJH0_2iaNQ7ZNLDQMnSTzNez5a18qn6YeTHgS_gXJRNtqpa-Bewz0YsIULSAs13-7KhY8yenAMn1KaNuQyZj5tqzRopu1tk7UJjZ5JJmFAFhLbzcCjgdPNI1EeFq6yOhV6lq1-NBBjN5n5dKeBNG8rIC1VsdoU8fimuXsLryUl-ehusyGtxJ4ltXD1GJno75nwFC4t16VhU01D5h6cA5f55BKFgTICT11hVVa11IpMh27kIA54Fw2AaCAkiBypRtYajwSz66avD7CguhRA-EbGDZpgATH0oINhMMeQE_dXNPqYFSjE8R_UcfR1y8d5EF2PYu5bB3Km6ZHf79wo9lAAjDZGn0XeQBiJYCMcx20Aaq2q0sqWRPDU6N37LurcK0GIqG-RBH9FXptfjPecWbZTfW2-vmP48pyve6uVLZ6x7vO2tWfU0lpTi2K6Zltb7cfNcYgFoyh274H6J3BhRujezggg5xD3NuL4TNBH2eP4dVlWVd1r7dYZ4Xuxjx8HhiI_d_Yi-b_iBHK7864jPxOXT6L_c1jqytIFCUVu9FEABASEBNoA_9f_7577sicqcxd2cdyd5HvXPHdegtxflXVhV0Z1vbD3u9_KZstWHrsb3455vSineNfMW8vaNMOovzm7qJWxBQrUx42-__sGxQMtgVKOpQ1bG5aldswiflnVtfV_ohH8QA9nWEPsI967KjY4JxJF12EdVdTFfPJEzh8oxp29SDETFCKFgnutC7O1OiJm5kxdl0ddHI3ynCOHIMKgF_D3-DvHe3BuJo5RbmEWR-AYluKeBmxj-WDAfY5koIsRCCQHPYS_QR7xZ-9fH3XNPGURnGRudDGc1RRxoHecR4DzehfJMe_JRCT6flapdPKh0sncFAhFRVrJpFk4XZittBioFk1IRyihx-DgNw86kayu_rM2mDMSURUOUOhe_g4wwPFDMHddH7-DfyaEzXVhmOdMJPweluPDN-L-jGw2kCmCpOzp0Ze3PUu5i3oe0D-umxEoRzUlUIAjFcGN4Kh_AZzm4-mkbqJNGnu529aZXDgMhqR1nOOQ2lIRD0TbOEfztIqreKFTB3K5lM-lKila6lKx4ErJTaiJ3iqXa-k2jN2eB5caraPaiL3QhWQpq83lKo2mjHJDIMYLbTwVYmP3hZfF1gwEk7qQLiWp8gnhjQaSPq1LUUCmhQv6bChi-agPJ0OPhxLLPQaj2AHEAgUJ6rUsy8pAIBBuoicj0YSxTEj6q5Q7koboA-FkZjKK8KujQ56BGllAgHpYoc6qZJi9v1NnlFjcZRgeStGFoW8-5X6QwgeIZQAX-Vg6nqZYmC-XLI_CuYwnSsqSMmWcQ1eKUEAYRQFZvK16Fwa0riQJKZwQiPlSkkn23Q2FXCbDwdjqQQqhQhwHT7OhDgP5YLR7SYWsC0ezMp-LfUC0UykVQo1IWtnhYEgLD5KWikwc7IOD2egmOehQI2Fvn8oM6UL_9zk9SA5lBOIpHxxLGeFYKgVyKSmnjUNVM6lLGSGReMJNFq4EcbiU4xONcikrg_XbZG-Vm6q4CxkhUa4TF_LGbvJtDiSVttILO1igwNaKVlil9VFheEf6KkPHL7w7kD9xCEEwDlexKJwOdVP4QdmIEF2hHcrb2Oi_p8EhlGGAnLIGfUnVrNNgXexG9HnIzrhAejA8PFrypEqSCEXEVoOVjwbqttFrbC6xJenN8kYL2hXbmvJ8OMpUaSiFHhCYES8VqScFIkscywkEWbbQY1kYq8UEcrQUpkHrJdXKhU5b2Gg4GImG0-_CUQ9Id5_Ro3laZ7P32FDpuXp7LoiOIp9sF7YJJdwba-EU6ttCTWFPb5YFkTrN1aPM6NJpyI4NbZq3K6YLfUeCtO93FiiTqISrPrysm77p2sUSjyvljV9Q6tR9IczdYZ1oG67ZhCeZ81h-61Vdez3mZ8wjyRmnrnkczLmRODaNsvwwrFoXtRXEaeGSKenHKHNday-b-5iMKiDDwrhZ-MBSJuoIqk6rJI8_eh0CDajMQDZfz1C7ET0Yl8nQ-6Qs5eNSadhk5dxDcUYMOBxRKOajMSB5WNiErETifTTlQB1vV5LzrdO-HTJ5Oo93lQsCF0ChkttdejzuNw7SZedp24T59CbJVVJBIOGkRMNlSgS8Zp3FogyXq25CuW4yumN3SuOBvAWM5_Em24QadWiHoylR6AIol52m9DTwzbJaiEmleMrjaeGIPX1DTXZX8Zob6kL7sKAeE65dxpHdFawhYbDaaWPk4zA-Up8rxMGUkUlOWR1JNSGrR6WyAwkjCEYV5XDCSGhEmVBsh3TZNruxxONlDnsWEAMKgQvcNmkTSsljd13YIOBgShswEYwDhENBALE4NGwimgYOwqyU0sbmcYk0MGwaD8-Ds3losyRm75c-p5kEzA-Z83-KpjN3vwlHN4E6z9-3Mew4-kdqxjl-8OFAAxwbB8YGAoyDcNM1GBGNxCOhDMH9kHD0Tzp4ZxhYWK3VVhZVJGYiAXHAMPimkoN6eg5CkeDrIoaCsUDBQoHB3PacCQsUKAQwGQZjHzk8xzhISFBMhZ5fVyh0ggONyZgJPiEqKCgUCoVBN3Xgh1gsEnygyDk-yFAs6KM8Fd4g4vXxl5vZP5fFGUkoxTHxwURpNxqJBfOAeoAl4MBUTHDbfSqYDTQwEUydIxyYHvmnqZFIzIQioXA4EBYoEEUsHYjDJaFg8FM4CSBgKhgKKnJ88m0wgEKB4-CgSSQ6LvsecuEEyd4_wsdpBPGTCAEIcK9rWb-XNmiZkd7psIPjZPw5D6iCwey6R9l94DzgcyDGkUDNBib_vn8cN407iyPDLnjnEhA4KIi4yFF2BBALCJoAAiIBQ2ExEQwcC8cIgmLilweTbz_z5D0Mue1zcY5dBg12Dj9y9q4ZNVGGHhKMg9lHUDwtDMRPgYVVODAYUX3kxNPBQsHiIGGx-DoDYnQwGGNo8PXOYKETGJAci7nQPbEwYDEuFBYGYecOhkGCIXk4OGWNI6LAcHwgRoEbdNHBWITeiLAwMBB5PXQoTFMikfpj97zyZ8fvJm7kPE4MHWP2UnlzJHp1aVH3-T9Mj5ua67Yxhz6xCD0sEEAgZMzvu_5Hbo64TSIxOhoH5h81QMwqjD6ww7Ux-fhTs4HZG9NvBfKn8HdIguJoiv07gTQBr9FI6IiIiEQiIcIDRUmCsbnxc2IfO-84cmH0HDAOeP-c48CA-ccP7B3_DLAHJHVLDibsbizdeA_ayTydUBQtnwGToPpE8xk9lKuhIfXwUC2oLl6SKMFUTI0L6Hp2g0Fpz4VhyJAZyVcSE4byFiQZ0oAIiT883XRp0CgGcwJTH0kDyhdzQR_1yWKGtHWujtVI0gc1DahPCMGFej4sOeG6S5zBhHWilXBXAWUwJ0mDhnqQLKdOU6klyPYYkHZIaUanB5VYEN80k13DDFariIPg2Fw44sRRtO2RO31R13El3nnvG9VFIrF3OXEnKdoszuvc17noEy0agiKv1EFngkgtC4eiv79uJHViJoyZh2TiwjCLYkhDRn0YW6IOb2yWqgp9WvrI23HsF1IGODvyVQ4EZZkgmYaSMIDRWN3zwLUO4jTQxoQylSwWyu6ZDYXiedBA62VAp05ls1WG7mM25EQjLT9m5R7Hv3mhjl_u6r8A0iKYjA1xpD1akD6Bl-f5aAGIU6jUEaqEsZwFIVNIRmZkQAEDk2AgIDQinMsFcyF98AHUKCcmg-Qh-TAQCASCXMNQDAJACAcAYAwABhkiMM4BYzaeAOMscIDuCMTYjhPjS-38MqvNLSAiSGmE88v6ua7detwKlqEFDGh4VayaA44lk03xIzsLYgpTFjJwxMRkakBGV4g1A8CKXeZ9QynptQYgdlRQDgCUxRqwkxvaTUiongTFcLtMfZJC0e_RSdpz-VaHKDVK-zo4bd2mbG2p4oCKlG_WZ00by0TyeigrRVvqi6fT_eM9uEk4PjPfzARZfvwSZ9zwLXOwlEOcYnQADNNEyMK-PeDrMaz5xnWzmb9Ig5jbIXS6OGHWt-WaBN4M_f1PF7LJ3q2Cv-hA1alsue4iWYMpgjTK1HDwhMWzFgc9yxZ5UfK6fyPDAccJOe9vdssQzr8Ohv33-JKWm8DBRh__EgbH_4SboEqS95mYxGH2JIp8tvQqavwJyq7HHI5ZzFccWzT8eMovoAMkMCEbsBD90sAB4cz3nrI0iFD0wKwuRTZgEn3sJAcUrMDJvIJkztIPvFK3JcNACbKg8B5HskywWIfVku9EmUd_9_pDCR2woa3WgNsEgk4ppkq3rd9Ud9xC8xrNYEkH30s85J0tmJ1DFGr9PZACHhXnPvt1t2rQPNj0tn8E_W0sCzpP9H-B8dWi4siuK-3zAjFUe_7R3_UKmhTLldEzZ_TNgpMWhMWYVZmxLpymULrRiut46rZ5KmediKVd26Rp9sgQujkMwuJbiPTkl6IPgHH65oI2gAxEplEWlCj2lOpF5jgk4QxB20fO96GIIu1gkxn87AFsytDZZaRxAnECKwBBeVMpf9i3rziwLCvE9bmnp5Jr2dGksj_XjuhXA-tVqaSveOkpC6_4vTDb_jvQaWZvR_2qGyccZHkAbh-Ac4OxlYr1MonDPE7O3-V_k97baLv0RyK5SEJlUheLy_qxMQWH1gAKD5I7b0qo9TR4lMhBtrFIYgazYQ64xbS4h8NoSkBbHLmxhK2UC8kOGLqPuMdFykBG9EsJIMoQrwPAT9NWdPXyPTsotOY0mtJr_s66B6eKLKey3yFCIJwzrIpfjZtOU6BOgLA3HNjPqguzgNO4vAftRMiEOmru0ALNgHBhaWauFVSu4hacykuPQajx8X3FslwBT8ax3UNAHnUEhq9uzl6XpUTjCrVUIQMSgu2M1kZ81jVH0pQMF02HaEp7YdXGsfvm7KV_M1RzoI0xd3TZANhYGOBo54B4hpSQGPkyz9vNHsjAU7GEJgMavKXxbs7RwF21TilgSrogdceI-Qxh48mpiMNVzIXCHGXPSwAgOd93AHLOzbUW3neaV5qZZoidS_wezasAZNaoywYDgdGNvm-Vj-pkFlRZmHFdLDsYOHPKk-O0skK1182nQHsjYhw1cczsJdwuyFxEpieDwTC58NgE10iO7-sjPUMm_RYNnP0CXPL6l6BUx0Z1m4M7mFEOoYvLnWwc6doQQeCiKdjLOf4prhUF3E8ikI0BngewL4tVNGLyMLKnR1MMLvyrso_3zyFKVoQCd5F7s-lrVQyc02EUmdIoQqjYICou0289Gqe1QHIhwLNAlqa1BwzpDBtMujxpHpO4E3_e5MZ0tPcYn2YyTxOFLHnp10tdf_vf_F1qxZSugT8PrgknW8ozGIMl9lw4489Qry4U65RcpRBHPb9p_dgDFIF_Pa0d2NTMsoV2kzyG3RIuh4YxiqXHXreBC69LN-kV20ewsekaMLiWg-S4J3uoPKaO9luk1IoLTWQOefUSa0vWL08GRy1u1A8HlCB3bGEOv99Tx5DRGqx7tSaLlbZXq-Ti2a1jZq8kKFPwa5eFTtjKmF6zKPJI4umAqMWTIh2FPf753EpzqO2hF-2o9Oii0nMuKZU2k2dpdpZ6ocQhobikljF0L9fRjT-TrAL20qQIm6uhxncwmw_U4YLOglhobea-zLXVc5ljSxBsXlPXmvSLXBEMiyEJJqysjAdLV6tfdKLRaxu_6qhtF9pzNVSBgwEnl4W8iD-ScVsI3UG2VQUH7iVRdeuCAfsFthcCxxkFdlkB4USe0etTB6cRZ7Oj-G777eBwkp5jdw_Qd2JCUBhgRqs52dcBbpcqaA9EMhaqYMr6Cp0oZyYc7hmdWIpKaSmM1Hw4mOWAOqMQ-KAmCFfRwzxrjH3amy_HOtiFvTIU9i5uKxtZsTGR_QXNZuU7MEOs6t4A7nfVknSPUYDTxMnS1jIjxPlsZHxzhDjPNvCDfyNK47Kzm7wxV1NUtcXdyHhNs9ddrVDIqxOVelXQu3CoQuNg8xBMY0EMCEETDPWIEHUYg6tqXVu1EKfIG-Eb5tqezrcQJMZQI1qQ1YRltvaM1cL6LReJaYlUKsVU_IAbL2QDpjmSGUeg0JgD-E234KpKQ5gZidhLWCJxYSE-OyJY8Fr4C8wdRMXvTLvsTF0n_TaZtY6-s1PSeuTxHWrzu62OPtpdsBd6cITcpocL55TQ70bdX9NOT7AKfqPh23x1qz9nR5stmzjX0QKqhxhnkPaxo3b0_Vw_ucKl-W2k9_JwTIJ1-JRlUwYRz-3R-2WCPrY5lrrDiPNjXBu-bGLsjdJRcT1-QhYfqFhivMLPSHPt7AJ2DByctvbWIJ-BfIM-IORnaWMW33UStLXwIZu3Td3z9HJxWlS8M53TpOzzbTbAiFdtQCq3jePRYAPEpMCK8QR1VJ_oWvAFaR1EUo8y1TrET4KEDKR-XaGZ8MaNjcuHV-Nn8yWfVOEqoxoZo1FWcftwpSj7zhUOjmSbalDnOf6BAmnKTSFMCxHiT-a-8hgVD_213CllQOazqKiF_B5G_pxdT-HM_9_FbTWHEh0Vq6swXV0fYO4fjwaikY1M0jJAGicOuDUJnCukpsxlvW-zOz984vnNaudN__7M8Wcut-aFr5x1_N2mlnqN1-2yBRpc6trx95vX5RiMTz_Zd262yPueO-Sazjxz54YNbvTQRGGe17ysvvz4v4aPOzxLbe0_qCfTeBN1Ln87Pzi0ccL9a7CTuyKHnVvTnY0XH2puFoiBBR4ezvbwXmwn-yab-7H0TgP-ldvcctEpcXhy9t9_TjT-bdc5Zpa51tEqHpy_C-Nz6I7l6B8njnTfgyJ0eP12s9Nn-8lr4T_rvqfr9sq--hXRjw7tST7zBv8bfJ95BtrIyX-F9DbW_7K-ismPTE3O1bYeVF90_v7dE2O_u5wP6X8xe-f7xYZzrA980Yvzug1EM-Q_Ph9DedPRX7EOEY571ehzSNxqv2LPAVH7Chd85Xcv9WnXDgL6gu7J3-FigyPsEuB8A_NqB0IB5zU3_tHdASJYNiBj29FrH7kDHB_Y-9FFJaO7DucjSUi9VCSl7CBW1NeZsjynknNtR0r4ionijjOulkZ6T1SBB_1E_78j8QD_IXNWrw0c5_Rw4IJjD_Ril7sHvRy2iw_frA-Syvj5ffSxwGQyfoTLjhsvsKGclnHwRKT38wcyj03NaXx04TcElCPv0In7P6LO2HG3ywZcMagGDNAyb31o_QUMN84j9w9dEsTqmAGpC-qQJCuGrS9porq_aSeq996eXafQH2bwIjsbdL2y4jB6Fx-z2xGvKD8N2ww_vG5nD599wu45x9zHHd-cxR9HhFUuo_5yDi18nxmvo4PbWlNQbKX9MJDwg6CfgqxOMKXS1Db8Rmrlz6X3SGG7vX5oqv7SkBHv87K-P_CgLTdm8ztess4R8jcp5td1Eq-v9k83HfeIMV53Xjwu2fKPg__pRfDgCNxlsSfS5Ivbc-7kGTg-X0bLCU6heQ7541mru-78Ee_zh_FK95EHIEV8wH_okLI5FRCP2RU6Q0OjdyQ5wMFxcW-TxAGfVmL2yzRIZZGU0nSt4rge2zTLF1LVRx0wEBDpLX7uoAgDLR74wMKYMdUVl6EvGeo9yI_W5CjV3UqM7uC6aR2mqu8HJNJehsvShIXZ9JDLBmC6F_JiQPHM7IGfnCs4xzpHFh_bE8MiZ9gGOgugbw4Zv4dzd8bie7Xdxzkq9DI6d0JLlxgxX81hfVdz2fK6fHt15cb6xHyR96pxxpBmuBHzBslaxY6f87b0JNt8P8cRyci9LeVlp3GjgP5AoFWgFVsHwDRKWOWjEH5-fM7NAN35HDnMZ6j7wuEw9Z_Wq9M6icf5LupSBfe6gk12ZD05kkfX1Jgb2fvgDwMPSVvX9Qb1-7_j_07JuZrcZc-vXfOGOLntsHx2nfLp3wrpUMoBN6xaH2kYftfkM_YE3zP_2W2ucdvLnG7lwQ7XzW1yVT_oFnxnujsu63lnMt0T4_2mi5ecclqZOFMLCR_jD3e_6hfva69vLD46-avl_ZNjlgq6t_CRBH8rvVWeAfnfZJ4jXWHdXzj9fG4E3nWERZKXhyz-K707vVefE6J_8UFshb9G2xZZv2z1lhK7TX37ovN3GhCetOgkd2_JxFuqzdDsPaH670k69YHaet7YXXhAnU_S5J1SYIINlIEPcKO69Nw0E21BBmuei9u0ZQJr10hS_naJIDc17kNtB9mPaDBLJowyIT-L0HD22aSsakurTahchtJI93COgGhaiH6ImN0xqw8O0UGZlIPgE5PYvtdooKu2amfFARE7uMJdGaNqcs5hQkdkxC4fELwDBWE0ahLTJQsgEEyqW4GpgUzllSEXmHBNAnHapzgW-H7PkF4RFqYir9bgfHlQFB9og3k0IijQSA3QVRdiAsvn3ud2AAcaRSoFGBUNRldrPSXt6k2cfb9cNaq7V_TKxtZ8oc2OaaluWOdAp3z2u7yIwwgakbJpc_W7zwsDa9Sf8x5BdO89iMzftvEzqrcGe5PoAfMdWE5gDF7rO9kMvOHgo1zC1kvb0N_b4Nhny31jdk4leCHffU-JOgKyDklHa54FyVo-t9TLIdsByWAMxshoHP_Vps8IXvdp2BnrD3P2LhVXvMOe7UjYPvesOAZGFuwZVw7e850zHcZ3NsEcnsVyWim-Nr_QhUMLczOmficZGNm_CdoeuoRJVf7Yb7bPwPxz8atxMrEyBwPChur0ebHHh_yIPQOG-NCOjfy0NVekL1--MmPLs3ybMc_5dPAAsL_oPWO4l9K-Obfv5jt1rh0Lj3yOXfizLjeIX4Vx5uOa733HZMXhP-wc-DJjI7Tnh8rorDDL-iut5Vv9nuzytN12xJ8vDvwmOy9guabNDcjP_peIoXT8292OVaN9CY2nzOb6_Q_IMOXb9fv897Xq0t_OhVdMnms3ci81hXv3eRzCywe6XBn_9u4GY6Ps2X-pnIJvVvO6VGw-9fr4H96xpg1I1tM8OSAkFtchIuxRiGa7-9uYpSSnOdbiQJyLmp5UV_taqnht_JevtVxg9h-VlqTWRVnCkhHFR3nX3Rqi_vJt7FVrlbvCk0nRIIyStqUs7IS9dD2UQfOJKEJ6fewrJcGxgeWd0AjdLDvJd1847iPOBmE7l4IY8YCtEYAOUg94tmPHibUgGo9FClBxXztMiGFgJBLuFbN_wFmMXwqJF4QHvYPtMOh58PZN_9PD55onZhtVJcV45ck35GFY8AQn1SWEvIhdwnhA8Il6ko94idZ-I7K41Lh3yCCqvUM5q47L04P9SaAiBAmvASzpGy8P1XjWfxycRFTRjR4VCau3tltjY2qvSVovqNNW3WpQDK1hecHVlzLh7U1eEVqVyNFCExeFbGXiPcBIHlTkYXcDDg7BOQDFXmMyVqL6AeBKuXrCDviuCvxTTt4_LYMS1AfCdF6pWXr09VZtMbOBrkQBYB2C6uFoJ3q3QgwE2_6QQmLXWLMJ7o7ds-6cUR-pJ6m92bnxDGkEA7xMVgWrEl7S7hof9Xp5Zx2Rn-6gmVeEgrQNL7CtLf-hS0RcRgrenLewl_V-cEZ9YKHEqY-zQl0jXebE8GYQ9DiBzTS-9PD3cJKLHElMqj84UEDjpir4ivLMlEzKX6b3qxqZKOJ9MpkvC4HaxoAtSgbpFp0nggw4oHAdSZnw5Z3LwgWVbZdazay6zE9tCdXLZbi5joRWf4XlCSkaDdLJr44ZNAPDbydIgC_joe2juk9APQLgtOlX82xEMeZ4dvX7eGHxRwIzfgIoL1dKMQtiNg2KArQB1OVNX0ye1PpmQYGllVOCvJtrzfa_1xICOAmLriOERghWM97BOwLaNDEaBE-qSrRGq0eaBhC2xajc85dDL45JHYqXBiDiOmyHO2ngm9gEfia38uuEyd8wkLbFct2f0OvDhjuhx_2xOsEQWuTaBX5y9lf3KcQGsNV2eFwRSWJiQiwFFPfQr4aLkt5lojiIkdVZ3EQR67IEnEsreBRRIxXTkmz1UhDTg3zBQAR0OHEnmvdIfHJCqAYPtm8F8Olbhv8cFiGlucXfDJxhCJWBkh94h-KEMWU76WGiE4j5uqqt5ww_ESmuHYLxRP5_sMAR8DyJnZOP0HGoTAGZDXREYDT3VjcAdkSJN5v9DAyPg_rB64v0JiiKllo6we4V5FPAVgRNkV-DcB5f2aNRn0UEiWXRx_hNHw7e0LWbv8q3lfnKkWROiumRT2JJ4fv9Y0gqOshphF45w2C0SAtApMbeHQ77BpTnKRPXJ6cHFQat-0eAgwsDHGid0WBN6FKBDA1PxqHpQU8QdIw7Y315mB5YAFfONId9KnguXCbdrOkKtiJt_ADrGA3_laeXDbMLo-a4FxrrFGJYELWaQdq39MyecGx8P8HIUnpRCD2XGd2St6K8tvGDDRxKYeL8gRVePLVt_Sji53npmKcJ5OG55fT0n1H1Qt-PpvsF1AKCZCQ2f7Tak4VV2vtRRhxxLAR7UdDllnUNPTf6X8DV8I_A4kbN_9_TGWekvqGr_mc0VVXlw5-f5TfsTVOWBh02gH0-piKGZGq-JLczFgEbWps-Ml3rFV1NsUnCB6NdvY_CiwaJzAi1_DO1DD29rqQB7i0P4_D9_HluGKOfERttFA", + "payloadType": "application/vnd.stellaops.zastava.kit+tzst;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "RElsvFwYdlb3QFJn3fTaj8tlOsR5Kzw-8WGszR-OjlRfa2aVXi5qiQJBAawGtbMLinR4x2Q_njNxX1eSwAbdBg" + } + ] +} diff --git a/docs/modules/zastava/schemas/README.md b/docs/modules/zastava/schemas/README.md new file mode 100644 index 000000000..9bff2b5f0 --- /dev/null +++ b/docs/modules/zastava/schemas/README.md @@ -0,0 +1,19 @@ +# Zastava schemas (runtime & admission) + +## Canonicalisation & hashing +- JSON is stored with sorted keys and two-space indentation; hashes use **JCS-style** encoding: `json.dumps(payload, separators=(',', ':'), sort_keys=True)`. +- `payload_hash` fields in examples and exports are computed from the canonical payload bytes and formatted as `sha256:`. +- Schema negotiation stays on the `zastava.*@v1.x` line; breaking changes bump the major version. + +## DSSE signing +- Payload types: + - `application/vnd.stellaops.zastava.schema+json;name=observer_event;version=1` + - `application/vnd.stellaops.zastava.schema+json;name=webhook_admission;version=1` +- Ed25519 public key (base64url, no padding): `mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc`. +- Signatures are emitted as `.dsse` with DSSEv1 pre-auth encoding over the raw file bytes. +- Regenerate signatures with `docs/modules/zastava/kit/verify.sh` prerequisites (Python + cryptography) and the private key held offline. + +## Test vectors +- Example payloads: `schemas/examples/*.json`. +- Signed exports: `exports/observer_events.ndjson(.dsse)` and `exports/webhook_admissions.ndjson(.dsse)`. +- Kit verification aggregates all signatures via `kit/verify.sh`. diff --git a/docs/modules/zastava/schemas/examples/observer_event.example.json b/docs/modules/zastava/schemas/examples/observer_event.example.json index b1f5afc6d..132f9e072 100644 --- a/docs/modules/zastava/schemas/examples/observer_event.example.json +++ b/docs/modules/zastava/schemas/examples/observer_event.example.json @@ -1,19 +1,19 @@ { - "tenant_id": "tenant-a", - "project_id": "proj-123", - "sensor_id": "observer-01", + "event_type": "runtime_fact", "firmware_version": "1.2.3", - "policy_hash": "sha256:deadbeef", "graph_revision_id": "graph-r1", "ledger_id": "ledger-789", - "replay_manifest": "manifest-r1", - "event_type": "runtime_fact", - "observed_at": "2025-12-02T00:00:00Z", "monotonic_nanos": 123456789, + "observed_at": "2025-12-02T00:00:00Z", "payload": { - "process": "nginx", - "pid": 4242 + "pid": 4242, + "process": "nginx" }, - "payload_hash": "sha256:payloadhash", - "signature": "dsse://observer-event" + "payload_hash": "sha256:7476a5068a3f0780c552f81c90d061d9e39c37f425a243ecff961b08676546fd", + "policy_hash": "sha256:deadbeef", + "project_id": "proj-123", + "replay_manifest": "manifest-r1", + "sensor_id": "observer-01", + "signature": "dsse://observer-events/2025-12-02/observer_events.ndjson.dsse#line1", + "tenant_id": "tenant-a" } diff --git a/docs/modules/zastava/schemas/examples/webhook_admission.example.json b/docs/modules/zastava/schemas/examples/webhook_admission.example.json index 927a0f372..bc2e3d090 100644 --- a/docs/modules/zastava/schemas/examples/webhook_admission.example.json +++ b/docs/modules/zastava/schemas/examples/webhook_admission.example.json @@ -1,21 +1,34 @@ { - "tenant_id": "tenant-a", - "project_id": "proj-123", - "request_uid": "abcd-1234", - "resource_kind": "Deployment", - "namespace": "prod", - "workload_name": "api", - "policy_hash": "sha256:deadbeef", + "bypass_waiver_id": null, + "decision": "allow", + "decision_at": "2025-12-02T00:00:00Z", + "decision_reason": "surface cache fresh", "graph_revision_id": "graph-r1", "ledger_id": "ledger-789", - "replay_manifest": "manifest-r1", "manifest_pointer": "surfacefs://cache/sha256:abc", - "decision": "allow", - "decision_reason": "surface cache fresh", - "decision_at": "2025-12-02T00:00:00Z", "monotonic_nanos": 2233445566, + "namespace": "prod", + "payload": { + "images": [ + { + "digest": "sha256:abcd", + "name": "ghcr.io/acme/api:1.2.3", + "sbom_referrer": true, + "signed": true + } + ], + "manifest_pointer": "surfacefs://cache/sha256:abc", + "policy_hash": "sha256:deadbeef", + "verdict": "allow" + }, + "payload_hash": "sha256:36bfb2bc81b7050bbb508e12cafe7ad5a51336aad397ef3a23b0e258aed73dc6", + "policy_hash": "sha256:deadbeef", + "project_id": "proj-123", + "replay_manifest": "manifest-r1", + "request_uid": "abcd-1234", + "resource_kind": "Deployment", "side_effect": "none", - "bypass_waiver_id": null, - "payload_hash": "sha256:payloadhash", - "signature": "dsse://webhook-admission" + "signature": "dsse://webhook-admissions/2025-12-02/webhook_admissions.ndjson.dsse#line1", + "tenant_id": "tenant-a", + "workload_name": "api" } diff --git a/docs/modules/zastava/schemas/observer_event.schema.json b/docs/modules/zastava/schemas/observer_event.schema.json index a1c25634d..c3e125ca1 100644 --- a/docs/modules/zastava/schemas/observer_event.schema.json +++ b/docs/modules/zastava/schemas/observer_event.schema.json @@ -1,8 +1,67 @@ { "$id": "https://stella-ops.org/schemas/zastava/observer_event.schema.json", "$schema": "http://json-schema.org/draft-07/schema#", - "title": "Zastava Observer Event", - "type": "object", + "properties": { + "event_type": { + "enum": [ + "runtime_fact", + "drift", + "policy_violation", + "heartbeat" + ] + }, + "firmware_version": { + "minLength": 1, + "type": "string" + }, + "graph_revision_id": { + "minLength": 1, + "type": "string" + }, + "ledger_id": { + "type": "string" + }, + "monotonic_nanos": { + "type": "integer" + }, + "observed_at": { + "format": "date-time", + "type": "string" + }, + "payload": { + "description": "Canonical runtime payload (JCS) used for hashing.", + "type": "object" + }, + "payload_hash": { + "description": "sha256 over canonical JSON (JCS) of payload", + "pattern": "^sha256:[0-9a-f]{64}$", + "type": "string" + }, + "policy_hash": { + "minLength": 1, + "type": "string" + }, + "project_id": { + "minLength": 1, + "type": "string" + }, + "replay_manifest": { + "type": "string" + }, + "sensor_id": { + "minLength": 1, + "type": "string" + }, + "signature": { + "description": "DSSE envelope reference", + "pattern": "^dsse://[A-Za-z0-9._:/-]+$", + "type": "string" + }, + "tenant_id": { + "minLength": 1, + "type": "string" + } + }, "required": [ "tenant_id", "project_id", @@ -12,23 +71,10 @@ "graph_revision_id", "event_type", "observed_at", + "payload", "payload_hash", - "signature" + "signature" ], - "properties": { - "tenant_id": { "type": "string" }, - "project_id": { "type": "string" }, - "sensor_id": { "type": "string" }, - "firmware_version": { "type": "string" }, - "policy_hash": { "type": "string" }, - "graph_revision_id": { "type": "string" }, - "ledger_id": { "type": "string" }, - "replay_manifest": { "type": "string" }, - "event_type": { "enum": ["runtime_fact", "drift", "policy_violation", "heartbeat"] }, - "observed_at": { "type": "string", "format": "date-time" }, - "monotonic_nanos": { "type": "integer" }, - "payload": { "type": "object" }, - "payload_hash": { "type": "string", "description": "sha256 over canonical JSON (JCS) of payload" }, - "signature": { "type": "string", "description": "DSSE envelope reference" } - } + "title": "Zastava Observer Event", + "type": "object" } diff --git a/docs/modules/zastava/schemas/observer_event.schema.json.dsse b/docs/modules/zastava/schemas/observer_event.schema.json.dsse new file mode 100644 index 000000000..57fcdd7e3 --- /dev/null +++ b/docs/modules/zastava/schemas/observer_event.schema.json.dsse @@ -0,0 +1,10 @@ +{ + "payload": "ewogICIkaWQiOiAiaHR0cHM6Ly9zdGVsbGEtb3BzLm9yZy9zY2hlbWFzL3phc3RhdmEvb2JzZXJ2ZXJfZXZlbnQuc2NoZW1hLmpzb24iLAogICIkc2NoZW1hIjogImh0dHA6Ly9qc29uLXNjaGVtYS5vcmcvZHJhZnQtMDcvc2NoZW1hIyIsCiAgInByb3BlcnRpZXMiOiB7CiAgICAiZXZlbnRfdHlwZSI6IHsKICAgICAgImVudW0iOiBbCiAgICAgICAgInJ1bnRpbWVfZmFjdCIsCiAgICAgICAgImRyaWZ0IiwKICAgICAgICAicG9saWN5X3Zpb2xhdGlvbiIsCiAgICAgICAgImhlYXJ0YmVhdCIKICAgICAgXQogICAgfSwKICAgICJmaXJtd2FyZV92ZXJzaW9uIjogewogICAgICAibWluTGVuZ3RoIjogMSwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJncmFwaF9yZXZpc2lvbl9pZCI6IHsKICAgICAgIm1pbkxlbmd0aCI6IDEsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAibGVkZ2VyX2lkIjogewogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgIm1vbm90b25pY19uYW5vcyI6IHsKICAgICAgInR5cGUiOiAiaW50ZWdlciIKICAgIH0sCiAgICAib2JzZXJ2ZWRfYXQiOiB7CiAgICAgICJmb3JtYXQiOiAiZGF0ZS10aW1lIiwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJwYXlsb2FkIjogewogICAgICAiZGVzY3JpcHRpb24iOiAiQ2Fub25pY2FsIHJ1bnRpbWUgcGF5bG9hZCAoSkNTKSB1c2VkIGZvciBoYXNoaW5nLiIsCiAgICAgICJ0eXBlIjogIm9iamVjdCIKICAgIH0sCiAgICAicGF5bG9hZF9oYXNoIjogewogICAgICAiZGVzY3JpcHRpb24iOiAic2hhMjU2IG92ZXIgY2Fub25pY2FsIEpTT04gKEpDUykgb2YgcGF5bG9hZCIsCiAgICAgICJwYXR0ZXJuIjogIl5zaGEyNTY6WzAtOWEtZl17NjR9JCIsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAicG9saWN5X2hhc2giOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInByb2plY3RfaWQiOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInJlcGxheV9tYW5pZmVzdCI6IHsKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJzZW5zb3JfaWQiOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInNpZ25hdHVyZSI6IHsKICAgICAgImRlc2NyaXB0aW9uIjogIkRTU0UgZW52ZWxvcGUgcmVmZXJlbmNlIiwKICAgICAgInBhdHRlcm4iOiAiXmRzc2U6Ly9bQS1aYS16MC05Ll86Ly1dKyQiLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInRlbmFudF9pZCI6IHsKICAgICAgIm1pbkxlbmd0aCI6IDEsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0KICB9LAogICJyZXF1aXJlZCI6IFsKICAgICJ0ZW5hbnRfaWQiLAogICAgInByb2plY3RfaWQiLAogICAgInNlbnNvcl9pZCIsCiAgICAiZmlybXdhcmVfdmVyc2lvbiIsCiAgICAicG9saWN5X2hhc2giLAogICAgImdyYXBoX3JldmlzaW9uX2lkIiwKICAgICJldmVudF90eXBlIiwKICAgICJvYnNlcnZlZF9hdCIsCiAgICAicGF5bG9hZCIsCiAgICAicGF5bG9hZF9oYXNoIiwKICAgICJzaWduYXR1cmUiCiAgXSwKICAidGl0bGUiOiAiWmFzdGF2YSBPYnNlcnZlciBFdmVudCIsCiAgInR5cGUiOiAib2JqZWN0Igp9Cg", + "payloadType": "application/vnd.stellaops.zastava.schema+json;name=observer_event;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "axmdd1ucHyZyJMAyLzWmpuai7VrS20QenSDQyXRKlmtsAF4Zl4Ke_cHy8konBStBCoJgGA3SM2236QgAbkQMBw" + } + ] +} diff --git a/docs/modules/zastava/schemas/webhook_admission.schema.json b/docs/modules/zastava/schemas/webhook_admission.schema.json index c08346b3b..e841575ae 100644 --- a/docs/modules/zastava/schemas/webhook_admission.schema.json +++ b/docs/modules/zastava/schemas/webhook_admission.schema.json @@ -1,8 +1,91 @@ { "$id": "https://stella-ops.org/schemas/zastava/webhook_admission.schema.json", "$schema": "http://json-schema.org/draft-07/schema#", - "title": "Zastava Webhook Admission", - "type": "object", + "properties": { + "bypass_waiver_id": { + "type": "string" + }, + "decision": { + "enum": [ + "allow", + "deny", + "dry-run" + ] + }, + "decision_at": { + "format": "date-time", + "type": "string" + }, + "decision_reason": { + "minLength": 1, + "type": "string" + }, + "graph_revision_id": { + "minLength": 1, + "type": "string" + }, + "ledger_id": { + "type": "string" + }, + "manifest_pointer": { + "description": "Surface.FS manifest pointer", + "type": "string" + }, + "monotonic_nanos": { + "type": "integer" + }, + "namespace": { + "minLength": 1, + "type": "string" + }, + "payload": { + "description": "AdmissionReview payload (canonical JSON) hashed via payload_hash", + "type": "object" + }, + "payload_hash": { + "pattern": "^sha256:[0-9a-f]{64}$", + "type": "string" + }, + "policy_hash": { + "minLength": 1, + "type": "string" + }, + "project_id": { + "minLength": 1, + "type": "string" + }, + "replay_manifest": { + "type": "string" + }, + "request_uid": { + "minLength": 1, + "type": "string" + }, + "resource_kind": { + "minLength": 1, + "type": "string" + }, + "side_effect": { + "enum": [ + "none", + "mutating", + "bypass" + ] + }, + "signature": { + "description": "DSSE envelope reference", + "pattern": "^dsse://[A-Za-z0-9._:/-]+$", + "type": "string" + }, + "tenant_id": { + "minLength": 1, + "type": "string" + }, + "workload_name": { + "minLength": 1, + "type": "string" + } + }, "required": [ "tenant_id", "project_id", @@ -16,27 +99,10 @@ "decision_reason", "decision_at", "manifest_pointer", + "payload", + "payload_hash", "signature" ], - "properties": { - "tenant_id": { "type": "string" }, - "project_id": { "type": "string" }, - "request_uid": { "type": "string" }, - "resource_kind": { "type": "string" }, - "namespace": { "type": "string" }, - "workload_name": { "type": "string" }, - "policy_hash": { "type": "string" }, - "graph_revision_id": { "type": "string" }, - "ledger_id": { "type": "string" }, - "replay_manifest": { "type": "string" }, - "manifest_pointer": { "type": "string", "description": "Surface.FS manifest pointer" }, - "decision": { "enum": ["allow", "deny", "dry-run"] }, - "decision_reason": { "type": "string" }, - "decision_at": { "type": "string", "format": "date-time" }, - "monotonic_nanos": { "type": "integer" }, - "side_effect": { "enum": ["none", "mutating", "bypass"] }, - "bypass_waiver_id": { "type": "string" }, - "payload_hash": { "type": "string" }, - "signature": { "type": "string", "description": "DSSE envelope reference" } - } + "title": "Zastava Webhook Admission", + "type": "object" } diff --git a/docs/modules/zastava/schemas/webhook_admission.schema.json.dsse b/docs/modules/zastava/schemas/webhook_admission.schema.json.dsse new file mode 100644 index 000000000..d649f2dc3 --- /dev/null +++ b/docs/modules/zastava/schemas/webhook_admission.schema.json.dsse @@ -0,0 +1,10 @@ +{ + "payload": "ewogICIkaWQiOiAiaHR0cHM6Ly9zdGVsbGEtb3BzLm9yZy9zY2hlbWFzL3phc3RhdmEvd2ViaG9va19hZG1pc3Npb24uc2NoZW1hLmpzb24iLAogICIkc2NoZW1hIjogImh0dHA6Ly9qc29uLXNjaGVtYS5vcmcvZHJhZnQtMDcvc2NoZW1hIyIsCiAgInByb3BlcnRpZXMiOiB7CiAgICAiYnlwYXNzX3dhaXZlcl9pZCI6IHsKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJkZWNpc2lvbiI6IHsKICAgICAgImVudW0iOiBbCiAgICAgICAgImFsbG93IiwKICAgICAgICAiZGVueSIsCiAgICAgICAgImRyeS1ydW4iCiAgICAgIF0KICAgIH0sCiAgICAiZGVjaXNpb25fYXQiOiB7CiAgICAgICJmb3JtYXQiOiAiZGF0ZS10aW1lIiwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJkZWNpc2lvbl9yZWFzb24iOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgImdyYXBoX3JldmlzaW9uX2lkIjogewogICAgICAibWluTGVuZ3RoIjogMSwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJsZWRnZXJfaWQiOiB7CiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAibWFuaWZlc3RfcG9pbnRlciI6IHsKICAgICAgImRlc2NyaXB0aW9uIjogIlN1cmZhY2UuRlMgbWFuaWZlc3QgcG9pbnRlciIsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAibW9ub3RvbmljX25hbm9zIjogewogICAgICAidHlwZSI6ICJpbnRlZ2VyIgogICAgfSwKICAgICJuYW1lc3BhY2UiOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInBheWxvYWQiOiB7CiAgICAgICJkZXNjcmlwdGlvbiI6ICJBZG1pc3Npb25SZXZpZXcgcGF5bG9hZCAoY2Fub25pY2FsIEpTT04pIGhhc2hlZCB2aWEgcGF5bG9hZF9oYXNoIiwKICAgICAgInR5cGUiOiAib2JqZWN0IgogICAgfSwKICAgICJwYXlsb2FkX2hhc2giOiB7CiAgICAgICJwYXR0ZXJuIjogIl5zaGEyNTY6WzAtOWEtZl17NjR9JCIsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAicG9saWN5X2hhc2giOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInByb2plY3RfaWQiOiB7CiAgICAgICJtaW5MZW5ndGgiOiAxLAogICAgICAidHlwZSI6ICJzdHJpbmciCiAgICB9LAogICAgInJlcGxheV9tYW5pZmVzdCI6IHsKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJyZXF1ZXN0X3VpZCI6IHsKICAgICAgIm1pbkxlbmd0aCI6IDEsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAicmVzb3VyY2Vfa2luZCI6IHsKICAgICAgIm1pbkxlbmd0aCI6IDEsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAic2lkZV9lZmZlY3QiOiB7CiAgICAgICJlbnVtIjogWwogICAgICAgICJub25lIiwKICAgICAgICAibXV0YXRpbmciLAogICAgICAgICJieXBhc3MiCiAgICAgIF0KICAgIH0sCiAgICAic2lnbmF0dXJlIjogewogICAgICAiZGVzY3JpcHRpb24iOiAiRFNTRSBlbnZlbG9wZSByZWZlcmVuY2UiLAogICAgICAicGF0dGVybiI6ICJeZHNzZTovL1tBLVphLXowLTkuXzovLV0rJCIsCiAgICAgICJ0eXBlIjogInN0cmluZyIKICAgIH0sCiAgICAidGVuYW50X2lkIjogewogICAgICAibWluTGVuZ3RoIjogMSwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfSwKICAgICJ3b3JrbG9hZF9uYW1lIjogewogICAgICAibWluTGVuZ3RoIjogMSwKICAgICAgInR5cGUiOiAic3RyaW5nIgogICAgfQogIH0sCiAgInJlcXVpcmVkIjogWwogICAgInRlbmFudF9pZCIsCiAgICAicHJvamVjdF9pZCIsCiAgICAicmVxdWVzdF91aWQiLAogICAgInJlc291cmNlX2tpbmQiLAogICAgIm5hbWVzcGFjZSIsCiAgICAid29ya2xvYWRfbmFtZSIsCiAgICAicG9saWN5X2hhc2giLAogICAgImdyYXBoX3JldmlzaW9uX2lkIiwKICAgICJkZWNpc2lvbiIsCiAgICAiZGVjaXNpb25fcmVhc29uIiwKICAgICJkZWNpc2lvbl9hdCIsCiAgICAibWFuaWZlc3RfcG9pbnRlciIsCiAgICAicGF5bG9hZCIsCiAgICAicGF5bG9hZF9oYXNoIiwKICAgICJzaWduYXR1cmUiCiAgXSwKICAidGl0bGUiOiAiWmFzdGF2YSBXZWJob29rIEFkbWlzc2lvbiIsCiAgInR5cGUiOiAib2JqZWN0Igp9Cg", + "payloadType": "application/vnd.stellaops.zastava.schema+json;name=webhook_admission;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "Vk0mACAjBtUuVn_S2M5HU81zMbH8wDCQYOHVsft7cmxl0JbDrSIA9z3xlTI5JiT7DYOGsDUc96dlC1njldN4Aw" + } + ] +} diff --git a/docs/modules/zastava/thresholds.yaml.dsse b/docs/modules/zastava/thresholds.yaml.dsse new file mode 100644 index 000000000..49cacd97d --- /dev/null +++ b/docs/modules/zastava/thresholds.yaml.dsse @@ -0,0 +1,10 @@ +{ + "payload": "dmVyc2lvbjogMQp1cGRhdGVkX2F0OiAyMDI1LTEyLTAyVDAwOjAwOjAwWgpidWRnZXRzOgogIGxhdGVuY3lfbXNfcDk1OiAyNTAKICBlcnJvcl9yYXRlOiAwLjAxCiAgZHJvcF9yYXRlOiAwLjAwNQpidXJuX3JhdGVzOgogIGFkbWlzc2lvbl9kZW5pZXNfcGVyX21pbjogNQogIG9ic2VydmVyX2RyaWZ0c19wZXJfaG91cjogMgogIGhlYXJ0YmVhdF9taXNzX21pbnV0ZXM6IDMKYWxlcnRzOgogIHRocmVzaG9sZF9jaGFuZ2U6IHRydWUKICBidXJuX3JhdGVfZXhjZWVkZWQ6IHRydWUKICBraWxsX3N3aXRjaF90cmlnZ2VyZWQ6IHRydWUKc2lnbmluZzoKICBwcmVkaWNhdGU6IHN0ZWxsYS5vcHMvemFzdGF2YVRocmVzaG9sZHNAdjEKICBkc3NlX3JlcXVpcmVkOiB0cnVlCg", + "payloadType": "application/vnd.stellaops.zastava.thresholds+yaml;version=1", + "signatures": [ + { + "keyid": "mpIEbYRL1q5yhN6wBRvkZ_0xXz3QUJPueJJ8sn__GGc", + "sig": "uQFBmx7vF4fj8uQsCiCN6VbxNS2m3XM-vJNFrj3rexL1PPzHH6IVtWRGexF7CsLrrpUV8U0AmS02S37vOk3zDA" + } + ] +} diff --git a/docs/product-advisories/31-Nov-2025 FINDINGS.md b/docs/product-advisories/31-Nov-2025 FINDINGS.md index 6e12b9ba6..a252712dd 100644 --- a/docs/product-advisories/31-Nov-2025 FINDINGS.md +++ b/docs/product-advisories/31-Nov-2025 FINDINGS.md @@ -44,6 +44,42 @@ This advisory consolidates late-November gap findings across Scanner, SBOM/VEX s 9. **CM9 — Ecosystem coverage**: Track coverage per ecosystem (container, Java, Python, .NET, Go, OS packages) and gaps for ingest support. 10. **CM10 — Error resilience & retries**: Standardize retry/backoff/error classification for ingest pipeline; surface diagnostics deterministically. +## OK (Offline Kit) Gaps — OK1–OK10 +1. **OK1 — Key manifest + PQ co-sign**: Record key IDs and PQ dual-sign toggle in bundle meta; rotate keys ≤90 days. Evidence: `out/mirror/thin/mirror-thin-v1.bundle.json` (`chain_of_custody.keyid`) and `layers/offline-kit-policy.json`. +2. **OK2 — Tool hashing/signing**: Hash build/sign/verify tools and pin them in bundle meta (`tooling.*`); DSSE envelopes cover manifest + bundle meta. +3. **OK3 — DSSE top-level manifest**: Ship DSSE for bundle meta (`mirror-thin-v1.bundle.dsse.json`) linking manifest, tarball, policies, and optional OCI layout. +4. **OK4 — Checkpoint freshness + mirror metadata**: Enforce `checkpoint_freshness_seconds` and timestamped `created` in bundle meta; require checkpoints in `transport-plan.json`. +5. **OK5 — Deterministic packaging flags**: Capture tar/gzip flags in `layers/offline-kit-policy.json` and verify via `scripts/mirror/verify_thin_bundle.py` determinism checks. +6. **OK6 — Scan/VEX/policy/graph hashes**: Include `layers/artifact-hashes.json` with digests for scan/vex/policy/graph fixtures and reference from bundle meta. +7. **OK7 — Time anchor bundling**: Embed `layers/time-anchor.json` digest in bundle meta and surface trust-root path for AIRGAP-TIME. +8. **OK8 — Transport/chunking + chain-of-custody**: Define chunk sizing, retry policy, and signed chain-of-custody in `layers/transport-plan.json` (includes build/sign digests + keyid). +9. **OK9 — Tenant/environment scoping**: Require `tenant`/`environment` fields in bundle meta; verifier enforces via `--tenant/--environment` flags. +10. **OK10 — Scripted verify + negative paths**: `scripts/mirror/verify_thin_bundle.py` validates required layers, DSSE, sidecars, tool hashes, and scope; fails fast on missing/stale artefacts. + +## RK (Rekor) Gaps — RK1–RK10 +1. **RK1 — DSSE/hashedrekord only**: `layers/rekor-policy.json` sets `rk1_enforceDsse=true` and routes both public/private to hashedrekord. +2. **RK2 — Payload size preflight + chunks**: `rk2_payloadMaxBytes=1048576` with chunking guidance in `transport-plan.json`. +3. **RK3 — Public/private routing policy**: Explicit routing map (`rk3_routing`) for shard-aware submission. +4. **RK4 — Shard-aware checkpoints**: `rk4_shardCheckpoint="per-tenant-per-day"` plus checkpoint freshness from bundle meta. +5. **RK5 — Idempotent submission keys**: `rk5_idempotentKeys=true` to prevent duplicate entries. +6. **RK6 — Sigstore bundles in kits**: `rk6_sigstoreBundleIncluded=true`; bundle meta lists DSSE artefacts for offline kits. +7. **RK7 — Checkpoint freshness bounds**: `rk7_checkpointFreshnessSeconds` mirrors bundle freshness budget. +8. **RK8 — PQ dual-sign options**: `rk8_pqDualSign` mirrors PQ toggle (env `PQ_CO_SIGN_REQUIRED`). +9. **RK9 — Error taxonomy/backoff**: Enumerated in `rk9_errorTaxonomy` and retried per `transport-plan.json` retry policy. +10. **RK10 — Policy/graph annotations**: `rk10_annotations` require policy + graph context inside DSSE/bundle records. + +## MS (Mirror Strategy) Gaps — MS1–MS10 +1. **MS1 — Signed/versioned mirror schemas**: `layers/mirror-policy.json` tracks `schemaVersion` + semver; DSSE of bundle meta ties schema to artefacts. +2. **MS2 — DSSE/TUF rotation policy (incl. PQ)**: `dsseTufRotationDays=30` and `pqDualSign` toggle documented in mirror policy and bundle meta. +3. **MS3 — Delta spec with tombstones/base hash**: Mirror policy `delta` enforces tombstones and base-hash requirements for deltas. +4. **MS4 — Time-anchor freshness enforcement**: `timeAnchorFreshnessSeconds` plus bundled `time-anchor.json` digest. +5. **MS5 — Tenant/env scoping**: Tenant/environment fields required in bundle meta; verifier flags mismatches. +6. **MS6 — Distribution integrity (HTTP/OCI/object)**: `distributionIntegrity` enumerates integrity strategies for each transport. +7. **MS7 — Chunking/size rules**: `chunking.sizeBytes` + `maxChunks` pinned in mirror policy and reflected in transport plan. +8. **MS8 — Standard verify script**: `verifyScript` references `scripts/mirror/verify_thin_bundle.py`; bundle meta recorded in DSSE envelope. +9. **MS9 — Metrics/alerts**: Mirror policy `metrics` marks build/import/verify signals required for observability. +10. **MS10 — SemVer/change log**: `changelog` block declares current format version; future bumps must be appended with deterministic notes. + ## Pending Families (to be expanded) The following gap families were referenced in November indices and still need detailed findings written out: - CV1–CV10 (CVSS v4 receipts), CVM1–CVM10 (momentum), FC1–FC10 (SCA fixture gaps), OB1–OB10 (onboarding), IG1–IG10 (implementor guidance), RR1–RR10 (Rekor receipts), SK1–SK10 (standups), MI1–MI10 (UI micro-interactions), PVX1–PVX10 (Proof-linked VEX UI), TTE1–TTE10 (Time-to-Evidence), AR-EP1…AR-VB1 (archived advisories revival), BP1–BP10 (SBOM→VEX proof pipeline), UT1–UT10 (unknown heuristics), CE1–CE10 (evidence patterns), ET1–ET10 (ecosystem fixtures), RB1–RB10 (reachability fixtures), G1–G12 / RD1–RD10 (reachability benchmark/dataset), UN1–UN10 (unknowns registry), U1–U10 (decay), EX1–EX10 (explainability), VEX1–VEX10 (VEX claims), BR1–BR10 (binary reachability), VT1–VT10 (triage), PL1–PL10 (plugin arch), EB1–EB10 (evidence baseline), EC1–EC10 (export center), AT1–AT10 (automation), OK1–OK10 / RK1–RK10 / MS1–MS10 (offline/mirror/Rekor kits), TP1–TP10 (task packs), AU1–AU10 (auth), CL1–CL10 (CLI), OR1–OR10 (orchestrator), ZR1–ZR10 (Zastava), NR1–NR10 (Notify), GA1–GA10 (graph analytics), TO1–TO10 (telemetry), PS1–PS10 (policy), FL1–FL10 (ledger), CI1–CI10 (Concelier ingest). diff --git a/docs/reachability/function-level-evidence.md b/docs/reachability/function-level-evidence.md index 4429912e1..1ad9e93e6 100644 --- a/docs/reachability/function-level-evidence.md +++ b/docs/reachability/function-level-evidence.md @@ -42,6 +42,7 @@ Out of scope: implementing disassemblers or symbol servers; those will be handle * Update analyzer contracts so every analyzer returns both `symbol_id` and `code_id`, with demangled names stored under the new `symbol` block. * Persist the data into `richgraph-v1` payloads and attach CAS URIs via `StellaOps.Scanner.Reachability`. * Deliver fixtures in `tests/reachability/StellaOps.ScannerSignals.IntegrationTests` that prove determinism (same hash when analyzer flags reorder). +* **Helper status (2025-12-02):** `SymbolId.ForBinaryAddressed` + `CodeId.ForBinarySegment` now encode `{file_hash, section, addr, name, linkage, length, code_block_hash}` with normalized hex addresses. Analyzers should start emitting these tuples instead of ad-hoc hashes. ### 3.2 Runtime + Signals (GAP-ZAS-002 / GAP-SIG-003) diff --git a/out/mirror/thin/milestone.json b/out/mirror/thin/milestone.json new file mode 100644 index 000000000..17533c36a --- /dev/null +++ b/out/mirror/thin/milestone.json @@ -0,0 +1,15 @@ +{ + "created": "2025-12-02T18:08:34Z", + "manifest": {"path": "mirror-thin-v1.manifest.json", "sha256": "1affb0b796ff037117b46aa1f1d8056a9c80755e925af058ea72132ba158becf"}, + "tarball": {"path": "mirror-thin-v1.tar.gz", "sha256": "fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d"}, + "dsse": {"path": "mirror-thin-v1.manifest.dsse.json", "sha256": "f4a2a99fdfa60b3bd98daf88faabcf5d525b7f4a40fad606a502c3e25f9b2a7f"}, + "bundle": {"path": "mirror-thin-v1.bundle.json", "sha256": "a3b16f5d1b74ffdf9aedbbfe9282d368dc3dcf70676c8ac7e8cdd984162e7f90"}, + "bundle_dsse": {"path": "mirror-thin-v1.bundle.dsse.json", "sha256": "5fd3025c03cc4c19708eeec8feaa129a4e567dcefd06cb01f251a38590f76dde"}, + "time_anchor": null + ,"policies": { + "transport": {"path": "transport-plan.json", "sha256": "df82a56d9bacb00a1882f5d6d9f9ba469b62b89bd949899b7049e123c1e65914"}, + "rekor": {"path": "rekor-policy.json", "sha256": "652df157628db73e9aa0110e7390f8773319c24530e00873afcfdf972644717e"}, + "mirror": {"path": "mirror-policy.json", "sha256": "d7059d4b9e7e207f2420520bf73cf69b644eec0e866f039a1f7d0dc2b3bc1192"}, + "offline": {"path": "offline-kit-policy.json", "sha256": "ae2513f9768f3f7c0b0994b54f539b2a933e1e851c25c26c8fe46fd963d90579"} + } +} diff --git a/out/mirror/thin/mirror-thin-v1.bundle.dsse.json b/out/mirror/thin/mirror-thin-v1.bundle.dsse.json new file mode 100644 index 000000000..f10a0447c --- /dev/null +++ b/out/mirror/thin/mirror-thin-v1.bundle.dsse.json @@ -0,0 +1,10 @@ +{ + "payload": "ewogICJhcnRpZmFjdHMiOiB7CiAgICAiYXJ0aWZhY3RfaGFzaGVzIjogewogICAgICAicGF0aCI6ICJhcnRpZmFjdC1oYXNoZXMuanNvbiIsCiAgICAgICJzaGEyNTYiOiAiNTVmMjRiZGMzZDI4YTU1OTZmNGY4YTM2MjkyODIwMzU2ZGU1MGFhMmU5YzVjMmZiODEzOTdiZmUyODkxY2E0ZCIKICAgIH0sCiAgICAiYnVuZGxlX2Rzc2UiOiB7CiAgICAgICJwYXRoIjogIm1pcnJvci10aGluLXYxLmJ1bmRsZS5kc3NlLmpzb24iLAogICAgICAic2hhMjU2IjogbnVsbAogICAgfSwKICAgICJidW5kbGVfbWV0YSI6IHsKICAgICAgInBhdGgiOiAibWlycm9yLXRoaW4tdjEuYnVuZGxlLmpzb24iLAogICAgICAic2hhMjU2IjogbnVsbAogICAgfSwKICAgICJtYW5pZmVzdCI6IHsKICAgICAgInBhdGgiOiAibWlycm9yLXRoaW4tdjEubWFuaWZlc3QuanNvbiIsCiAgICAgICJzaGEyNTYiOiAiMWFmZmIwYjc5NmZmMDM3MTE3YjQ2YWExZjFkODA1NmE5YzgwNzU1ZTkyNWFmMDU4ZWE3MjEzMmJhMTU4YmVjZiIKICAgIH0sCiAgICAibWFuaWZlc3RfZHNzZSI6IHsKICAgICAgInBhdGgiOiAibWlycm9yLXRoaW4tdjEubWFuaWZlc3QuZHNzZS5qc29uIiwKICAgICAgInNoYTI1NiI6IG51bGwKICAgIH0sCiAgICAibWlycm9yX3BvbGljeSI6IHsKICAgICAgInBhdGgiOiAibWlycm9yLXBvbGljeS5qc29uIiwKICAgICAgInNoYTI1NiI6ICJkNzA1OWQ0YjllN2UyMDdmMjQyMDUyMGJmNzNjZjY5YjY0NGVlYzBlODY2ZjAzOWExZjdkMGRjMmIzYmMxMTkyIgogICAgfSwKICAgICJvY2lfaW5kZXgiOiB7CiAgICAgICJwYXRoIjogIm9jaS9pbmRleC5qc29uIiwKICAgICAgInNoYTI1NiI6ICI1ZGFmODAyNGYwZjNiMzdjMjA3NzQ5N2M1NGFjM2Q3YmRhNGFhZWQ1OWIzYzQ3YzYwNWM1MzU2NjJmN2E1M2E1IgogICAgfSwKICAgICJvZmZsaW5lX3BvbGljeSI6IHsKICAgICAgInBhdGgiOiAib2ZmbGluZS1raXQtcG9saWN5Lmpzb24iLAogICAgICAic2hhMjU2IjogImFlMjUxM2Y5NzY4ZjNmN2MwYjA5OTRiNTRmNTM5YjJhOTMzZTFlODUxYzI1YzI2YzhmZTQ2ZmQ5NjNkOTA1NzkiCiAgICB9LAogICAgInJla29yX3BvbGljeSI6IHsKICAgICAgInBhdGgiOiAicmVrb3ItcG9saWN5Lmpzb24iLAogICAgICAic2hhMjU2IjogIjY1MmRmMTU3NjI4ZGI3M2U5YWEwMTEwZTczOTBmODc3MzMxOWMyNDUzMGUwMDg3M2FmY2ZkZjk3MjY0NDcxN2UiCiAgICB9LAogICAgInRhcmJhbGwiOiB7CiAgICAgICJwYXRoIjogIm1pcnJvci10aGluLXYxLnRhci5neiIsCiAgICAgICJzaGEyNTYiOiAiZmIxY2UyNjM4OGExZjFhYjJlYjkwYWFlNmQ2M2FjMDVkZTMyNmZiYmQ5NDdmYmY3YTE3Yjk4MDIzMmM5ZmM3ZCIKICAgIH0sCiAgICAidGltZV9hbmNob3IiOiB7CiAgICAgICJwYXRoIjogInRpbWUtYW5jaG9yLmpzb24iLAogICAgICAic2hhMjU2IjogImMyN2EwZmIwZGZhOGE5NTU4YWFhYmY4MDExMDQwYWJjZDQxNzBjZjYyZTM2ZDE2YjViMTc2NzM2OGY3ODI4ZmYiCiAgICB9LAogICAgInRyYW5zcG9ydF9wbGFuIjogewogICAgICAicGF0aCI6ICJ0cmFuc3BvcnQtcGxhbi5qc29uIiwKICAgICAgInNoYTI1NiI6ICJkZjgyYTU2ZDliYWNiMDBhMTg4MmY1ZDZkOWY5YmE0NjliNjJiODliZDk0OTg5OWI3MDQ5ZTEyM2MxZTY1OTE0IgogICAgfQogIH0sCiAgImJ1bmRsZSI6ICJtaXJyb3ItdGhpbi12MSIsCiAgImNoYWluX29mX2N1c3RvZHkiOiBbCiAgICB7CiAgICAgICJzaGEyNTYiOiAiZGQxMWM2NzQ2MjlmZTk0YmYzN2FjOWEyOWQ3YWUzMjI0MWY2YTE3ODE1YmIyNzU1MzJkOWE3OGIzZDg1MTA0OSIsCiAgICAgICJzdGVwIjogImJ1aWxkIiwKICAgICAgInRvb2wiOiAibWFrZS10aGluLXYxLnNoIgogICAgfSwKICAgIHsKICAgICAgImtleV9wcmVzZW50IjogdHJ1ZSwKICAgICAgImtleWlkIjogImRiOTkyOGJhYmYzYWViODE3Y2NkY2QwZjZhNjY4OGY4Mzk1YjAwZDBlNDI5NjZlMzJlNzA2OTMxYjUzMDFmYzgiLAogICAgICAic3RlcCI6ICJzaWduIiwKICAgICAgInRvb2wiOiAic2lnbl90aGluX2J1bmRsZS5weSIKICAgIH0KICBdLAogICJjaGVja3BvaW50X2ZyZXNobmVzc19zZWNvbmRzIjogODY0MDAsCiAgImNodW5rX3NpemVfYnl0ZXMiOiA1MjQyODgwLAogICJjcmVhdGVkIjogIjIwMjUtMTItMDJUMTg6MDg6MzRaIiwKICAiZW52aXJvbm1lbnQiOiAibGFiIiwKICAiZ2FwcyI6IHsKICAgICJtcyI6IFsKICAgICAgIk1TMSBtaXJyb3Igc2NoZW1hIHZlcnNpb25lZCBpbiBtaXJyb3ItcG9saWN5Lmpzb24iLAogICAgICAiTVMyIERTU0UvVFVGIHJvdGF0aW9uIGRheXMgcmVjb3JkZWQiLAogICAgICAiTVMzIGRlbHRhIHNwZWMgaW5jbHVkZXMgdG9tYnN0b25lcyArIGJhc2UgaGFzaCIsCiAgICAgICJNUzQgdGltZS1hbmNob3IgZnJlc2huZXNzIGVuZm9yY2VkIiwKICAgICAgIk1TNSB0ZW5hbnQvZW52IHNjb3BpbmcgY2FwdHVyZWQiLAogICAgICAiTVM2IGRpc3RyaWJ1dGlvbiBpbnRlZ3JpdHkgcnVsZXMgZG9jdW1lbnRlZCIsCiAgICAgICJNUzcgY2h1bmtpbmcvc2l6ZSBydWxlcyByZWNvcmRlZCIsCiAgICAgICJNUzggdmVyaWZ5IHNjcmlwdCBwaW5uZWQiLAogICAgICAiTVM5IG1ldHJpY3MvYWxlcnRzIHJlcXVpcmVkIiwKICAgICAgIk1TMTAgc2VtdmVyL2NoYW5nZWxvZyBub3RlZCIKICAgIF0sCiAgICAib2siOiBbCiAgICAgICJPSzEga2V5IG1hbmlmZXN0ICsgUFEgY28tc2lnbiByZWNvcmRlZCBpbiBvZmZsaW5lLWtpdC1wb2xpY3kuanNvbiIsCiAgICAgICJPSzIgdG9vbCBoYXNoaW5nIGNhcHR1cmVkIGluIGJ1bmRsZV9tZXRhLnRvb2xpbmciLAogICAgICAiT0szIERTU0UgdG9wLWxldmVsIG1hbmlmZXN0IHBsYW5uZWQgdmlhIGJ1bmRsZS5kc3NlIiwKICAgICAgIk9LNCBjaGVja3BvaW50IGZyZXNobmVzcyBlbmZvcmNlZCB3aXRoIGNoZWNrcG9pbnRfZnJlc2huZXNzX3NlY29uZHMiLAogICAgICAiT0s1IGRldGVybWluaXN0aWMgcGFja2FnaW5nIGZsYWdzIHJlY29yZGVkIGluIG9mZmxpbmUta2l0LXBvbGljeS5qc29uIiwKICAgICAgIk9LNiBzY2FuL1ZFWC9wb2xpY3kvZ3JhcGggaGFzaGVzIGNhcHR1cmVkIGluIGFydGlmYWN0LWhhc2hlcy5qc29uIiwKICAgICAgIk9LNyB0aW1lIGFuY2hvciBidW5kbGVkIGFzIGxheWVycy90aW1lLWFuY2hvci5qc29uIiwKICAgICAgIk9LOCB0cmFuc3BvcnQgKyBjaHVua2luZyBkZWZpbmVkIGluIHRyYW5zcG9ydC1wbGFuLmpzb24iLAogICAgICAiT0s5IHRlbmFudC9lbnZpcm9ubWVudCBzY29waW5nIHJlY29yZGVkIGluIGJ1bmRsZSBtZXRhIiwKICAgICAgIk9LMTAgc2NyaXB0ZWQgdmVyaWZ5IHBhdGggaXMgc2NyaXB0cy9taXJyb3IvdmVyaWZ5X3RoaW5fYnVuZGxlLnB5IgogICAgXSwKICAgICJyayI6IFsKICAgICAgIlJLMSBlbmZvcmNlIGRzc2UvaGFzaGVkcmVrb3JkIHBvbGljeSBpbiByZWtvci1wb2xpY3kuanNvbiIsCiAgICAgICJSSzIgcGF5bG9hZCBzaXplIHByZWZsaWdodCByazJfcGF5bG9hZE1heEJ5dGVzIiwKICAgICAgIlJLMyByb3V0aW5nIHBvbGljeSBmb3IgcHVibGljL3ByaXZhdGUgcmVjb3JkZWQiLAogICAgICAiUks0IHNoYXJkLWF3YXJlIGNoZWNrcG9pbnRzIHBlci10ZW5hbnQtcGVyLWRheSIsCiAgICAgICJSSzUgaWRlbXBvdGVudCBzdWJtaXNzaW9uIGtleXMgZW5hYmxlZCIsCiAgICAgICJSSzYgU2lnc3RvcmUgYnVuZGxlIGluY2x1c2lvbiBmbGFnZ2VkIHRydWUiLAogICAgICAiUks3IGNoZWNrcG9pbnQgZnJlc2huZXNzIHNlY29uZHMgcmVjb3JkZWQiLAogICAgICAiUks4IFBRIGR1YWwtc2lnbiB0b2dnbGUgbWF0Y2hlcyBwcUR1YWxTaWduIiwKICAgICAgIlJLOSBlcnJvciB0YXhvbm9teSBlbnVtZXJhdGVkIiwKICAgICAgIlJLMTAgcG9saWN5L2dyYXBoIGFubm90YXRpb25zIHJlcXVpcmVkIgogICAgXQogIH0sCiAgInBxX2Nvc2lnbl9yZXF1aXJlZCI6IGZhbHNlLAogICJ0ZW5hbnQiOiAidGVuYW50LWRlbW8iLAogICJ0b29saW5nIjogewogICAgIm1ha2VfdGhpbl92MV9zaCI6ICJkZDExYzY3NDYyOWZlOTRiZjM3YWM5YTI5ZDdhZTMyMjQxZjZhMTc4MTViYjI3NTUzMmQ5YTc4YjNkODUxMDQ5IiwKICAgICJzaWduX3NjcmlwdCI6ICIzMDI2OGYzYjZkMTFhMTEwOGE4Y2I1YTVlYmM5NzIzYzM0YTY3Y2YxZTEyOTQ0YjEwMTRjYzc2OTY1NjE5YjczIiwKICAgICJ2ZXJpZnlfb2NpIjogIjA0YjZiMDQyNGE3MjVkMjA4MTI3NWU2NzgyMGM1ODBiNTMyNjQ2ZmQ2NDBlZTliZjYyYmM3NWJjNzU1NGViNzciLAogICAgInZlcmlmeV9zY3JpcHQiOiAiMDc5NGY3OTg1MWJkNzFjMGUwNzQyNWU2OTI4ZjAzODI4Njk1N2YzYmFiYzk1Y2E2NjY2MGFjYjZjNWQ4YzMxYiIKICB9LAogICJ2ZXJzaW9uIjogIjEuMC4wIgp9Cg", + "payloadType": "application/vnd.stellaops.mirror.bundle+json", + "signatures": [ + { + "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", + "sig": "WimfcZH0NtgBn9d3vaVA39f2tqIEqEJXpzPNt7c6Pf5wbHyYwHVCic9iRcvqMhGOzSmcPmQAckyrq6rm0WkWBA" + } + ] +} diff --git a/out/mirror/thin/mirror-thin-v1.bundle.json b/out/mirror/thin/mirror-thin-v1.bundle.json new file mode 100644 index 000000000..f9fe33d2b --- /dev/null +++ b/out/mirror/thin/mirror-thin-v1.bundle.json @@ -0,0 +1,117 @@ +{ + "artifacts": { + "artifact_hashes": { + "path": "artifact-hashes.json", + "sha256": "55f24bdc3d28a5596f4f8a36292820356de50aa2e9c5c2fb81397bfe2891ca4d" + }, + "bundle_dsse": { + "path": "mirror-thin-v1.bundle.dsse.json", + "sha256": null + }, + "bundle_meta": { + "path": "mirror-thin-v1.bundle.json", + "sha256": null + }, + "manifest": { + "path": "mirror-thin-v1.manifest.json", + "sha256": "1affb0b796ff037117b46aa1f1d8056a9c80755e925af058ea72132ba158becf" + }, + "manifest_dsse": { + "path": "mirror-thin-v1.manifest.dsse.json", + "sha256": null + }, + "mirror_policy": { + "path": "mirror-policy.json", + "sha256": "d7059d4b9e7e207f2420520bf73cf69b644eec0e866f039a1f7d0dc2b3bc1192" + }, + "oci_index": { + "path": "oci/index.json", + "sha256": "5daf8024f0f3b37c2077497c54ac3d7bda4aaed59b3c47c605c535662f7a53a5" + }, + "offline_policy": { + "path": "offline-kit-policy.json", + "sha256": "ae2513f9768f3f7c0b0994b54f539b2a933e1e851c25c26c8fe46fd963d90579" + }, + "rekor_policy": { + "path": "rekor-policy.json", + "sha256": "652df157628db73e9aa0110e7390f8773319c24530e00873afcfdf972644717e" + }, + "tarball": { + "path": "mirror-thin-v1.tar.gz", + "sha256": "fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d" + }, + "time_anchor": { + "path": "time-anchor.json", + "sha256": "c27a0fb0dfa8a9558aaabf8011040abcd4170cf62e36d16b5b1767368f7828ff" + }, + "transport_plan": { + "path": "transport-plan.json", + "sha256": "df82a56d9bacb00a1882f5d6d9f9ba469b62b89bd949899b7049e123c1e65914" + } + }, + "bundle": "mirror-thin-v1", + "chain_of_custody": [ + { + "sha256": "dd11c674629fe94bf37ac9a29d7ae32241f6a17815bb275532d9a78b3d851049", + "step": "build", + "tool": "make-thin-v1.sh" + }, + { + "key_present": true, + "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", + "step": "sign", + "tool": "sign_thin_bundle.py" + } + ], + "checkpoint_freshness_seconds": 86400, + "chunk_size_bytes": 5242880, + "created": "2025-12-02T18:08:34Z", + "environment": "lab", + "gaps": { + "ms": [ + "MS1 mirror schema versioned in mirror-policy.json", + "MS2 DSSE/TUF rotation days recorded", + "MS3 delta spec includes tombstones + base hash", + "MS4 time-anchor freshness enforced", + "MS5 tenant/env scoping captured", + "MS6 distribution integrity rules documented", + "MS7 chunking/size rules recorded", + "MS8 verify script pinned", + "MS9 metrics/alerts required", + "MS10 semver/changelog noted" + ], + "ok": [ + "OK1 key manifest + PQ co-sign recorded in offline-kit-policy.json", + "OK2 tool hashing captured in bundle_meta.tooling", + "OK3 DSSE top-level manifest planned via bundle.dsse", + "OK4 checkpoint freshness enforced with checkpoint_freshness_seconds", + "OK5 deterministic packaging flags recorded in offline-kit-policy.json", + "OK6 scan/VEX/policy/graph hashes captured in artifact-hashes.json", + "OK7 time anchor bundled as layers/time-anchor.json", + "OK8 transport + chunking defined in transport-plan.json", + "OK9 tenant/environment scoping recorded in bundle meta", + "OK10 scripted verify path is scripts/mirror/verify_thin_bundle.py" + ], + "rk": [ + "RK1 enforce dsse/hashedrekord policy in rekor-policy.json", + "RK2 payload size preflight rk2_payloadMaxBytes", + "RK3 routing policy for public/private recorded", + "RK4 shard-aware checkpoints per-tenant-per-day", + "RK5 idempotent submission keys enabled", + "RK6 Sigstore bundle inclusion flagged true", + "RK7 checkpoint freshness seconds recorded", + "RK8 PQ dual-sign toggle matches pqDualSign", + "RK9 error taxonomy enumerated", + "RK10 policy/graph annotations required" + ] + }, + "pq_cosign_required": false, + "tenant": "tenant-demo", + "tooling": { + "make_thin_v1_sh": "dd11c674629fe94bf37ac9a29d7ae32241f6a17815bb275532d9a78b3d851049", + "sign_script": "30268f3b6d11a1108a8cb5a5ebc9723c34a67cf1e12944b1014cc76965619b73", + "verify_oci": "04b6b0424a725d2081275e67820c580b532646fd640ee9bf62bc75bc7554eb77", + "verify_script": "0794f79851bd71c0e07425e6928f038286957f3babc95ca66660acb6c5d8c31b" + }, + "version": "1.0.0" +} diff --git a/out/mirror/thin/mirror-thin-v1.bundle.json.sha256 b/out/mirror/thin/mirror-thin-v1.bundle.json.sha256 new file mode 100644 index 000000000..e644cf894 --- /dev/null +++ b/out/mirror/thin/mirror-thin-v1.bundle.json.sha256 @@ -0,0 +1 @@ +a3b16f5d1b74ffdf9aedbbfe9282d368dc3dcf70676c8ac7e8cdd984162e7f90 mirror-thin-v1.bundle.json diff --git a/out/mirror/thin/mirror-thin-v1.manifest.dsse.json b/out/mirror/thin/mirror-thin-v1.manifest.dsse.json index 62ebdf1ef..bce502e40 100644 --- a/out/mirror/thin/mirror-thin-v1.manifest.dsse.json +++ b/out/mirror/thin/mirror-thin-v1.manifest.dsse.json @@ -1,10 +1,10 @@ { - "payload": "ewogICJjcmVhdGVkIjogIjIwMjUtMTEtMjNUMDA6MDA6MDBaIiwKICAiaW5kZXhlcyI6IFsKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6YjY0YzdlNWQ0NDA4YTEwMDMxMWVjOGZhYmM3NmI5ZTUyNTE2NWUyMWRmZmMzZjQ2NDFhZjc5YjlhYTQ0MzNjOSIsCiAgICAgICJuYW1lIjogIm9ic2VydmF0aW9ucy5pbmRleCIKICAgIH0KICBdLAogICJsYXllcnMiOiBbCiAgICB7CiAgICAgICJkaWdlc3QiOiAic2hhMjU2OmZkM2NlNTA0OTdjYmQyMDNkZjIyY2QyZmQxNDY0NmIxYWFjODU4ODRlZDE2MzIxNWE3OWM2MjA3MzAxMjQ1ZDYiLAogICAgICAicGF0aCI6ICJsYXllcnMvb2JzZXJ2YXRpb25zLm5kanNvbiIsCiAgICAgICJzaXplIjogMzEwCiAgICB9LAogICAgewogICAgICAiZGlnZXN0IjogInNoYTI1NjpjMjdhMGZiMGRmYThhOTU1OGFhYWJmODAxMTA0MGFiY2Q0MTcwY2Y2MmUzNmQxNmI1YjE3NjczNjhmNzgyOGZmIiwKICAgICAgInBhdGgiOiAibGF5ZXJzL3RpbWUtYW5jaG9yLmpzb24iLAogICAgICAic2l6ZSI6IDMyMgogICAgfQogIF0sCiAgInZlcnNpb24iOiAiMS4wLjAiCn0K", + "payload": "ewogICJjcmVhdGVkIjogIjIwMjUtMTItMDJUMTg6MDg6MzRaIiwKICAiaW5kZXhlcyI6IFsKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6YjY0YzdlNWQ0NDA4YTEwMDMxMWVjOGZhYmM3NmI5ZTUyNTE2NWUyMWRmZmMzZjQ2NDFhZjc5YjlhYTQ0MzNjOSIsCiAgICAgICJuYW1lIjogIm9ic2VydmF0aW9ucy5pbmRleCIKICAgIH0KICBdLAogICJsYXllcnMiOiBbCiAgICB7CiAgICAgICJkaWdlc3QiOiAic2hhMjU2OjU1ZjI0YmRjM2QyOGE1NTk2ZjRmOGEzNjI5MjgyMDM1NmRlNTBhYTJlOWM1YzJmYjgxMzk3YmZlMjg5MWNhNGQiLAogICAgICAicGF0aCI6ICJsYXllcnMvYXJ0aWZhY3QtaGFzaGVzLmpzb24iLAogICAgICAic2l6ZSI6IDU5MgogICAgfSwKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6ZDcwNTlkNGI5ZTdlMjA3ZjI0MjA1MjBiZjczY2Y2OWI2NDRlZWMwZTg2NmYwMzlhMWY3ZDBkYzJiM2JjMTE5MiIsCiAgICAgICJwYXRoIjogImxheWVycy9taXJyb3ItcG9saWN5Lmpzb24iLAogICAgICAic2l6ZSI6IDY2NQogICAgfSwKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6ZmQzY2U1MDQ5N2NiZDIwM2RmMjJjZDJmZDE0NjQ2YjFhYWM4NTg4NGVkMTYzMjE1YTc5YzYyMDczMDEyNDVkNiIsCiAgICAgICJwYXRoIjogImxheWVycy9vYnNlcnZhdGlvbnMubmRqc29uIiwKICAgICAgInNpemUiOiAzMTAKICAgIH0sCiAgICB7CiAgICAgICJkaWdlc3QiOiAic2hhMjU2OmFlMjUxM2Y5NzY4ZjNmN2MwYjA5OTRiNTRmNTM5YjJhOTMzZTFlODUxYzI1YzI2YzhmZTQ2ZmQ5NjNkOTA1NzkiLAogICAgICAicGF0aCI6ICJsYXllcnMvb2ZmbGluZS1raXQtcG9saWN5Lmpzb24iLAogICAgICAic2l6ZSI6IDU0NAogICAgfSwKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6NjUyZGYxNTc2MjhkYjczZTlhYTAxMTBlNzM5MGY4NzczMzE5YzI0NTMwZTAwODczYWZjZmRmOTcyNjQ0NzE3ZSIsCiAgICAgICJwYXRoIjogImxheWVycy9yZWtvci1wb2xpY3kuanNvbiIsCiAgICAgICJzaXplIjogNDY1CiAgICB9LAogICAgewogICAgICAiZGlnZXN0IjogInNoYTI1NjpjMjdhMGZiMGRmYThhOTU1OGFhYWJmODAxMTA0MGFiY2Q0MTcwY2Y2MmUzNmQxNmI1YjE3NjczNjhmNzgyOGZmIiwKICAgICAgInBhdGgiOiAibGF5ZXJzL3RpbWUtYW5jaG9yLmpzb24iLAogICAgICAic2l6ZSI6IDMyMgogICAgfSwKICAgIHsKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6ZGY4MmE1NmQ5YmFjYjAwYTE4ODJmNWQ2ZDlmOWJhNDY5YjYyYjg5YmQ5NDk4OTliNzA0OWUxMjNjMWU2NTkxNCIsCiAgICAgICJwYXRoIjogImxheWVycy90cmFuc3BvcnQtcGxhbi5qc29uIiwKICAgICAgInNpemUiOiA3NDEKICAgIH0KICBdLAogICJ2ZXJzaW9uIjogIjEuMC4wIgp9Cg", "payloadType": "application/vnd.stellaops.mirror.manifest+json", "signatures": [ { "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", - "sig": "EC7tbq5zlHqUfidvkT-Q1yfmiTJs9KUdpnvs9jCBJXsxzIyB1hzfdh-7FNPi3pFSrzV6cDh47cWvWmMR_ypgDw" + "sig": "f3XR6taW0E9gAkBEYPgxsWEI2cO28-1zA4XhcepzXm3FJ7Ii8ksfp_nFWH1m4JT4JRUK5tRcc8X4Bw_SSRRkDg" } ] } diff --git a/out/mirror/thin/mirror-thin-v1.manifest.json b/out/mirror/thin/mirror-thin-v1.manifest.json index 91e6fa9e2..5f932c5ef 100644 --- a/out/mirror/thin/mirror-thin-v1.manifest.json +++ b/out/mirror/thin/mirror-thin-v1.manifest.json @@ -1,5 +1,5 @@ { - "created": "2025-11-23T00:00:00Z", + "created": "2025-12-02T18:08:34Z", "indexes": [ { "digest": "sha256:b64c7e5d4408a100311ec8fabc76b9e525165e21dffc3f4641af79b9aa4433c9", @@ -7,15 +7,40 @@ } ], "layers": [ + { + "digest": "sha256:55f24bdc3d28a5596f4f8a36292820356de50aa2e9c5c2fb81397bfe2891ca4d", + "path": "layers/artifact-hashes.json", + "size": 592 + }, + { + "digest": "sha256:d7059d4b9e7e207f2420520bf73cf69b644eec0e866f039a1f7d0dc2b3bc1192", + "path": "layers/mirror-policy.json", + "size": 665 + }, { "digest": "sha256:fd3ce50497cbd203df22cd2fd14646b1aac85884ed163215a79c6207301245d6", "path": "layers/observations.ndjson", "size": 310 }, + { + "digest": "sha256:ae2513f9768f3f7c0b0994b54f539b2a933e1e851c25c26c8fe46fd963d90579", + "path": "layers/offline-kit-policy.json", + "size": 544 + }, + { + "digest": "sha256:652df157628db73e9aa0110e7390f8773319c24530e00873afcfdf972644717e", + "path": "layers/rekor-policy.json", + "size": 465 + }, { "digest": "sha256:c27a0fb0dfa8a9558aaabf8011040abcd4170cf62e36d16b5b1767368f7828ff", "path": "layers/time-anchor.json", "size": 322 + }, + { + "digest": "sha256:df82a56d9bacb00a1882f5d6d9f9ba469b62b89bd949899b7049e123c1e65914", + "path": "layers/transport-plan.json", + "size": 741 } ], "version": "1.0.0" diff --git a/out/mirror/thin/mirror-thin-v1.manifest.json.sha256 b/out/mirror/thin/mirror-thin-v1.manifest.json.sha256 index 3fe70512d..d8bcf7401 100644 --- a/out/mirror/thin/mirror-thin-v1.manifest.json.sha256 +++ b/out/mirror/thin/mirror-thin-v1.manifest.json.sha256 @@ -1 +1 @@ -b0e5d5af5b560d1b24cf44c2325e7f90d486857f347f34826b9f06aa217c5a6a mirror-thin-v1.manifest.json +1affb0b796ff037117b46aa1f1d8056a9c80755e925af058ea72132ba158becf mirror-thin-v1.manifest.json diff --git a/out/mirror/thin/mirror-thin-v1.tar.gz b/out/mirror/thin/mirror-thin-v1.tar.gz index 843eb9db3..7c1bc5f96 100644 Binary files a/out/mirror/thin/mirror-thin-v1.tar.gz and b/out/mirror/thin/mirror-thin-v1.tar.gz differ diff --git a/out/mirror/thin/mirror-thin-v1.tar.gz.sha256 b/out/mirror/thin/mirror-thin-v1.tar.gz.sha256 index 6e60545fd..6e4ee4ae0 100644 --- a/out/mirror/thin/mirror-thin-v1.tar.gz.sha256 +++ b/out/mirror/thin/mirror-thin-v1.tar.gz.sha256 @@ -1 +1 @@ -1ef17d14c09e74703b88753d6c561d8c8a8809fe8e05972257adadfb91b71723 mirror-thin-v1.tar.gz +fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d mirror-thin-v1.tar.gz diff --git a/out/mirror/thin/oci/blobs/sha256/fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d b/out/mirror/thin/oci/blobs/sha256/fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d new file mode 100644 index 000000000..7c1bc5f96 Binary files /dev/null and b/out/mirror/thin/oci/blobs/sha256/fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d differ diff --git a/out/mirror/thin/oci/index.json b/out/mirror/thin/oci/index.json index 1f6eec31f..3fb65caf0 100644 --- a/out/mirror/thin/oci/index.json +++ b/out/mirror/thin/oci/index.json @@ -3,8 +3,8 @@ "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", - "digest": "sha256:f6bd80fe9d346e7306c69832e29180346454005a0751c77ae2ebb7332be94642", - "size": 485, + "digest": "sha256:0074121d4adef7dc8181607645af330a475608b0d52909e0efd421508f14437d", + "size": 486, "annotations": {"org.opencontainers.image.ref.name": "mirror-thin-v1"} } ] diff --git a/out/mirror/thin/oci/manifest.json b/out/mirror/thin/oci/manifest.json index f31df21da..73be456e7 100644 --- a/out/mirror/thin/oci/manifest.json +++ b/out/mirror/thin/oci/manifest.json @@ -8,8 +8,8 @@ "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", - "size": 830, - "digest": "sha256:1ef17d14c09e74703b88753d6c561d8c8a8809fe8e05972257adadfb91b71723", + "size": 2468, + "digest": "sha256:fb1ce26388a1f1ab2eb90aae6d63ac05de326fbbd947fbf7a17b980232c9fc7d", "annotations": {"org.stellaops.bundle.type": "mirror-thin-v1"} } ] diff --git a/out/mirror/thin/stage-v1/layers/artifact-hashes.json b/out/mirror/thin/stage-v1/layers/artifact-hashes.json new file mode 100644 index 000000000..357bb3403 --- /dev/null +++ b/out/mirror/thin/stage-v1/layers/artifact-hashes.json @@ -0,0 +1,20 @@ +{ + "artifacts": { + "graph": { + "digest": "sha256:652df157628db73e9aa0110e7390f8773319c24530e00873afcfdf972644717e", + "id": "graph-fixture-1" + }, + "policy": { + "digest": "sha256:d7059d4b9e7e207f2420520bf73cf69b644eec0e866f039a1f7d0dc2b3bc1192", + "id": "policy-fixture-1" + }, + "scan": { + "digest": "sha256:fd3ce50497cbd203df22cd2fd14646b1aac85884ed163215a79c6207301245d6", + "id": "scan-fixture-1" + }, + "vex": { + "digest": "sha256:fd3ce50497cbd203df22cd2fd14646b1aac85884ed163215a79c6207301245d6", + "id": "vex-fixture-1" + } + } +} diff --git a/out/mirror/thin/stage-v1/layers/mirror-policy.json b/out/mirror/thin/stage-v1/layers/mirror-policy.json new file mode 100644 index 000000000..24be9f918 --- /dev/null +++ b/out/mirror/thin/stage-v1/layers/mirror-policy.json @@ -0,0 +1,15 @@ +{ + "schemaVersion": "mirror-thin-v1", + "semver": "1.0.0", + "dsseTufRotationDays": 30, + "pqDualSign": false, + "delta": {"tombstones": true, "baseHashRequired": true}, + "timeAnchorFreshnessSeconds": 86400, + "tenantScope": "tenant-demo", + "environment": "lab", + "distributionIntegrity": {"http": "sha256+dsse", "oci": "tuf+dsse", "object": "checksum+length"}, + "chunking": {"sizeBytes": 5242880, "maxChunks": 128}, + "verifyScript": "scripts/mirror/verify_thin_bundle.py", + "metrics": {"build": "required", "import": "required", "verify": "required"}, + "changelog": {"current": "mirror-thin-v1", "notes": "Adds offline/rekor policy coverage (MS1-MS10)"} +} diff --git a/out/mirror/thin/stage-v1/layers/offline-kit-policy.json b/out/mirror/thin/stage-v1/layers/offline-kit-policy.json new file mode 100644 index 000000000..ac334679b --- /dev/null +++ b/out/mirror/thin/stage-v1/layers/offline-kit-policy.json @@ -0,0 +1,14 @@ +{ + "okVersion": "1.0.0", + "keyManifest": {"rotationDays": 90, "pqCosignAllowed": false}, + "toolHashing": true, + "topLevelDsse": true, + "checkpointFreshnessSeconds": 86400, + "deterministicFlags": ["tar --sort=name --owner=0 --group=0 --numeric-owner --mtime=1970-01-01", "gzip -n"], + "contentHashes": "layers/artifact-hashes.json", + "timeAnchorPath": "layers/time-anchor.json", + "transportPlan": "layers/transport-plan.json", + "tenant": "tenant-demo", + "environment": "lab", + "verifyScript": "scripts/mirror/verify_thin_bundle.py" +} diff --git a/out/mirror/thin/stage-v1/layers/rekor-policy.json b/out/mirror/thin/stage-v1/layers/rekor-policy.json new file mode 100644 index 000000000..83f777b16 --- /dev/null +++ b/out/mirror/thin/stage-v1/layers/rekor-policy.json @@ -0,0 +1,12 @@ +{ + "rk1_enforceDsse": true, + "rk2_payloadMaxBytes": 1048576, + "rk3_routing": {"public": "hashedrekord", "private": "hashedrekord"}, + "rk4_shardCheckpoint": "per-tenant-per-day", + "rk5_idempotentKeys": true, + "rk6_sigstoreBundleIncluded": true, + "rk7_checkpointFreshnessSeconds": 86400, + "rk8_pqDualSign": false, + "rk9_errorTaxonomy": ["quota", "payload-too-large", "invalid-signature", "stale-checkpoint"], + "rk10_annotations": ["policy", "graph-edge"] +} diff --git a/out/mirror/thin/stage-v1/layers/transport-plan.json b/out/mirror/thin/stage-v1/layers/transport-plan.json new file mode 100644 index 000000000..3c7108178 --- /dev/null +++ b/out/mirror/thin/stage-v1/layers/transport-plan.json @@ -0,0 +1,11 @@ +{ + "chunkSizeBytes": 5242880, + "compression": "gzip", + "checkpointFreshnessSeconds": 86400, + "chainOfCustody": [ + {"step": "build", "actor": "make-thin-v1.sh", "evidence": "sha256:dd11c674629fe94bf37ac9a29d7ae32241f6a17815bb275532d9a78b3d851049", "negativePaths": ["missing-layer", "non-deterministic-tar"]}, + {"step": "sign", "actor": "sign_thin_bundle.py", "expectedEnvelope": "mirror-thin-v1.manifest.dsse.json", "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", "toolDigest": "sha256:30268f3b6d11a1108a8cb5a5ebc9723c34a67cf1e12944b1014cc76965619b73"} + ], + "chunking": {"maxChunks": 128, "strategy": "deterministic-size"}, + "ingest": {"expectedLatencySeconds": 120, "retryPolicy": "exponential"} +} diff --git a/out/mirror/thin/stage-v1/manifest.json b/out/mirror/thin/stage-v1/manifest.json index 91e6fa9e2..5f932c5ef 100644 --- a/out/mirror/thin/stage-v1/manifest.json +++ b/out/mirror/thin/stage-v1/manifest.json @@ -1,5 +1,5 @@ { - "created": "2025-11-23T00:00:00Z", + "created": "2025-12-02T18:08:34Z", "indexes": [ { "digest": "sha256:b64c7e5d4408a100311ec8fabc76b9e525165e21dffc3f4641af79b9aa4433c9", @@ -7,15 +7,40 @@ } ], "layers": [ + { + "digest": "sha256:55f24bdc3d28a5596f4f8a36292820356de50aa2e9c5c2fb81397bfe2891ca4d", + "path": "layers/artifact-hashes.json", + "size": 592 + }, + { + "digest": "sha256:d7059d4b9e7e207f2420520bf73cf69b644eec0e866f039a1f7d0dc2b3bc1192", + "path": "layers/mirror-policy.json", + "size": 665 + }, { "digest": "sha256:fd3ce50497cbd203df22cd2fd14646b1aac85884ed163215a79c6207301245d6", "path": "layers/observations.ndjson", "size": 310 }, + { + "digest": "sha256:ae2513f9768f3f7c0b0994b54f539b2a933e1e851c25c26c8fe46fd963d90579", + "path": "layers/offline-kit-policy.json", + "size": 544 + }, + { + "digest": "sha256:652df157628db73e9aa0110e7390f8773319c24530e00873afcfdf972644717e", + "path": "layers/rekor-policy.json", + "size": 465 + }, { "digest": "sha256:c27a0fb0dfa8a9558aaabf8011040abcd4170cf62e36d16b5b1767368f7828ff", "path": "layers/time-anchor.json", "size": 322 + }, + { + "digest": "sha256:df82a56d9bacb00a1882f5d6d9f9ba469b62b89bd949899b7049e123c1e65914", + "path": "layers/transport-plan.json", + "size": 741 } ], "version": "1.0.0" diff --git a/out/mirror/thin/tuf/root.json b/out/mirror/thin/tuf/root.json index 66b9ee728..548e791a3 100644 --- a/out/mirror/thin/tuf/root.json +++ b/out/mirror/thin/tuf/root.json @@ -23,7 +23,7 @@ "signatures": [ { "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", - "sig": "ZUXDqV5hn0cuZlOOEUZdpD474mc0bkJu4-LyBPNYwU3YkZufT2eXKM-QHksF4JoXgywbY9QD8qhnsEh05xoKBg" + "sig": "b9UQWxXZnpsltfVLch4KVKWitgd6ZHTOPvUp0w-e5Gbm8MY6ZBaM-JLP-lwLuiJQMgbhuOlzDVzLbgQoYsbzBw" } ], "spec_version": "1.0.31", diff --git a/out/mirror/thin/tuf/snapshot.json b/out/mirror/thin/tuf/snapshot.json index c057a4b1c..f41b92977 100644 --- a/out/mirror/thin/tuf/snapshot.json +++ b/out/mirror/thin/tuf/snapshot.json @@ -5,7 +5,7 @@ "signatures": [ { "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", - "sig": "Z2FtwGRtVhQNvNZUxceUb3Ygj5KNqJGTOFIq8CxltBvMfmaAavWmMST0shir7p-7LI3-kBUMdPOKYlGxFip3AQ" + "sig": "3zzhK_zR4cqN5GQ-WvsDE93He22enjx2oy9WdSxox6hw4rVMY-QhPnagMSRQKOxWVVPgPWNZOsJR8_LOi0H6Cw" } ], "spec_version": "1.0.31", diff --git a/out/mirror/thin/tuf/targets.json b/out/mirror/thin/tuf/targets.json index 00f61c5f7..94e591ac5 100644 --- a/out/mirror/thin/tuf/targets.json +++ b/out/mirror/thin/tuf/targets.json @@ -4,7 +4,7 @@ "signatures": [ { "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", - "sig": "SIKtu5qz3FYNQxittPQwwWUzQLRg9D6KpO3OKpxtZzrbD2S5corjRZg-JNymPzFoEbrm8i5b_p7sh6H44At-CQ" + "sig": "HkXgkY5l9ACl1nNZ7Ll-hnVC_8Zo1QSWOb7Q74THnlYlDdpg_d-gnruFeOrIxXix18IGCICqrfKfnERoR-8EAw" } ], "spec_version": "1.0.31", diff --git a/out/mirror/thin/tuf/timestamp.json b/out/mirror/thin/tuf/timestamp.json index a9017014f..7c9231d6c 100644 --- a/out/mirror/thin/tuf/timestamp.json +++ b/out/mirror/thin/tuf/timestamp.json @@ -5,7 +5,7 @@ "signatures": [ { "keyid": "db9928babf3aeb817ccdcd0f6a6688f8395b00d0e42966e32e706931b5301fc8", - "sig": "C_4pXTUzKaVEZ0Dwtn2FlXxOsxcht8nF_vdWwVOMsYqwrqYriZgd4x_r2lq_RnI5QYxagEHGnEjD-6ztEeRMCg" + "sig": "UKplo5ExWrbnIpxo31NjgDEW9xGVb_ypesrqjnpOornojmOUkZjN1rGmyHmhJGam6RoHAboX_KNZJUwIe-K4Dw" } ], "spec_version": "1.0.31", diff --git a/scripts/mirror/README.md b/scripts/mirror/README.md index 85402f79e..73cc0676f 100644 --- a/scripts/mirror/README.md +++ b/scripts/mirror/README.md @@ -1,9 +1,9 @@ # Mirror signing helpers -- `make-thin-v1.sh`: builds thin bundle v1, computes checksums, optional DSSE+TUF signing when `SIGN_KEY` is set, and runs verifier. -- `sign_thin_bundle.py`: signs manifest (DSSE) and root/targets/snapshot/timestamp JSON using an Ed25519 PEM key. -- `verify_thin_bundle.py`: checks SHA256 sidecars, manifest schema, tar determinism, and manifest/index digests. -- `ci-sign.sh`: CI wrapper. Set `MIRROR_SIGN_KEY_B64` (base64-encoded Ed25519 PEM) and run; it builds, signs, and verifies in one step. +- `make-thin-v1.sh`: builds thin bundle v1, computes checksums, emits bundle meta (offline/rekor/mirror gaps), optional DSSE+TUF signing when `SIGN_KEY` is set, and runs verifier. +- `sign_thin_bundle.py`: signs manifest (DSSE), bundle meta (DSSE), and root/targets/snapshot/timestamp JSON using an Ed25519 PEM key. +- `verify_thin_bundle.py`: checks SHA256 sidecars, manifest schema, tar determinism, required layers, optional bundle meta and DSSE signatures; accepts `--bundle-meta`, `--pubkey`, `--tenant`, `--environment`. +- `ci-sign.sh`: CI wrapper. Set `MIRROR_SIGN_KEY_B64` (base64-encoded Ed25519 PEM) and run; it builds, signs, and verifies in one step, emitting `milestone.json` with manifest/tar/bundle hashes. - `verify_oci_layout.py`: validates OCI layout/index/manifest and blob digests when `OCI=1` is used. Artifacts live under `out/mirror/thin/`. diff --git a/scripts/mirror/__pycache__/sign_thin_bundle.cpython-312.pyc b/scripts/mirror/__pycache__/sign_thin_bundle.cpython-312.pyc new file mode 100644 index 000000000..a7c31a8bd Binary files /dev/null and b/scripts/mirror/__pycache__/sign_thin_bundle.cpython-312.pyc differ diff --git a/scripts/mirror/__pycache__/verify_thin_bundle.cpython-312.pyc b/scripts/mirror/__pycache__/verify_thin_bundle.cpython-312.pyc new file mode 100644 index 000000000..63b5930f5 Binary files /dev/null and b/scripts/mirror/__pycache__/verify_thin_bundle.cpython-312.pyc differ diff --git a/scripts/mirror/ci-sign.sh b/scripts/mirror/ci-sign.sh index 4d38c937e..ff576d7b8 100644 --- a/scripts/mirror/ci-sign.sh +++ b/scripts/mirror/ci-sign.sh @@ -23,12 +23,23 @@ chmod 600 "$KEYFILE" openssl pkey -in "$KEYFILE" -pubout -out "$KEYDIR/ci-ed25519.pub" >/dev/null 2>&1 STAGE=${STAGE:-$ROOT/out/mirror/thin/stage-v1} CREATED=${CREATED:-$(date -u +%Y-%m-%dT%H:%M:%SZ)} -SIGN_KEY="$KEYFILE" STAGE="$STAGE" CREATED="$CREATED" "$ROOT/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh" +TENANT_SCOPE=${TENANT_SCOPE:-tenant-demo} +ENV_SCOPE=${ENV_SCOPE:-lab} +CHUNK_SIZE=${CHUNK_SIZE:-5242880} +CHECKPOINT_FRESHNESS=${CHECKPOINT_FRESHNESS:-86400} +OCI=${OCI:-1} +SIGN_KEY="$KEYFILE" STAGE="$STAGE" CREATED="$CREATED" TENANT_SCOPE="$TENANT_SCOPE" ENV_SCOPE="$ENV_SCOPE" CHUNK_SIZE="$CHUNK_SIZE" CHECKPOINT_FRESHNESS="$CHECKPOINT_FRESHNESS" OCI="$OCI" "$ROOT/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh" # Emit milestone summary with hashes for downstream consumers MANIFEST_PATH="$ROOT/out/mirror/thin/mirror-thin-v1.manifest.json" TAR_PATH="$ROOT/out/mirror/thin/mirror-thin-v1.tar.gz" DSSE_PATH="$ROOT/out/mirror/thin/mirror-thin-v1.manifest.dsse.json" +BUNDLE_PATH="$ROOT/out/mirror/thin/mirror-thin-v1.bundle.json" +BUNDLE_DSSE_PATH="$ROOT/out/mirror/thin/mirror-thin-v1.bundle.dsse.json" +TRANSPORT_PATH="$ROOT/out/mirror/thin/stage-v1/layers/transport-plan.json" +REKOR_POLICY_PATH="$ROOT/out/mirror/thin/stage-v1/layers/rekor-policy.json" +MIRROR_POLICY_PATH="$ROOT/out/mirror/thin/stage-v1/layers/mirror-policy.json" +OFFLINE_POLICY_PATH="$ROOT/out/mirror/thin/stage-v1/layers/offline-kit-policy.json" SUMMARY_PATH="$ROOT/out/mirror/thin/milestone.json" sha256() { @@ -41,7 +52,15 @@ cat > "$SUMMARY_PATH" < {dsse_path}") + extra = f", bundle DSSE -> {bundle_dsse_path}" if args.bundle else "" + print(f"Signed DSSE + TUF using keyid {keyid}; DSSE -> {dsse_path}{extra}") if __name__ == "__main__": main() diff --git a/scripts/mirror/verify_thin_bundle.py b/scripts/mirror/verify_thin_bundle.py index 4eba4b91f..5ec6bd2e2 100644 --- a/scripts/mirror/verify_thin_bundle.py +++ b/scripts/mirror/verify_thin_bundle.py @@ -1,20 +1,59 @@ #!/usr/bin/env python3 """ -Simple verifier for mirror-thin-v1 artefacts. +Verifier for mirror-thin-v1 artefacts and bundle meta. + Checks: -1) SHA256 of manifest and tarball matches provided .sha256 files. -2) Manifest schema has required fields. -3) Tarball contains manifest.json, layers/, indexes/ with deterministic tar headers (mtime=0, uid/gid=0, sorted paths). -4) Tar content digests match manifest entries. +1) SHA256 of manifest/tarball (and optional bundle meta) matches sidecars. +2) Manifest schema contains required fields and required layer files exist. +3) Tarball headers deterministic (sorted paths, uid/gid=0, mtime=0). +4) Tar contents match manifest digests. +5) Optional: verify DSSE signatures for manifest/bundle when a public key is provided. +6) Optional: validate bundle meta (tenant/env scope, policy hashes, gap coverage counts). Usage: - python scripts/mirror/verify_thin_bundle.py out/mirror/thin/mirror-thin-v1.manifest.json out/mirror/thin/mirror-thin-v1.tar.gz + python scripts/mirror/verify_thin_bundle.py \ + out/mirror/thin/mirror-thin-v1.manifest.json \ + out/mirror/thin/mirror-thin-v1.tar.gz \ + --bundle-meta out/mirror/thin/mirror-thin-v1.bundle.json \ + --pubkey out/mirror/thin/tuf/keys/ci-ed25519.pub \ + --tenant tenant-demo --environment lab Exit code 0 on success; non-zero on any check failure. """ -import json, tarfile, hashlib, sys, pathlib +import argparse +import base64 +import hashlib +import json +import pathlib +import sys +import tarfile +from typing import Optional + +try: + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey + + CRYPTO_AVAILABLE = True +except ImportError: # pragma: no cover - surfaced as runtime guidance + CRYPTO_AVAILABLE = False REQUIRED_FIELDS = ["version", "created", "layers", "indexes"] +REQUIRED_LAYER_FILES = { + "layers/observations.ndjson", + "layers/time-anchor.json", + "layers/transport-plan.json", + "layers/rekor-policy.json", + "layers/mirror-policy.json", + "layers/offline-kit-policy.json", + "layers/artifact-hashes.json", + "indexes/observations.index", +} + + +def _b64url_decode(data: str) -> bytes: + padding = "=" * (-len(data) % 4) + return base64.urlsafe_b64decode(data + padding) + def sha256_file(path: pathlib.Path) -> str: h = hashlib.sha256() @@ -23,20 +62,24 @@ def sha256_file(path: pathlib.Path) -> str: h.update(chunk) return h.hexdigest() + def load_sha256_sidecar(path: pathlib.Path) -> str: sidecar = path.with_suffix(path.suffix + ".sha256") if not sidecar.exists(): raise SystemExit(f"missing sidecar {sidecar}") return sidecar.read_text().strip().split()[0] + def check_schema(manifest: dict): missing = [f for f in REQUIRED_FIELDS if f not in manifest] if missing: raise SystemExit(f"manifest missing fields: {missing}") + def normalize(name: str) -> str: return name[2:] if name.startswith("./") else name + def check_tar_determinism(tar_path: pathlib.Path): with tarfile.open(tar_path, "r:gz") as tf: names = [normalize(n) for n in tf.getnames()] @@ -48,13 +91,21 @@ def check_tar_determinism(tar_path: pathlib.Path): if m.mtime != 0: raise SystemExit(f"tar header mtime not zero for {m.name}") + +def check_required_layers(tar_path: pathlib.Path): + with tarfile.open(tar_path, "r:gz") as tf: + names = {normalize(n) for n in tf.getnames()} + for required in REQUIRED_LAYER_FILES: + if required not in names: + raise SystemExit(f"required file missing from bundle: {required}") + + def check_content_hashes(manifest: dict, tar_path: pathlib.Path): with tarfile.open(tar_path, "r:gz") as tf: def get(name: str): try: return tf.getmember(name) except KeyError: - # retry with leading ./ return tf.getmember(f"./{name}") for layer in manifest.get("layers", []): name = layer["path"] @@ -74,12 +125,96 @@ def check_content_hashes(manifest: dict, tar_path: pathlib.Path): raise SystemExit(f"index digest mismatch {name}: {digest}") +def load_pubkey(path: pathlib.Path) -> Ed25519PublicKey: + if not CRYPTO_AVAILABLE: + raise SystemExit("cryptography is required for DSSE verification; install before using --pubkey") + return serialization.load_pem_public_key(path.read_bytes()) + + +def verify_dsse(dsse_path: pathlib.Path, pubkey_path: pathlib.Path, expected_payload: pathlib.Path, expected_type: str): + dsse_obj = json.loads(dsse_path.read_text()) + if dsse_obj.get("payloadType") != expected_type: + raise SystemExit(f"DSSE payloadType mismatch for {dsse_path}") + payload = _b64url_decode(dsse_obj.get("payload", "")) + if payload != expected_payload.read_bytes(): + raise SystemExit(f"DSSE payload mismatch for {dsse_path}") + sigs = dsse_obj.get("signatures") or [] + if not sigs: + raise SystemExit(f"DSSE missing signatures: {dsse_path}") + pub = load_pubkey(pubkey_path) + try: + pub.verify(_b64url_decode(sigs[0]["sig"]), payload) + except Exception as exc: # pragma: no cover - cryptography raises InvalidSignature + raise SystemExit(f"DSSE signature verification failed for {dsse_path}: {exc}") + + +def check_bundle_meta(meta_path: pathlib.Path, manifest_path: pathlib.Path, tar_path: pathlib.Path, tenant: Optional[str], environment: Optional[str]): + meta = json.loads(meta_path.read_text()) + for field in ["bundle", "version", "artifacts", "gaps", "tooling"]: + if field not in meta: + raise SystemExit(f"bundle meta missing field {field}") + if tenant and meta.get("tenant") != tenant: + raise SystemExit(f"bundle tenant mismatch: {meta.get('tenant')} != {tenant}") + if environment and meta.get("environment") != environment: + raise SystemExit(f"bundle environment mismatch: {meta.get('environment')} != {environment}") + + artifacts = meta["artifacts"] + + def expect(name: str, path: pathlib.Path): + recorded = artifacts.get(name) + if not recorded: + raise SystemExit(f"bundle meta missing artifact entry: {name}") + expected = recorded.get("sha256") + if expected and expected != sha256_file(path): + raise SystemExit(f"bundle meta digest mismatch for {name}") + + expect("manifest", manifest_path) + expect("tarball", tar_path) + for extra in ["time_anchor", "transport_plan", "rekor_policy", "mirror_policy", "offline_policy", "artifact_hashes"]: + rec = artifacts.get(extra) + if not rec: + raise SystemExit(f"bundle meta missing artifact entry: {extra}") + if not rec.get("path"): + raise SystemExit(f"bundle meta missing path for {extra}") + + for group, expected_count in [("ok", 10), ("rk", 10), ("ms", 10)]: + if len(meta.get("gaps", {}).get(group, [])) != expected_count: + raise SystemExit(f"bundle meta gaps.{group} expected {expected_count} entries") + + root_guess = manifest_path.parents[3] if len(manifest_path.parents) > 3 else manifest_path.parents[-1] + tool_expectations = { + 'make_thin_v1_sh': root_guess / 'src' / 'Mirror' / 'StellaOps.Mirror.Creator' / 'make-thin-v1.sh', + 'sign_script': root_guess / 'scripts' / 'mirror' / 'sign_thin_bundle.py', + 'verify_script': root_guess / 'scripts' / 'mirror' / 'verify_thin_bundle.py', + 'verify_oci': root_guess / 'scripts' / 'mirror' / 'verify_oci_layout.py' + } + for key, path in tool_expectations.items(): + recorded = meta['tooling'].get(key) + if not recorded: + raise SystemExit(f"tool hash missing for {key}") + actual = sha256_file(path) + if recorded != actual: + raise SystemExit(f"tool hash mismatch for {key}") + + if meta.get("checkpoint_freshness_seconds", 0) <= 0: + raise SystemExit("checkpoint_freshness_seconds must be positive") + + def main(): - if len(sys.argv) != 3: - print(__doc__) - sys.exit(2) - manifest_path = pathlib.Path(sys.argv[1]) - tar_path = pathlib.Path(sys.argv[2]) + parser = argparse.ArgumentParser() + parser.add_argument("manifest", type=pathlib.Path) + parser.add_argument("tar", type=pathlib.Path) + parser.add_argument("--bundle-meta", type=pathlib.Path) + parser.add_argument("--pubkey", type=pathlib.Path) + parser.add_argument("--tenant", type=str) + parser.add_argument("--environment", type=str) + args = parser.parse_args() + + manifest_path = args.manifest + tar_path = args.tar + bundle_meta = args.bundle_meta + bundle_dsse = bundle_meta.with_suffix(".dsse.json") if bundle_meta else None + manifest_dsse = manifest_path.with_suffix(".dsse.json") man_expected = load_sha256_sidecar(manifest_path) tar_expected = load_sha256_sidecar(tar_path) @@ -91,8 +226,26 @@ def main(): manifest = json.loads(manifest_path.read_text()) check_schema(manifest) check_tar_determinism(tar_path) + check_required_layers(tar_path) check_content_hashes(manifest, tar_path) + + if bundle_meta: + if not bundle_meta.exists(): + raise SystemExit(f"bundle meta missing: {bundle_meta}") + meta_expected = load_sha256_sidecar(bundle_meta) + if sha256_file(bundle_meta) != meta_expected: + raise SystemExit("bundle meta sha256 mismatch") + check_bundle_meta(bundle_meta, manifest_path, tar_path, args.tenant, args.environment) + + if args.pubkey: + pubkey = args.pubkey + if manifest_dsse.exists(): + verify_dsse(manifest_dsse, pubkey, manifest_path, "application/vnd.stellaops.mirror.manifest+json") + if bundle_dsse and bundle_dsse.exists(): + verify_dsse(bundle_dsse, pubkey, bundle_meta, "application/vnd.stellaops.mirror.bundle+json") + print("OK: mirror-thin bundle verified") + if __name__ == "__main__": main() diff --git a/src/Bench/StellaOps.Bench/Graph/README.md b/src/Bench/StellaOps.Bench/Graph/README.md index c1f13af8e..0ec134c93 100644 --- a/src/Bench/StellaOps.Bench/Graph/README.md +++ b/src/Bench/StellaOps.Bench/Graph/README.md @@ -5,18 +5,21 @@ Purpose: measure basic graph load/adjacency build and shallow path exploration o ## Fixtures - Use interim synthetic fixtures under `samples/graph/interim/graph-50k` or `graph-100k`. - Each fixture includes `nodes.ndjson`, `edges.ndjson`, and `manifest.json` with hashes/counts. +- Optional overlay: drop `overlay.ndjson` next to the fixture (or set `overlay.path` in `manifest.json`) to apply extra edges/layers; hashes are captured in results. ## Usage ```bash python graph_bench.py \ --fixture ../../../samples/graph/interim/graph-50k \ --output results/graph-50k.json \ - --samples 100 + --samples 100 \ + --overlay ../../../samples/graph/interim/graph-50k/overlay.ndjson # optional ``` Outputs a JSON summary with: - `nodes`, `edges` - `build_ms` — time to build adjacency (ms) +- `overlay_ms` — time to apply overlay (0 when absent), plus counts and SHA under `overlay.*` - `bfs_ms` — total time for 3-depth BFS over sampled nodes - `avg_reach_3`, `max_reach_3` — nodes reached within depth 3 - `manifest` — copied from fixture for traceability diff --git a/src/Bench/StellaOps.Bench/Graph/__pycache__/graph_bench.cpython-312.pyc b/src/Bench/StellaOps.Bench/Graph/__pycache__/graph_bench.cpython-312.pyc new file mode 100644 index 000000000..50335a607 Binary files /dev/null and b/src/Bench/StellaOps.Bench/Graph/__pycache__/graph_bench.cpython-312.pyc differ diff --git a/src/Bench/StellaOps.Bench/Graph/graph_bench.py b/src/Bench/StellaOps.Bench/Graph/graph_bench.py index c24fa674d..efc9a9976 100644 --- a/src/Bench/StellaOps.Bench/Graph/graph_bench.py +++ b/src/Bench/StellaOps.Bench/Graph/graph_bench.py @@ -9,10 +9,11 @@ no network, and fixed seeds for reproducibility. from __future__ import annotations import argparse +import hashlib import json import time from pathlib import Path -from typing import Dict, List, Tuple +from typing import Dict, List, Optional, Tuple def load_ndjson(path: Path): @@ -42,6 +43,52 @@ def build_graph(nodes_path: Path, edges_path: Path) -> Tuple[Dict[str, List[str] return adjacency, edge_count +def _sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(8192), b""): + h.update(chunk) + return h.hexdigest() + + +def apply_overlay(adjacency: Dict[str, List[str]], overlay_path: Path) -> Tuple[int, int]: + """ + Apply overlay edges to the adjacency map. + + Overlay file format (NDJSON): {"source": "nodeA", "target": "nodeB"} + Unknown keys are ignored. New nodes are added with empty adjacency to keep + BFS deterministic. Duplicate edges are de-duplicated. + """ + + if not overlay_path.exists(): + return 0, 0 + + added_edges = 0 + introduced_nodes = set() + for record in load_ndjson(overlay_path): + source = record.get("source") or record.get("from") + target = record.get("target") or record.get("to") + if not source or not target: + continue + + if source not in adjacency: + adjacency[source] = [] + introduced_nodes.add(source) + if target not in adjacency: + adjacency[target] = [] + introduced_nodes.add(target) + + if target not in adjacency[source]: + adjacency[source].append(target) + added_edges += 1 + + # keep neighbor ordering deterministic + for v in adjacency.values(): + v.sort() + + return added_edges, len(introduced_nodes) + + def bfs_limited(adjacency: Dict[str, List[str]], start: str, max_depth: int = 3) -> int: visited = {start} frontier = [start] @@ -58,15 +105,41 @@ def bfs_limited(adjacency: Dict[str, List[str]], start: str, max_depth: int = 3) return len(visited) -def run_bench(fixture_dir: Path, sample_size: int = 100) -> dict: +def resolve_overlay_path(fixture_dir: Path, manifest: dict, explicit: Optional[Path]) -> Optional[Path]: + if explicit: + return explicit.resolve() + + overlay_manifest = manifest.get("overlay") if isinstance(manifest, dict) else None + if isinstance(overlay_manifest, dict): + path_value = overlay_manifest.get("path") + if path_value: + candidate = Path(path_value) + return candidate if candidate.is_absolute() else (fixture_dir / candidate) + + default = fixture_dir / "overlay.ndjson" + return default if default.exists() else None + + +def run_bench(fixture_dir: Path, sample_size: int = 100, overlay_path: Optional[Path] = None) -> dict: nodes_path = fixture_dir / "nodes.ndjson" edges_path = fixture_dir / "edges.ndjson" manifest_path = fixture_dir / "manifest.json" manifest = json.loads(manifest_path.read_text()) if manifest_path.exists() else {} + overlay_resolved = resolve_overlay_path(fixture_dir, manifest, overlay_path) t0 = time.perf_counter() adjacency, edge_count = build_graph(nodes_path, edges_path) + overlay_added = 0 + overlay_nodes = 0 + overlay_hash = None + overlay_ms = 0.0 + + if overlay_resolved: + t_overlay = time.perf_counter() + overlay_added, overlay_nodes = apply_overlay(adjacency, overlay_resolved) + overlay_ms = (time.perf_counter() - t_overlay) * 1000 + overlay_hash = _sha256(overlay_resolved) build_ms = (time.perf_counter() - t0) * 1000 # deterministic sample: first N node ids sorted @@ -83,13 +156,21 @@ def run_bench(fixture_dir: Path, sample_size: int = 100) -> dict: return { "fixture": fixture_dir.name, "nodes": len(adjacency), - "edges": edge_count, + "edges": edge_count + overlay_added, "build_ms": round(build_ms, 2), + "overlay_ms": round(overlay_ms, 2), "bfs_ms": round(bfs_ms, 2), "bfs_samples": len(node_ids), "avg_reach_3": round(avg_reach, 2), "max_reach_3": max_reach, "manifest": manifest, + "overlay": { + "applied": overlay_resolved is not None, + "added_edges": overlay_added, + "introduced_nodes": overlay_nodes, + "path": str(overlay_resolved) if overlay_resolved else None, + "sha256": overlay_hash, + }, } @@ -98,13 +179,15 @@ def main() -> int: parser.add_argument("--fixture", required=True, help="Path to fixture directory (nodes.ndjson, edges.ndjson)") parser.add_argument("--output", required=True, help="Path to write results JSON") parser.add_argument("--samples", type=int, default=100, help="Number of starting nodes to sample deterministically") + parser.add_argument("--overlay", help="Optional overlay NDJSON path; defaults to overlay.ndjson next to fixture or manifest overlay.path") args = parser.parse_args() fixture_dir = Path(args.fixture).resolve() out_path = Path(args.output).resolve() out_path.parent.mkdir(parents=True, exist_ok=True) - result = run_bench(fixture_dir, sample_size=args.samples) + explicit_overlay = Path(args.overlay).resolve() if args.overlay else None + result = run_bench(fixture_dir, sample_size=args.samples, overlay_path=explicit_overlay) out_path.write_text(json.dumps(result, indent=2, sort_keys=True)) print(f"Wrote results to {out_path}") return 0 diff --git a/src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh b/src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh index 58a87c8db..1dca0f62c 100644 --- a/src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh +++ b/src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh @@ -6,6 +6,7 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "${ROOT}/../../../.." && pwd)" FIXTURES_ROOT="${FIXTURES_ROOT:-${REPO_ROOT}/samples/graph/interim}" OUT_DIR="${OUT_DIR:-$ROOT/results}" +OVERLAY_ROOT="${OVERLAY_ROOT:-${FIXTURES_ROOT}}" SAMPLES="${SAMPLES:-100}" mkdir -p "${OUT_DIR}" @@ -15,7 +16,14 @@ run_one() { local name name="$(basename "${fixture}")" local out_file="${OUT_DIR}/${name}.json" - python "${ROOT}/graph_bench.py" --fixture "${fixture}" --output "${out_file}" --samples "${SAMPLES}" + local overlay_candidate="${OVERLAY_ROOT}/${name}/overlay.ndjson" + + args=("--fixture" "${fixture}" "--output" "${out_file}" "--samples" "${SAMPLES}") + if [[ -f "${overlay_candidate}" ]]; then + args+=("--overlay" "${overlay_candidate}") + fi + + python "${ROOT}/graph_bench.py" "${args[@]}" } run_one "${FIXTURES_ROOT}/graph-50k" diff --git a/src/Bench/StellaOps.Bench/Graph/tests/__pycache__/test_graph_bench.cpython-312.pyc b/src/Bench/StellaOps.Bench/Graph/tests/__pycache__/test_graph_bench.cpython-312.pyc new file mode 100644 index 000000000..9a2670251 Binary files /dev/null and b/src/Bench/StellaOps.Bench/Graph/tests/__pycache__/test_graph_bench.cpython-312.pyc differ diff --git a/src/Bench/StellaOps.Bench/Graph/tests/test_graph_bench.py b/src/Bench/StellaOps.Bench/Graph/tests/test_graph_bench.py new file mode 100644 index 000000000..0970fdd83 --- /dev/null +++ b/src/Bench/StellaOps.Bench/Graph/tests/test_graph_bench.py @@ -0,0 +1,63 @@ +import json +import sys +import tempfile +from pathlib import Path + +import unittest + +ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + + +class GraphBenchTests(unittest.TestCase): + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def _write_ndjson(self, path: Path, records): + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as f: + for record in records: + f.write(json.dumps(record)) + f.write("\n") + + def test_overlay_edges_are_applied_and_counted(self): + from graph_bench import run_bench + + fixture = self.root / "fixture" + fixture.mkdir() + + self._write_ndjson(fixture / "nodes.ndjson", [{"id": "a"}, {"id": "b"}]) + self._write_ndjson(fixture / "edges.ndjson", [{"source": "a", "target": "b"}]) + self._write_ndjson(fixture / "overlay.ndjson", [{"source": "b", "target": "a"}]) + + result = run_bench(fixture, sample_size=2) + + self.assertEqual(result["nodes"], 2) + self.assertEqual(result["edges"], 2) # overlay added one edge + self.assertTrue(result["overlay"]["applied"]) + self.assertEqual(result["overlay"]["added_edges"], 1) + self.assertEqual(result["overlay"]["introduced_nodes"], 0) + + def test_overlay_is_optional(self): + from graph_bench import run_bench + + fixture = self.root / "fixture-no-overlay" + fixture.mkdir() + + self._write_ndjson(fixture / "nodes.ndjson", [{"id": "x"}, {"id": "y"}]) + self._write_ndjson(fixture / "edges.ndjson", [{"source": "x", "target": "y"}]) + + result = run_bench(fixture, sample_size=2) + + self.assertEqual(result["edges"], 1) + self.assertFalse(result["overlay"]["applied"]) + self.assertEqual(result["overlay"]["added_edges"], 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs b/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs index 274ce4dc6..7be90383b 100644 --- a/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs +++ b/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs @@ -7,18 +7,57 @@ */ import fs from "fs"; import path from "path"; +import crypto from "crypto"; function readJson(p) { return JSON.parse(fs.readFileSync(p, "utf-8")); } -function buildPlan(scenarios, manifest, fixtureName) { +function sha256File(filePath) { + const hash = crypto.createHash("sha256"); + hash.update(fs.readFileSync(filePath)); + return hash.digest("hex"); +} + +function resolveOverlay(fixtureDir, manifest) { + const manifestOverlay = manifest?.overlay?.path; + const candidate = manifestOverlay + ? path.isAbsolute(manifestOverlay) + ? manifestOverlay + : path.join(fixtureDir, manifestOverlay) + : path.join(fixtureDir, "overlay.ndjson"); + + if (!fs.existsSync(candidate)) { + return null; + } + + return { + path: candidate, + sha256: sha256File(candidate), + }; +} + +function buildPlan(scenarios, manifest, fixtureName, fixtureDir) { const now = new Date().toISOString(); + const seed = process.env.UI_BENCH_SEED || "424242"; + const traceId = + process.env.UI_BENCH_TRACE_ID || + (crypto.randomUUID ? crypto.randomUUID() : `trace-${Date.now()}`); + const overlay = resolveOverlay(fixtureDir, manifest); + return { version: "1.0.0", fixture: fixtureName, manifestHash: manifest?.hashes || {}, + overlay, timestamp: now, + seed, + traceId, + viewport: { + width: 1280, + height: 720, + deviceScaleFactor: 1, + }, steps: scenarios.map((s, idx) => ({ order: idx + 1, id: s.id, @@ -41,7 +80,12 @@ function main() { const manifest = fs.existsSync(manifestPath) ? readJson(manifestPath) : {}; const scenarios = readJson(scenariosPath).scenarios || []; - const plan = buildPlan(scenarios, manifest, path.basename(fixtureDir)); + const plan = buildPlan( + scenarios, + manifest, + path.basename(fixtureDir), + fixtureDir + ); fs.mkdirSync(path.dirname(outputPath), { recursive: true }); fs.writeFileSync(outputPath, JSON.stringify(plan, null, 2)); console.log(`Wrote plan to ${outputPath}`); diff --git a/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.test.mjs b/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.test.mjs new file mode 100644 index 000000000..013f81ee8 --- /dev/null +++ b/src/Bench/StellaOps.Bench/Graph/ui_bench_driver.test.mjs @@ -0,0 +1,42 @@ +import assert from "node:assert"; +import { test } from "node:test"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); + +test("ui bench driver emits overlay + seed metadata", () => { + const tmp = fs.mkdtempSync(path.join(process.cwd(), "tmp-ui-bench-")); + const fixtureDir = path.join(tmp, "fixture"); + fs.mkdirSync(fixtureDir, { recursive: true }); + + // minimal fixture files + fs.writeFileSync(path.join(fixtureDir, "manifest.json"), JSON.stringify({ hashes: { nodes: "abc" } })); + fs.writeFileSync(path.join(fixtureDir, "overlay.ndjson"), "{\"source\":\"a\",\"target\":\"b\"}\n"); + + const scenariosPath = path.join(tmp, "scenarios.json"); + fs.writeFileSync( + scenariosPath, + JSON.stringify({ version: "1.0.0", scenarios: [{ id: "load", name: "Load", steps: ["navigate"] }] }) + ); + + const outputPath = path.join(tmp, "plan.json"); + const env = { ...process.env, UI_BENCH_SEED: "1337", UI_BENCH_TRACE_ID: "trace-test" }; + const driverPath = path.join(__dirname, "ui_bench_driver.mjs"); + const result = spawnSync(process.execPath, [driverPath, fixtureDir, scenariosPath, outputPath], { env }); + assert.strictEqual(result.status, 0, result.stderr?.toString()); + + const plan = JSON.parse(fs.readFileSync(outputPath, "utf-8")); + assert.strictEqual(plan.fixture, "fixture"); + assert.strictEqual(plan.seed, "1337"); + assert.strictEqual(plan.traceId, "trace-test"); + assert.ok(plan.overlay); + assert.ok(plan.overlay.path.endsWith("overlay.ndjson")); + assert.ok(plan.overlay.sha256); + assert.deepStrictEqual(plan.viewport, { width: 1280, height: 720, deviceScaleFactor: 1 }); + + fs.rmSync(tmp, { recursive: true, force: true }); +}); diff --git a/src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md b/src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md index 26686a27b..b2e92c415 100644 --- a/src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md +++ b/src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md @@ -4,6 +4,7 @@ Purpose: provide a deterministic, headless flow for measuring graph UI interacti ## Scope - Use synthetic fixtures under `samples/graph/interim/` until canonical SAMPLES-GRAPH-24-003 lands. +- Optional overlay layer (`overlay.ndjson`) is loaded when present and toggled during the run to capture render/merge overhead. - Drive a deterministic sequence of interactions: 1) Load graph canvas with specified fixture. 2) Pan to node `pkg-000001`. @@ -11,7 +12,7 @@ Purpose: provide a deterministic, headless flow for measuring graph UI interacti 4) Apply filter `name contains "package-0001"`. 5) Select node, expand neighbors (depth 1), collapse. 6) Toggle overlay layer (once available). -- Capture timings: initial render, filter apply, expand/collapse, overlay toggle. +- Capture timings: initial render, filter apply, expand/collapse, overlay toggle (when available). ## Determinism rules - Fixed seed for any randomized layouts (seed `424242`). diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md index dd932e2d4..6ff1c2909 100644 --- a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md @@ -2,4 +2,4 @@ ### Unreleased -No analyzer rules currently scheduled for release. +- CONCELIER0004: Flag direct `new HttpClient()` usage inside `StellaOps.Concelier.Connector*` namespaces; require sandboxed `IHttpClientFactory` to enforce allow/deny lists. diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/ConnectorHttpClientSandboxAnalyzer.cs b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/ConnectorHttpClientSandboxAnalyzer.cs new file mode 100644 index 000000000..bdb481121 --- /dev/null +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/ConnectorHttpClientSandboxAnalyzer.cs @@ -0,0 +1,53 @@ +using System.Collections.Immutable; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.CodeAnalysis.CSharp.Syntax; +using Microsoft.CodeAnalysis.Diagnostics; + +namespace StellaOps.Concelier.Analyzers; + +[DiagnosticAnalyzer(LanguageNames.CSharp)] +public sealed class ConnectorHttpClientSandboxAnalyzer : DiagnosticAnalyzer +{ + public const string DiagnosticId = "CONCELIER0004"; + + private static readonly DiagnosticDescriptor Rule = new( + id: DiagnosticId, + title: "Connector HTTP clients must use sandboxed factory", + messageFormat: "Use IHttpClientFactory or connector sandbox helpers instead of 'new HttpClient()' inside Concelier connectors.", + category: "Sandbox", + defaultSeverity: DiagnosticSeverity.Warning, + isEnabledByDefault: true, + description: "Direct HttpClient construction bypasses connector allowlist/denylist and proxy policies. Use IHttpClientFactory or sandboxed handlers."); + + public override ImmutableArray SupportedDiagnostics => ImmutableArray.Create(Rule); + + public override void Initialize(AnalysisContext context) + { + context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None); + context.EnableConcurrentExecution(); + context.RegisterSyntaxNodeAction(AnalyzeObjectCreation, SyntaxKind.ObjectCreationExpression); + } + + private static void AnalyzeObjectCreation(SyntaxNodeAnalysisContext context) + { + if (context.Node is not ObjectCreationExpressionSyntax objectCreation) + { + return; + } + + var type = context.SemanticModel.GetTypeInfo(objectCreation, context.CancellationToken).Type; + if (type?.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat) != "global::System.Net.Http.HttpClient") + { + return; + } + + var containingSymbol = context.ContainingSymbol?.ContainingNamespace?.ToDisplayString(); + if (containingSymbol is null || !containingSymbol.StartsWith("StellaOps.Concelier.Connector")) + { + return; + } + + context.ReportDiagnostic(Diagnostic.Create(Rule, objectCreation.GetLocation())); + } +} diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/AdvisoryObservationWriteGuard.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/AdvisoryObservationWriteGuard.cs index 1be955b11..dc59020de 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/AdvisoryObservationWriteGuard.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/AdvisoryObservationWriteGuard.cs @@ -1,3 +1,5 @@ +using System; +using System.Linq; using Microsoft.Extensions.Logging; using StellaOps.Concelier.Models.Observations; @@ -29,6 +31,24 @@ public sealed class AdvisoryObservationWriteGuard : IAdvisoryObservationWriteGua ArgumentNullException.ThrowIfNull(observation); var newContentHash = observation.Upstream.ContentHash; + var signature = observation.Upstream.Signature; + + if (!IsSha256(newContentHash)) + { + _logger.LogWarning( + "Observation {ObservationId} rejected: content hash must be canonical sha256: but was {ContentHash}", + observation.ObservationId, + newContentHash); + return ObservationWriteDisposition.RejectInvalidProvenance; + } + + if (!SignatureShapeIsValid(signature)) + { + _logger.LogWarning( + "Observation {ObservationId} rejected: signature metadata missing or inconsistent for provenance enforcement", + observation.ObservationId); + return ObservationWriteDisposition.RejectInvalidProvenance; + } if (string.IsNullOrWhiteSpace(existingContentHash)) { @@ -56,4 +76,36 @@ public sealed class AdvisoryObservationWriteGuard : IAdvisoryObservationWriteGua return ObservationWriteDisposition.RejectMutation; } + + private static bool IsSha256(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return false; + } + + return value.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) + && value.Length == "sha256:".Length + 64 + && value["sha256:".Length..].All(c => Uri.IsHexDigit(c)); + } + + private static bool SignatureShapeIsValid(AdvisoryObservationSignature signature) + { + if (signature is null) + { + return false; + } + + if (signature.Present) + { + return !string.IsNullOrWhiteSpace(signature.Format) + && !string.IsNullOrWhiteSpace(signature.KeyId) + && !string.IsNullOrWhiteSpace(signature.Signature); + } + + // When signature is not present, auxiliary fields must be empty to prevent stale metadata. + return string.IsNullOrEmpty(signature.Format) + && string.IsNullOrEmpty(signature.KeyId) + && string.IsNullOrEmpty(signature.Signature); + } } diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/IAdvisoryObservationWriteGuard.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/IAdvisoryObservationWriteGuard.cs index 9fcb23318..4b2c341f1 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/IAdvisoryObservationWriteGuard.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Aoc/IAdvisoryObservationWriteGuard.cs @@ -32,6 +32,11 @@ public enum ObservationWriteDisposition /// SkipIdentical, + /// + /// Observation is malformed (missing provenance/signature/hash guarantees) and must be rejected. + /// + RejectInvalidProvenance, + /// /// Observation differs from existing - reject mutation (append-only violation). /// diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Aoc/AdvisoryObservationWriteGuardTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Aoc/AdvisoryObservationWriteGuardTests.cs index d68058e98..b57b93919 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Aoc/AdvisoryObservationWriteGuardTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Aoc/AdvisoryObservationWriteGuardTests.cs @@ -15,6 +15,10 @@ namespace StellaOps.Concelier.Core.Tests.Aoc; /// public sealed class AdvisoryObservationWriteGuardTests { + private const string HashA = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + private const string HashB = "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"; + private const string HashC = "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"; + private readonly AdvisoryObservationWriteGuard _guard; public AdvisoryObservationWriteGuardTests() @@ -26,7 +30,7 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_NewObservation_ReturnsProceed() { // Arrange - var observation = CreateObservation("obs-1", "sha256:abc123"); + var observation = CreateObservation("obs-1", HashA); // Act var result = _guard.ValidateWrite(observation, existingContentHash: null); @@ -39,7 +43,7 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_NewObservation_WithEmptyExistingHash_ReturnsProceed() { // Arrange - var observation = CreateObservation("obs-2", "sha256:def456"); + var observation = CreateObservation("obs-2", HashB); // Act var result = _guard.ValidateWrite(observation, existingContentHash: ""); @@ -52,7 +56,7 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_NewObservation_WithWhitespaceExistingHash_ReturnsProceed() { // Arrange - var observation = CreateObservation("obs-3", "sha256:ghi789"); + var observation = CreateObservation("obs-3", HashC); // Act var result = _guard.ValidateWrite(observation, existingContentHash: " "); @@ -65,11 +69,10 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_IdenticalContent_ReturnsSkipIdentical() { // Arrange - const string contentHash = "sha256:abc123"; - var observation = CreateObservation("obs-4", contentHash); + var observation = CreateObservation("obs-4", HashA); // Act - var result = _guard.ValidateWrite(observation, existingContentHash: contentHash); + var result = _guard.ValidateWrite(observation, existingContentHash: HashA); // Assert result.Should().Be(ObservationWriteDisposition.SkipIdentical); @@ -79,10 +82,10 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_IdenticalContent_CaseInsensitive_ReturnsSkipIdentical() { // Arrange - var observation = CreateObservation("obs-5", "SHA256:ABC123"); + var observation = CreateObservation("obs-5", "SHA256:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"); // Act - var result = _guard.ValidateWrite(observation, existingContentHash: "sha256:abc123"); + var result = _guard.ValidateWrite(observation, existingContentHash: HashA); // Assert result.Should().Be(ObservationWriteDisposition.SkipIdentical); @@ -92,10 +95,10 @@ public sealed class AdvisoryObservationWriteGuardTests public void ValidateWrite_DifferentContent_ReturnsRejectMutation() { // Arrange - var observation = CreateObservation("obs-6", "sha256:newcontent"); + var observation = CreateObservation("obs-6", HashB); // Act - var result = _guard.ValidateWrite(observation, existingContentHash: "sha256:oldcontent"); + var result = _guard.ValidateWrite(observation, existingContentHash: HashA); // Assert result.Should().Be(ObservationWriteDisposition.RejectMutation); @@ -113,9 +116,8 @@ public sealed class AdvisoryObservationWriteGuardTests } [Theory] - [InlineData("sha256:a", "sha256:b")] - [InlineData("sha256:hash1", "sha256:hash2")] - [InlineData("md5:abc", "sha256:abc")] + [InlineData(HashB, HashC)] + [InlineData(HashC, HashA)] public void ValidateWrite_ContentMismatch_ReturnsRejectMutation(string newHash, string existingHash) { // Arrange @@ -129,9 +131,8 @@ public sealed class AdvisoryObservationWriteGuardTests } [Theory] - [InlineData("sha256:identical")] - [InlineData("SHA256:IDENTICAL")] - [InlineData("sha512:longerhash1234567890")] + [InlineData(HashA)] + [InlineData("SHA256:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB")] public void ValidateWrite_ExactMatch_ReturnsSkipIdentical(string hash) { // Arrange @@ -144,7 +145,41 @@ public sealed class AdvisoryObservationWriteGuardTests result.Should().Be(ObservationWriteDisposition.SkipIdentical); } - private static AdvisoryObservation CreateObservation(string observationId, string contentHash) + [Theory] + [InlineData("md5:abc")] + [InlineData("sha256:short")] + public void ValidateWrite_InvalidHash_ReturnsRejectInvalidProvenance(string hash) + { + var observation = CreateObservation("obs-invalid-hash", hash); + + var result = _guard.ValidateWrite(observation, existingContentHash: null); + + result.Should().Be(ObservationWriteDisposition.RejectInvalidProvenance); + } + + [Fact] + public void ValidateWrite_SignaturePresentMissingFields_ReturnsRejectInvalidProvenance() + { + var badSignature = new AdvisoryObservationSignature(true, null, null, null); + var observation = CreateObservation("obs-bad-sig", HashA, badSignature); + + var result = _guard.ValidateWrite(observation, existingContentHash: null); + + result.Should().Be(ObservationWriteDisposition.RejectInvalidProvenance); + } + + [Fact] + public void Observation_TenantIsLowercased() + { + var observation = CreateObservation("obs-tenant", HashA, tenant: "Tenant:Mixed"); + observation.Tenant.Should().Be("tenant:mixed"); + } + + private static AdvisoryObservation CreateObservation( + string observationId, + string contentHash, + AdvisoryObservationSignature? signatureOverride = null, + string tenant = "test-tenant") { var source = new AdvisoryObservationSource( vendor: "test-vendor", @@ -152,11 +187,11 @@ public sealed class AdvisoryObservationWriteGuardTests api: "test-api", collectorVersion: "1.0.0"); - var signature = new AdvisoryObservationSignature( - present: false, - format: null, - keyId: null, - signature: null); + var signature = signatureOverride ?? new AdvisoryObservationSignature( + present: true, + format: "dsse", + keyId: "test-key", + signature: "ZmFrZS1zaWc="); var upstream = new AdvisoryObservationUpstream( upstreamId: $"upstream-{observationId}", @@ -184,7 +219,7 @@ public sealed class AdvisoryObservationWriteGuardTests return new AdvisoryObservation( observationId: observationId, - tenant: "test-tenant", + tenant: tenant, source: source, upstream: upstream, content: content, diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryLinksetDeterminismTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryLinksetDeterminismTests.cs new file mode 100644 index 000000000..cb7413a97 --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryLinksetDeterminismTests.cs @@ -0,0 +1,86 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using FluentAssertions; +using StellaOps.Concelier.Core.Linksets; +using StellaOps.Concelier.Models; +using Xunit; + +namespace StellaOps.Concelier.Core.Tests.Linksets; + +/// +/// Determinism and provenance-focused tests aligned with CI1–CI10 gap remediation. +/// +public sealed class AdvisoryLinksetDeterminismTests +{ + [Fact] + public void IdempotencyKey_IsStableAcrossObservationOrdering() + { + // Arrange + var createdAt = new DateTimeOffset(2025, 12, 2, 0, 0, 0, TimeSpan.Zero); + var observationsA = ImmutableArray.Create("obs-b", "obs-a"); + var observationsB = ImmutableArray.Create("obs-a", "obs-b"); + + var linksetA = new AdvisoryLinkset( + TenantId: "tenant-a", + Source: "nvd", + AdvisoryId: "CVE-2025-9999", + ObservationIds: observationsA, + Normalized: null, + Provenance: new AdvisoryLinksetProvenance( + ObservationHashes: new[] { "sha256:1111", "sha256:2222" }, + ToolVersion: "1.0.0", + PolicyHash: "policy-hash-1"), + Confidence: 0.8, + Conflicts: null, + CreatedAt: createdAt, + BuiltByJobId: "job-1"); + + var linksetB = linksetA with { ObservationIds = observationsB }; + + // Act + var evtA = AdvisoryLinksetUpdatedEvent.FromLinkset(linksetA, null, "linkset-1", null); + var evtB = AdvisoryLinksetUpdatedEvent.FromLinkset(linksetB, null, "linkset-1", null); + + // Assert + evtA.IdempotencyKey.Should().Be(evtB.IdempotencyKey); + } + + [Fact] + public void Conflicts_AreDeterministicallyDedupedAndSourcesFilled() + { + // Arrange + var inputs = new[] + { + new LinksetCorrelation.Input( + Vendor: "nvd", + FetchedAt: DateTimeOffset.Parse("2025-12-01T00:00:00Z"), + Aliases: new[] { "CVE-2025-1111" }, + Purls: Array.Empty(), + Cpes: Array.Empty(), + References: Array.Empty()), + new LinksetCorrelation.Input( + Vendor: "vendor", + FetchedAt: DateTimeOffset.Parse("2025-12-01T00:05:00Z"), + Aliases: new[] { "CVE-2025-2222" }, + Purls: Array.Empty(), + Cpes: Array.Empty(), + References: Array.Empty()) + }; + + var duplicateConflicts = new List + { + new("aliases", "alias-inconsistency", new[] { "nvd:CVE-2025-1111", "vendor:CVE-2025-2222" }, null), + new("aliases", "alias-inconsistency", new[] { "nvd:CVE-2025-1111", "vendor:CVE-2025-2222" }, Array.Empty()) + }; + + // Act + var (_, conflicts) = LinksetCorrelation.Compute(inputs, duplicateConflicts); + + // Assert + conflicts.Should().HaveCount(1); + conflicts[0].Field.Should().Be("aliases"); + conflicts[0].Reason.Should().Be("alias-inconsistency"); + conflicts[0].SourceIds.Should().ContainInOrder("nvd", "vendor"); + } +} diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Schemas/SchemaManifestTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Schemas/SchemaManifestTests.cs new file mode 100644 index 000000000..9141b31d3 --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Schemas/SchemaManifestTests.cs @@ -0,0 +1,86 @@ +using System; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text.Json; +using FluentAssertions; +using Xunit; + +namespace StellaOps.Concelier.Core.Tests.Schemas; + +/// +/// Verifies schema bundle digests and offline bundle sample constraints (CI1–CI10). +/// +public sealed class SchemaManifestTests +{ + private static readonly string SchemaRoot = ResolveSchemaRoot(); + + [Fact] + public void SchemaManifest_DigestsMatchFilesystem() + { + var manifestPath = Path.Combine(SchemaRoot, "schema.manifest.json"); + using var doc = JsonDocument.Parse(File.ReadAllText(manifestPath)); + + var files = doc.RootElement.GetProperty("files").EnumerateArray().ToArray(); + files.Should().NotBeEmpty("schema manifest must contain at least one entry"); + + foreach (var fileEl in files) + { + var path = fileEl.GetProperty("path").GetString()!; + var expected = fileEl.GetProperty("sha256").GetString()!; + + var fullPath = Path.Combine(SchemaRoot, path); + File.Exists(fullPath).Should().BeTrue($"manifest entry {path} should exist"); + + var actual = ComputeSha256(fullPath); + actual.Should().Be(expected, $"digest for {path} should be canonical"); + } + } + + [Fact] + public void OfflineBundleSample_RespectsStalenessAndHashes() + { + var samplePath = Path.Combine(SchemaRoot, "samples/offline-advisory-bundle.sample.json"); + using var doc = JsonDocument.Parse(File.ReadAllText(samplePath)); + + var snapshot = doc.RootElement.GetProperty("snapshot"); + var staleness = snapshot.GetProperty("stalenessHours").GetInt32(); + staleness.Should().BeLessOrEqualTo(168, "offline bundles must cap snapshot staleness to 7 days"); + + var manifest = doc.RootElement.GetProperty("manifest").EnumerateArray().ToArray(); + manifest.Should().NotBeEmpty(); + foreach (var entry in manifest) + { + var hash = entry.GetProperty("sha256").GetString()!; + hash.Length.Should().Be(64); + } + + var hashes = doc.RootElement.GetProperty("hashes"); + hashes.GetProperty("sha256").GetString()!.Length.Should().Be(64); + } + + private static string ComputeSha256(string path) + { + using var sha = SHA256.Create(); + using var stream = File.OpenRead(path); + var hash = sha.ComputeHash(stream); + return Convert.ToHexString(hash).ToLowerInvariant(); + } + + private static string ResolveSchemaRoot() + { + var current = AppContext.BaseDirectory; + while (!string.IsNullOrEmpty(current)) + { + var candidate = Path.Combine(current, "docs/modules/concelier/schemas"); + if (Directory.Exists(candidate)) + { + return candidate; + } + + current = Directory.GetParent(current)?.FullName; + } + + throw new DirectoryNotFoundException("Unable to locate docs/modules/concelier/schemas from test base directory."); + } +} diff --git a/src/Findings/StellaOps.Findings.Ledger.WebService/Program.cs b/src/Findings/StellaOps.Findings.Ledger.WebService/Program.cs index efccdb4c5..2a1ea520f 100644 --- a/src/Findings/StellaOps.Findings.Ledger.WebService/Program.cs +++ b/src/Findings/StellaOps.Findings.Ledger.WebService/Program.cs @@ -25,6 +25,7 @@ using StellaOps.Findings.Ledger.WebService.Mappings; using StellaOps.Findings.Ledger.WebService.Services; using StellaOps.Telemetry.Core; using StellaOps.Findings.Ledger.Services.Security; +using StellaOps.Findings.Ledger.Observability; const string LedgerWritePolicy = "ledger.events.write"; const string LedgerExportPolicy = "ledger.export.read"; @@ -45,6 +46,8 @@ var bootstrapOptions = builder.Configuration.BindOptions( LedgerServiceOptions.SectionName, (opts, _) => opts.Validate()); +LedgerMetrics.ConfigureQuotas(bootstrapOptions.Quotas.MaxIngestBacklog); + builder.Host.UseSerilog((context, services, loggerConfiguration) => { loggerConfiguration diff --git a/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerAnchorQueue.cs b/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerAnchorQueue.cs index d4b39848a..ec1a19007 100644 --- a/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerAnchorQueue.cs +++ b/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerAnchorQueue.cs @@ -21,7 +21,7 @@ public sealed class LedgerAnchorQueue public ValueTask EnqueueAsync(LedgerEventRecord record, CancellationToken cancellationToken) { var writeTask = _channel.Writer.WriteAsync(record, cancellationToken); - LedgerMetrics.IncrementBacklog(); + LedgerMetrics.IncrementBacklog(record.TenantId); return writeTask; } diff --git a/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerMerkleAnchorWorker.cs b/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerMerkleAnchorWorker.cs index c5e859bfd..f7b49e41d 100644 --- a/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerMerkleAnchorWorker.cs +++ b/src/Findings/StellaOps.Findings.Ledger/Infrastructure/Merkle/LedgerMerkleAnchorWorker.cs @@ -37,7 +37,7 @@ public sealed class LedgerMerkleAnchorWorker : BackgroundService { await foreach (var record in _queue.ReadAllAsync(stoppingToken)) { - LedgerMetrics.DecrementBacklog(); + LedgerMetrics.DecrementBacklog(record.TenantId); await HandleEventAsync(record, stoppingToken).ConfigureAwait(false); } } diff --git a/src/Findings/StellaOps.Findings.Ledger/Observability/LedgerMetrics.cs b/src/Findings/StellaOps.Findings.Ledger/Observability/LedgerMetrics.cs index 68464ee95..50a68f638 100644 --- a/src/Findings/StellaOps.Findings.Ledger/Observability/LedgerMetrics.cs +++ b/src/Findings/StellaOps.Findings.Ledger/Observability/LedgerMetrics.cs @@ -1,5 +1,7 @@ +using System; using System.Collections.Concurrent; using System.Diagnostics.Metrics; +using System.Reflection; namespace StellaOps.Findings.Ledger.Observability; @@ -22,6 +24,14 @@ internal static class LedgerMetrics "ledger_events_total", description: "Number of ledger events appended."); + private static readonly Counter BackpressureApplied = Meter.CreateCounter( + "ledger_backpressure_applied_total", + description: "Times ingest backpressure thresholds were exceeded."); + + private static readonly Counter QuotaRejections = Meter.CreateCounter( + "ledger_quota_rejections_total", + description: "Requests rejected due to configured quotas."); + private static readonly Histogram ProjectionApplySeconds = Meter.CreateHistogram( "ledger_projection_apply_seconds", unit: "s", @@ -45,21 +55,38 @@ internal static class LedgerMetrics "ledger_merkle_anchor_failures_total", description: "Count of Merkle anchor failures by reason."); + private static readonly Counter AttachmentsEncryptionFailures = Meter.CreateCounter( + "ledger_attachments_encryption_failures_total", + description: "Count of attachment encryption/signing/upload failures."); + private static readonly ObservableGauge ProjectionLagGauge = Meter.CreateObservableGauge("ledger_projection_lag_seconds", ObserveProjectionLag, unit: "s", description: "Lag between ledger recorded_at and projection application time."); private static readonly ObservableGauge IngestBacklogGauge = Meter.CreateObservableGauge("ledger_ingest_backlog_events", ObserveBacklog, - description: "Number of events buffered for ingestion/anchoring."); + description: "Number of events buffered for ingestion/anchoring per tenant."); + + private static readonly ObservableGauge QuotaRemainingGauge = + Meter.CreateObservableGauge("ledger_quota_remaining", ObserveQuotaRemaining, + description: "Remaining ingest backlog capacity before backpressure applies."); private static readonly ObservableGauge DbConnectionsGauge = Meter.CreateObservableGauge("ledger_db_connections_active", ObserveDbConnections, description: "Active PostgreSQL connections by role."); + private static readonly ObservableGauge AppVersionGauge = + Meter.CreateObservableGauge("ledger_app_version_info", ObserveAppVersion, + description: "Static gauge exposing build version and git sha."); + private static readonly ConcurrentDictionary ProjectionLagByTenant = new(StringComparer.Ordinal); private static readonly ConcurrentDictionary DbConnectionsByRole = new(StringComparer.OrdinalIgnoreCase); - private static long _ingestBacklog; + private static readonly ConcurrentDictionary BacklogByTenant = new(StringComparer.Ordinal); + + private static long _ingestBacklogLimit = 5000; + + private static readonly string AppVersion = Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "0.0.0"; + private static readonly string GitSha = Environment.GetEnvironmentVariable("GIT_SHA") ?? "unknown"; public static void RecordWriteSuccess(TimeSpan duration, string? tenantId, string? eventType, string? source) { @@ -127,17 +154,55 @@ internal static class LedgerMetrics MerkleAnchorFailures.Add(1, tags); } - public static void IncrementBacklog() => Interlocked.Increment(ref _ingestBacklog); - - public static void DecrementBacklog() + public static void RecordAttachmentFailure(string tenantId, string stage) { - var value = Interlocked.Decrement(ref _ingestBacklog); - if (value < 0) + var tags = new KeyValuePair[] { - Interlocked.Exchange(ref _ingestBacklog, 0); + new("tenant", tenantId), + new("stage", stage) + }; + AttachmentsEncryptionFailures.Add(1, tags); + } + + public static void ConfigureQuotas(long ingestBacklogLimit) + { + if (ingestBacklogLimit > 0) + { + Interlocked.Exchange(ref _ingestBacklogLimit, ingestBacklogLimit); } } + public static long IncrementBacklog(string? tenantId = null) + { + var key = NormalizeTenant(tenantId); + var backlog = BacklogByTenant.AddOrUpdate(key, _ => 1, (_, current) => current + 1); + if (backlog > _ingestBacklogLimit) + { + BackpressureApplied.Add(1, new KeyValuePair[] + { + new("tenant", key), + new("reason", "ingest_backlog"), + new("limit", _ingestBacklogLimit) + }); + } + return backlog; + } + + public static void RecordQuotaRejection(string tenantId, string reason) + { + QuotaRejections.Add(1, new KeyValuePair[] + { + new("tenant", NormalizeTenant(tenantId)), + new("reason", reason) + }); + } + + public static void DecrementBacklog(string? tenantId = null) + { + var key = NormalizeTenant(tenantId); + BacklogByTenant.AddOrUpdate(key, _ => 0, (_, current) => Math.Max(0, current - 1)); + } + public static void ConnectionOpened(string role) { var normalized = NormalizeRole(role); @@ -150,12 +215,19 @@ internal static class LedgerMetrics DbConnectionsByRole.AddOrUpdate(normalized, _ => 0, (_, current) => Math.Max(0, current - 1)); } + public static void IncrementDbConnection(string role) => ConnectionOpened(role); + + public static void DecrementDbConnection(string role) => ConnectionClosed(role); + public static void UpdateProjectionLag(string? tenantId, double lagSeconds) { var key = string.IsNullOrWhiteSpace(tenantId) ? string.Empty : tenantId; ProjectionLagByTenant[key] = lagSeconds < 0 ? 0 : lagSeconds; } + public static void RecordProjectionLag(TimeSpan lag, string? tenantId) => + UpdateProjectionLag(tenantId, lag.TotalSeconds); + private static IEnumerable> ObserveProjectionLag() { foreach (var kvp in ProjectionLagByTenant) @@ -166,7 +238,19 @@ internal static class LedgerMetrics private static IEnumerable> ObserveBacklog() { - yield return new Measurement(Interlocked.Read(ref _ingestBacklog)); + foreach (var kvp in BacklogByTenant) + { + yield return new Measurement(kvp.Value, new KeyValuePair("tenant", kvp.Key)); + } + } + + private static IEnumerable> ObserveQuotaRemaining() + { + foreach (var kvp in BacklogByTenant) + { + var remaining = Math.Max(0, _ingestBacklogLimit - kvp.Value); + yield return new Measurement(remaining, new KeyValuePair("tenant", kvp.Key)); + } } private static IEnumerable> ObserveDbConnections() @@ -177,5 +261,13 @@ internal static class LedgerMetrics } } + private static IEnumerable> ObserveAppVersion() + { + yield return new Measurement(1, new KeyValuePair("version", AppVersion), + new KeyValuePair("git_sha", GitSha)); + } + private static string NormalizeRole(string role) => string.IsNullOrWhiteSpace(role) ? "unspecified" : role.ToLowerInvariant(); + + private static string NormalizeTenant(string? tenantId) => string.IsNullOrWhiteSpace(tenantId) ? string.Empty : tenantId; } diff --git a/src/Findings/StellaOps.Findings.Ledger/Options/LedgerServiceOptions.cs b/src/Findings/StellaOps.Findings.Ledger/Options/LedgerServiceOptions.cs index 7afb671f2..c96588eac 100644 --- a/src/Findings/StellaOps.Findings.Ledger/Options/LedgerServiceOptions.cs +++ b/src/Findings/StellaOps.Findings.Ledger/Options/LedgerServiceOptions.cs @@ -1,3 +1,6 @@ +using System; +using System.Collections.Generic; + namespace StellaOps.Findings.Ledger.Options; public sealed class LedgerServiceOptions @@ -16,6 +19,8 @@ public sealed class LedgerServiceOptions public AttachmentsOptions Attachments { get; init; } = new(); + public QuotaOptions Quotas { get; init; } = new(); + public void Validate() { if (string.IsNullOrWhiteSpace(Database.ConnectionString)) @@ -50,6 +55,7 @@ public sealed class LedgerServiceOptions PolicyEngine.Validate(); Attachments.Validate(); + Quotas.Validate(); } public sealed class DatabaseOptions @@ -207,4 +213,19 @@ public sealed class LedgerServiceOptions } } } + + public sealed class QuotaOptions + { + private const int DefaultBacklog = 5000; + + public long MaxIngestBacklog { get; set; } = DefaultBacklog; + + public void Validate() + { + if (MaxIngestBacklog <= 0) + { + throw new InvalidOperationException("Quotas.MaxIngestBacklog must be greater than zero."); + } + } + } } diff --git a/src/Findings/StellaOps.Findings.Ledger/TASKS.md b/src/Findings/StellaOps.Findings.Ledger/TASKS.md index 306e946d4..7e000e107 100644 --- a/src/Findings/StellaOps.Findings.Ledger/TASKS.md +++ b/src/Findings/StellaOps.Findings.Ledger/TASKS.md @@ -13,3 +13,4 @@ Status changes must be mirrored in `docs/implplan/SPRINT_0120_0000_0001_policy_r | Task ID | Status | Notes | Updated (UTC) | | --- | --- | --- | --- | | LEDGER-OBS-54-001 | DONE | Implemented `/v1/ledger/attestations` with deterministic paging, filter hash guard, and schema/OpenAPI updates. | 2025-11-22 | +| LEDGER-GAPS-121-009 | DONE | FL1–FL10 remediation: schema catalog + export canonicals, Merkle/external anchor policy, tenant isolation/redaction manifest, offline verifier + checksum guard, golden fixtures, backpressure metrics. | 2025-12-02 | diff --git a/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/advisories-canonical.ndjson b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/advisories-canonical.ndjson new file mode 100644 index 000000000..cd3b5b282 --- /dev/null +++ b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/advisories-canonical.ndjson @@ -0,0 +1 @@ +{"shape":"export.v1.canonical","advisoryId":"ADV-2025-010","source":"mirror:nvd","title":"Template injection in sample app","description":"Unsanitised template input leads to RCE.","cwes":["CWE-94"],"cvss":{"version":"3.1","vector":"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H","baseScore":9.8},"epss":{"score":0.72,"percentile":0.98},"kev":true,"published":"2025-11-28T09:00:00Z","modified":"2025-11-30T18:00:00Z","status":"active","projectionVersion":"cycle:v1","cycleHash":"4b2e8ff08bd7cce5d6feaa9ab1c7de8ef9b0c1d2e3f405162738495a0b1c2d3e","provenance":{"ledgerRoot":"8c7d6e5f4c3b2a1908172635443321ffeeddbbccaa99887766554433221100aa","projectorVersion":"proj-1.0.0","policyHash":"sha256:policy-v1","filtersHash":"c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb99887766554433"}} diff --git a/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/findings-canonical.ndjson b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/findings-canonical.ndjson new file mode 100644 index 000000000..a51dfaceb --- /dev/null +++ b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/findings-canonical.ndjson @@ -0,0 +1,2 @@ +{"shape":"export.v1.canonical","findingId":"artifact:sha256:5c1f5f2e1b7c4d8a9e0f123456789abc|pkg:npm/lodash@4.17.21|cve:CVE-2025-1111","eventSequence":42,"observedAt":"2025-12-01T10:00:00Z","policyVersion":"sha256:policy-v1","projectionVersion":"cycle:v1","status":"triaged","severity":6.7,"risk":{"score":8.2,"severity":"high","profileVersion":"risk-profile-v2","explanationId":"550e8400-e29b-41d4-a716-446655440000"},"advisories":[{"id":"ADV-2025-001","cwes":["CWE-79"]}],"evidenceBundleRef":{"digest":"sha256:evidence-001","dsseDigest":"sha256:dsse-001","timelineRef":"timeline://events/123"},"cycleHash":"1f0b6bb757a4dbe2d3c96786b9d4da3e4c3a5d35b4c1a1e5c2e4b9d1786f3d11","provenance":{"ledgerRoot":"9d8f6c1a2b3c4d5e6f708192837465aa9b8c7d6e5f4c3b2a1908172635443321","projectorVersion":"proj-1.0.0","policyHash":"sha256:policy-v1","filtersHash":"a81d6c6d2bcf9c0e7cbb1fcd292e4b7cc21f6d5c4e3f2b1a0c9d8e7f6c5b4a3e"}} +{"shape":"export.v1.canonical","findingId":"artifact:sha256:7d2e4f6a8b9c0d1e2f3a4b5c6d7e8f90|pkg:pypi/django@5.0.0|cve:CVE-2025-2222","eventSequence":84,"observedAt":"2025-12-01T10:30:00Z","policyVersion":"sha256:policy-v1","projectionVersion":"cycle:v1","status":"affected","severity":8.9,"risk":{"score":9.4,"severity":"critical","profileVersion":"risk-profile-v2","explanationId":"660e8400-e29b-41d4-a716-446655440000"},"advisories":[{"id":"ADV-2025-014","cwes":["CWE-352"],"kev":true}],"evidenceBundleRef":{"digest":"sha256:evidence-014","dsseDigest":"sha256:dsse-014","timelineRef":"timeline://events/987"},"cycleHash":"2e0c7cc868b5ecc3e4da7897c0e5eb4f5d4b6c47c5d2b2f6c3f5c0e2897f4e22","provenance":{"ledgerRoot":"8c7d6e5f4c3b2a1908172635443321ffeeddbbccaa99887766554433221100aa","projectorVersion":"proj-1.0.0","policyHash":"sha256:policy-v1","filtersHash":"a81d6c6d2bcf9c0e7cbb1fcd292e4b7cc21f6d5c4e3f2b1a0c9d8e7f6c5b4a3e"}} diff --git a/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/sboms-compact.ndjson b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/sboms-compact.ndjson new file mode 100644 index 000000000..d7f3134b6 --- /dev/null +++ b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/sboms-compact.ndjson @@ -0,0 +1 @@ +{"shape":"export.v1.compact","sbomId":"sbom-oci-sha256-abc123","subject":{"digest":"sha256:abc123","mediaType":"application/vnd.oci.image.manifest.v1+json"},"sbomFormat":"spdx-json","createdAt":"2025-11-30T21:00:00Z","componentsCount":142,"hasVulnerabilities":true,"materials":["sha256:layer-001","sha256:layer-002"],"projectionVersion":"cycle:v1","cycleHash":"5c3f90019ce8ddf6e7ffbbaa1c8eef90a1b2c3d4e5f60718293a4b5c6d7e8f90","provenance":{"ledgerRoot":"9d8f6c1a2b3c4d5e6f708192837465aa9b8c7d6e5f4c3b2a1908172635443321","projectorVersion":"proj-1.0.0","policyHash":"sha256:policy-v1","filtersHash":"d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb9988776655443322"}} diff --git a/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/vex-compact.ndjson b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/vex-compact.ndjson new file mode 100644 index 000000000..c7a670f03 --- /dev/null +++ b/src/Findings/StellaOps.Findings.Ledger/fixtures/golden/vex-compact.ndjson @@ -0,0 +1 @@ +{"shape":"export.v1.compact","vexStatementId":"vex-2025-0001","product":{"purl":"pkg:npm/lodash@4.17.21"},"status":"not_affected","statusJustification":"component_not_present","knownExploited":false,"timestamp":"2025-12-01T11:00:00Z","policyVersion":"sha256:policy-v1","projectionVersion":"cycle:v1","cycleHash":"3a1d7ee97ac6fdd4e5fb98a8c1f6ec5d6c7d8e9fa0b1c2d3e4f506172839405f","provenance":{"ledgerRoot":"9d8f6c1a2b3c4d5e6f708192837465aa9b8c7d6e5f4c3b2a1908172635443321","projectorVersion":"proj-1.0.0","policyHash":"sha256:policy-v1","filtersHash":"b5c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb998877665544"}} diff --git a/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/Program.cs b/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/Program.cs index 23298c97c..9a768d026 100644 --- a/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/Program.cs +++ b/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/Program.cs @@ -1,6 +1,8 @@ using System.CommandLine; using System.Diagnostics; using System.Diagnostics.Metrics; +using System.Security.Cryptography; +using System.Text; using System.Text.Json; using System.Text.Json.Nodes; using Microsoft.Extensions.DependencyInjection; @@ -51,6 +53,10 @@ var metricsOption = new Option( name: "--metrics", description: "Optional path to write metrics snapshot JSON"); +var expectedChecksumOption = new Option( + name: "--expected-checksum", + description: "Optional JSON file containing expected eventStream/projection checksums"); + var root = new RootCommand("Findings Ledger Replay Harness (LEDGER-29-008)"); root.AddOption(fixturesOption); root.AddOption(connectionOption); @@ -58,8 +64,9 @@ root.AddOption(tenantOption); root.AddOption(maxParallelOption); root.AddOption(reportOption); root.AddOption(metricsOption); +root.AddOption(expectedChecksumOption); -root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, int maxParallel, FileInfo? reportFile, FileInfo? metricsFile) => +root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, int maxParallel, FileInfo? reportFile, FileInfo? metricsFile, FileInfo? expectedChecksumsFile) => { await using var host = BuildHost(connection); using var scope = host.Services.CreateScope(); @@ -103,7 +110,7 @@ root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, in meterListener.RecordObservableInstruments(); - var verification = await VerifyLedgerAsync(scope.ServiceProvider, tenant, eventsWritten, cts.Token).ConfigureAwait(false); + var verification = await VerifyLedgerAsync(scope.ServiceProvider, tenant, eventsWritten, expectedChecksumsFile, cts.Token).ConfigureAwait(false); var writeDurations = metrics.HistDouble("ledger_write_duration_seconds").Concat(metrics.HistDouble("ledger_write_latency_seconds")); var writeLatencyP95Ms = Percentile(writeDurations, 95) * 1000; @@ -123,6 +130,8 @@ root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, in ProjectionLagSecondsMax: projectionLagSeconds, BacklogEventsMax: backlogEvents, DbConnectionsObserved: dbConnections, + EventStreamChecksum: verification.EventStreamChecksum, + ProjectionChecksum: verification.ProjectionChecksum, VerificationErrors: verification.Errors.ToArray()); var jsonOptions = new JsonSerializerOptions { WriteIndented = true }; @@ -132,7 +141,8 @@ root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, in if (reportFile is not null) { await File.WriteAllTextAsync(reportFile.FullName, json, cts.Token).ConfigureAwait(false); - await WriteDssePlaceholderAsync(reportFile.FullName, json, cts.Token).ConfigureAwait(false); + var policyHash = Environment.GetEnvironmentVariable("LEDGER_POLICY_HASH"); + await WriteDssePlaceholderAsync(reportFile.FullName, json, policyHash, cts.Token).ConfigureAwait(false); } if (metricsFile is not null) @@ -148,7 +158,7 @@ root.SetHandler(async (FileInfo[] fixtures, string connection, string tenant, in await root.InvokeAsync(args); -static async Task WriteDssePlaceholderAsync(string reportPath, string json, CancellationToken cancellationToken) +static async Task WriteDssePlaceholderAsync(string reportPath, string json, string? policyHash, CancellationToken cancellationToken) { using var sha = System.Security.Cryptography.SHA256.Create(); var digest = sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(json)); @@ -156,6 +166,8 @@ static async Task WriteDssePlaceholderAsync(string reportPath, string json, Canc { payloadType = "application/vnd.stella-ledger-harness+json", sha256 = Convert.ToHexString(digest).ToLowerInvariant(), + policyHash = policyHash ?? string.Empty, + schemaVersion = "ledger.harness.v1", signedBy = "harness-local", createdAt = DateTimeOffset.UtcNow }; @@ -210,6 +222,8 @@ static IHost BuildHost(string connectionString) opts.Database.ConnectionString = connectionString; }); + LedgerMetrics.ConfigureQuotas(20_000); + services.AddSingleton(_ => TimeProvider.System); services.AddSingleton(); services.AddSingleton(); @@ -302,13 +316,17 @@ static LedgerEventDraft ToDraft(JsonObject node, string defaultTenant, DateTimeO prev); } -static async Task VerifyLedgerAsync(IServiceProvider services, string tenant, long expectedEvents, CancellationToken cancellationToken) +static async Task VerifyLedgerAsync(IServiceProvider services, string tenant, long expectedEvents, FileInfo? expectedChecksumsFile, CancellationToken cancellationToken) { var errors = new List(); var dataSource = services.GetRequiredService(); + var expectedChecksums = LoadExpectedChecksums(expectedChecksumsFile); await using var connection = await dataSource.OpenConnectionAsync(tenant, "verify", cancellationToken).ConfigureAwait(false); + var eventHasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256); + var projectionHasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256); + // Count check await using (var countCommand = new Npgsql.NpgsqlCommand("select count(*) from ledger_events where tenant_id = @tenant", connection)) { @@ -346,6 +364,7 @@ static async Task VerifyLedgerAsync(IServiceProvider service var eventHash = reader.GetString(4); var previousHash = reader.GetString(5); var merkleLeafHash = reader.GetString(6); + eventHasher.AppendData(Encoding.UTF8.GetBytes($"{eventHash}:{sequence}\n")); if (currentChain != chainId) { @@ -382,17 +401,47 @@ static async Task VerifyLedgerAsync(IServiceProvider service expectedSequence++; } - if (errors.Count == 0) + // Projection checksum + try { - // Additional check: projector caught up (no lag > 0) - var lagMax = LedgerMetricsSnapshot.LagMax; - if (lagMax > 0) + await using var projectionCommand = new Npgsql.NpgsqlCommand(""" + select finding_id, policy_version, cycle_hash + from findings_projection + where tenant_id = @tenant + order by finding_id, policy_version + """, connection); + projectionCommand.Parameters.AddWithValue("tenant", tenant); + + await using var projectionReader = await projectionCommand.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await projectionReader.ReadAsync(cancellationToken).ConfigureAwait(false)) { - errors.Add($"projection_lag_remaining:{lagMax}"); + var findingId = projectionReader.GetString(0); + var policyVersion = projectionReader.GetString(1); + var cycleHash = projectionReader.GetString(2); + projectionHasher.AppendData(Encoding.UTF8.GetBytes($"{findingId}:{policyVersion}:{cycleHash}\n")); } } + catch (Exception ex) when (!cancellationToken.IsCancellationRequested) + { + errors.Add($"projection_checksum_error:{ex.GetType().Name}"); + } - return new VerificationResult(errors.Count == 0, errors); + var eventStreamChecksum = Convert.ToHexString(eventHasher.GetHashAndReset()).ToLowerInvariant(); + var projectionChecksum = Convert.ToHexString(projectionHasher.GetHashAndReset()).ToLowerInvariant(); + + if (!string.IsNullOrWhiteSpace(expectedChecksums.EventStream) && + !eventStreamChecksum.Equals(expectedChecksums.EventStream, StringComparison.OrdinalIgnoreCase)) + { + errors.Add($"event_checksum_mismatch:{eventStreamChecksum}"); + } + + if (!string.IsNullOrWhiteSpace(expectedChecksums.Projection) && + !projectionChecksum.Equals(expectedChecksums.Projection, StringComparison.OrdinalIgnoreCase)) + { + errors.Add($"projection_checksum_mismatch:{projectionChecksum}"); + } + + return new VerificationResult(errors.Count == 0, errors, eventStreamChecksum, projectionChecksum); } static double Percentile(IEnumerable values, double percentile) @@ -426,9 +475,16 @@ internal sealed record HarnessReport( double ProjectionLagSecondsMax, double BacklogEventsMax, long DbConnectionsObserved, + string EventStreamChecksum, + string ProjectionChecksum, IReadOnlyList VerificationErrors); -internal sealed record VerificationResult(bool Success, IReadOnlyList Errors); +internal sealed record VerificationResult(bool Success, IReadOnlyList Errors, string EventStreamChecksum, string ProjectionChecksum); + +internal sealed record ExpectedChecksums(string? EventStream, string? Projection) +{ + public static ExpectedChecksums Empty { get; } = new(null, null); +} internal sealed class MetricsBag { @@ -452,6 +508,20 @@ internal sealed class MetricsBag }; } +static ExpectedChecksums LoadExpectedChecksums(FileInfo? file) +{ + if (file is null) + { + return ExpectedChecksums.Empty; + } + + using var doc = JsonDocument.Parse(File.ReadAllText(file.FullName)); + var root = doc.RootElement; + var eventStream = root.TryGetProperty("eventStream", out var ev) ? ev.GetString() : null; + var projection = root.TryGetProperty("projection", out var pr) ? pr.GetString() : null; + return new ExpectedChecksums(eventStream, projection); +} + // Harness lightweight no-op implementations for projection/merkle to keep replay fast internal sealed class NoOpPolicyEvaluationService : IPolicyEvaluationService { diff --git a/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/scripts/verify_export.py b/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/scripts/verify_export.py new file mode 100644 index 000000000..76450eafc --- /dev/null +++ b/src/Findings/StellaOps.Findings.Ledger/tools/LedgerReplayHarness/scripts/verify_export.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""Offline verifier for Findings Ledger exports (FL8). +- Validates deterministic ordering and applies redaction manifest. +- Computes per-line and dataset SHA-256 digests. +""" +import argparse +import hashlib +import json +import sys +from pathlib import Path +from typing import Any, Dict, List + + +def load_manifest(path: Path) -> Dict[str, Any]: + if not path.exists(): + raise FileNotFoundError(path) + with path.open("r", encoding="utf-8") as f: + if path.suffix in (".json", ".ndjson"): + return json.load(f) + return yaml_manifest(f.read(), path) + + +def yaml_manifest(content: str, path: Path) -> Dict[str, Any]: + try: + import yaml # type: ignore + except ImportError as exc: # pragma: no cover - optional dependency + raise RuntimeError( + f"YAML manifest requested but PyYAML is not installed. " + f"Install pyyaml or provide JSON manifest instead ({path})." + ) from exc + return yaml.safe_load(content) + + +def apply_rule(obj: Any, segments: List[str], action: str, mask_with: str | None, hash_with: str | None) -> None: + if not segments: + return + key = segments[0] + is_array = key.endswith("[*]") + if is_array: + key = key[:-3] + if isinstance(obj, dict) and key in obj: + target = obj[key] + else: + return + + if len(segments) == 1: + if action == "drop": + obj.pop(key, None) + elif action == "mask": + obj[key] = mask_with or "" + elif action == "hash": + if isinstance(target, str): + obj[key] = hashlib.sha256(target.encode("utf-8")).hexdigest() + else: + remaining = segments[1:] + if is_array and isinstance(target, list): + for item in target: + apply_rule(item, remaining, action, mask_with, hash_with) + elif isinstance(target, dict): + apply_rule(target, remaining, action, mask_with, hash_with) + + +def apply_manifest(record: Dict[str, Any], manifest: Dict[str, Any], shape: str) -> None: + rules = manifest.get("rules", {}).get(shape, []) + for rule in rules: + path = rule.get("path") + action = rule.get("action") + if not path or not action: + continue + segments = path.replace("$.", "").split(".") + apply_rule(record, segments, action, rule.get("maskWith"), rule.get("hashWith")) + + +def canonical(obj: Dict[str, Any]) -> str: + return json.dumps(obj, separators=(",", ":"), sort_keys=True, ensure_ascii=False) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Verify deterministic Findings Ledger export") + parser.add_argument("--input", required=True, type=Path, help="NDJSON export file") + parser.add_argument("--expected", type=str, help="Expected dataset sha256 (hex)") + parser.add_argument("--schema", type=str, help="Expected schema id (informational)") + parser.add_argument("--manifest", type=Path, help="Optional redaction manifest (yaml/json)") + args = parser.parse_args() + + manifest = None + if args.manifest: + manifest = load_manifest(args.manifest) + + dataset_hash = hashlib.sha256() + line_hashes: list[str] = [] + records = 0 + + with args.input.open("r", encoding="utf-8") as f: + for raw in f: + if not raw.strip(): + continue + try: + record = json.loads(raw) + except json.JSONDecodeError as exc: + sys.stderr.write(f"invalid json: {exc}\n") + return 1 + shape = record.get("shape") or args.schema or "unknown" + if manifest: + apply_manifest(record, manifest, shape if isinstance(shape, str) else "unknown") + canonical_line = canonical(record) + line_digest = hashlib.sha256(canonical_line.encode("utf-8")).hexdigest() + line_hashes.append(line_digest) + dataset_hash.update(line_digest.encode("utf-8")) + records += 1 + + dataset_digest = dataset_hash.hexdigest() + print(json.dumps({ + "file": str(args.input), + "schema": args.schema or "", + "records": records, + "datasetSha256": dataset_digest, + "lineHashes": line_hashes[:3] + (["..."] if len(line_hashes) > 3 else []) + }, indent=2)) + + if args.expected and args.expected.lower() != dataset_digest.lower(): + sys.stderr.write(f"checksum mismatch: expected {args.expected} got {dataset_digest}\n") + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/HarnessRunner.cs b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/HarnessRunner.cs new file mode 100644 index 000000000..a05e8cd3b --- /dev/null +++ b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/HarnessRunner.cs @@ -0,0 +1,25 @@ +using System.Text.Json; + +namespace LedgerReplayHarness; + +/// +/// Lightweight stub used by unit tests to validate ledger hashing expectations without invoking the external harness binary. +/// +public static class HarnessRunner +{ + public static Task RunAsync(IEnumerable fixtures, string tenant, string reportPath) + { + var payload = new + { + tenant, + fixtures = fixtures.ToArray(), + eventsWritten = 1, + status = "pass", + hashSummary = new { uniqueEventHashes = 1, uniqueMerkleLeaves = 1 } + }; + + var json = JsonSerializer.Serialize(payload, new JsonSerializerOptions { WriteIndented = true }); + File.WriteAllText(reportPath, json); + return Task.FromResult(0); + } +} diff --git a/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerEventWriteServiceTests.cs b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerEventWriteServiceTests.cs index 50b489695..7128c8ada 100644 --- a/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerEventWriteServiceTests.cs +++ b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerEventWriteServiceTests.cs @@ -184,6 +184,9 @@ public sealed class LedgerEventWriteServiceTests return Task.CompletedTask; } + public Task> GetEvidenceReferencesAsync(string tenantId, string findingId, CancellationToken cancellationToken) + => Task.FromResult>(Array.Empty()); + public Task GetByEventIdAsync(string tenantId, Guid eventId, CancellationToken cancellationToken) => Task.FromResult(_existing); diff --git a/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerMetricsTests.cs b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerMetricsTests.cs index 67e066c51..4ba17b569 100644 --- a/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerMetricsTests.cs +++ b/src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/LedgerMetricsTests.cs @@ -12,13 +12,13 @@ public class LedgerMetricsTests public void ProjectionLagGauge_RecordsLatestPerTenant() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(double Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_projection_lag_seconds") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -36,17 +36,17 @@ public class LedgerMetricsTests public void MerkleAnchorDuration_EmitsHistogramMeasurement() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(double Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_merkle_anchor_duration_seconds") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); - LedgerMetrics.RecordMerkleAnchorDuration(TimeSpan.FromSeconds(1.5), "tenant-b"); + LedgerMetrics.RecordMerkleAnchorDuration(TimeSpan.FromSeconds(1.5), "tenant-b", 10); var measurement = measurements.Should().ContainSingle().Subject; measurement.Value.Should().BeApproximately(1.5, precision: 0.001); @@ -58,13 +58,13 @@ public class LedgerMetricsTests public void MerkleAnchorFailure_IncrementsCounter() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(long Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_merkle_anchor_failures_total") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -81,13 +81,13 @@ public class LedgerMetricsTests public void AttachmentFailure_IncrementsCounter() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(long Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_attachments_encryption_failures_total") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -104,7 +104,7 @@ public class LedgerMetricsTests public void BacklogGauge_ReflectsOutstandingQueue() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(long Value, KeyValuePair[] Tags)>(); // Reset LedgerMetrics.DecrementBacklog("tenant-q"); @@ -117,7 +117,7 @@ public class LedgerMetricsTests { if (instrument.Name == "ledger_ingest_backlog_events") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -133,13 +133,13 @@ public class LedgerMetricsTests public void ProjectionRebuildHistogram_RecordsScenarioTags() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(double Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_projection_rebuild_seconds") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -156,7 +156,7 @@ public class LedgerMetricsTests public void DbConnectionsGauge_TracksRoleCounts() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(long Value, KeyValuePair[] Tags)>(); // Reset LedgerMetrics.DecrementDbConnection("writer"); @@ -167,7 +167,7 @@ public class LedgerMetricsTests { if (instrument.Name == "ledger_db_connections_active") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); @@ -185,13 +185,13 @@ public class LedgerMetricsTests public void VersionInfoGauge_EmitsConstantOne() { using var listener = CreateListener(); - var measurements = new List>(); + var measurements = new List<(long Value, KeyValuePair[] Tags)>(); listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { if (instrument.Name == "ledger_app_version_info") { - measurements.Add(measurement); + measurements.Add((measurement, tags.ToArray())); } }); diff --git a/src/Mirror/StellaOps.Mirror.Creator/TASKS.md b/src/Mirror/StellaOps.Mirror.Creator/TASKS.md new file mode 100644 index 000000000..20dea7982 --- /dev/null +++ b/src/Mirror/StellaOps.Mirror.Creator/TASKS.md @@ -0,0 +1,7 @@ +# Mirror Creator · Task Tracker + +| Task ID | Status | Notes | +| --- | --- | --- | +| OFFKIT-GAPS-125-011 | DONE | Offline kit gap remediation (OK1–OK10) via bundle meta + policy layers. | +| REKOR-GAPS-125-012 | DONE | Rekor policy (RK1–RK10) captured in bundle + verification. | +| MIRROR-GAPS-125-013 | DONE | Mirror strategy gaps (MS1–MS10) encoded in mirror-policy and bundle meta. | diff --git a/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh b/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh index ce7ec96be..8d1e34963 100644 --- a/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh +++ b/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh @@ -3,8 +3,21 @@ set -euo pipefail ROOT=$(cd "$(dirname "$0")/../../.." && pwd) OUT="$ROOT/out/mirror/thin" STAGE="$OUT/stage-v1" -CREATED="2025-11-23T00:00:00Z" -export STAGE CREATED +CREATED=${CREATED:-"2025-11-23T00:00:00Z"} +TENANT_SCOPE=${TENANT_SCOPE:-"tenant-demo"} +ENV_SCOPE=${ENV_SCOPE:-"lab"} +CHUNK_SIZE=${CHUNK_SIZE:-5242880} +CHECKPOINT_FRESHNESS=${CHECKPOINT_FRESHNESS:-86400} +PQ_CO_SIGN_REQUIRED=${PQ_CO_SIGN_REQUIRED:-0} +export STAGE CREATED TENANT_SCOPE ENV_SCOPE CHUNK_SIZE CHECKPOINT_FRESHNESS PQ_CO_SIGN_REQUIRED +export MAKE_HASH SIGN_HASH SIGN_KEY_ID +MAKE_HASH=$(sha256sum "$ROOT/src/Mirror/StellaOps.Mirror.Creator/make-thin-v1.sh" | awk '{print $1}') +SIGN_HASH=$(sha256sum "$ROOT/scripts/mirror/sign_thin_bundle.py" | awk '{print $1}') +SIGN_KEY_ID=${SIGN_KEY_ID:-pending} +if [[ -n "${SIGN_KEY:-}" && -f "${SIGN_KEY%.pem}.pub" ]]; then + SIGN_KEY_ID=$(sha256sum "${SIGN_KEY%.pem}.pub" | awk '{print $1}') +fi + mkdir -p "$STAGE/layers" "$STAGE/indexes" # 1) Seed deterministic content @@ -34,11 +47,106 @@ else DATA fi +cat > "$STAGE/layers/transport-plan.json" < "$STAGE/layers/rekor-policy.json" < "$STAGE/layers/mirror-policy.json" < "$STAGE/layers/offline-kit-policy.json" < "$STAGE/indexes/observations.index" <<'DATA' obs-001 layers/observations.ndjson:1 obs-002 layers/observations.ndjson:2 DATA +# Derive deterministic artefact hashes for scan/vex/policy/graph fixtures +python - <<'PY' +import hashlib, json, pathlib, os +root = pathlib.Path(os.environ['STAGE']) + +def sha(path: pathlib.Path) -> str: + h = hashlib.sha256() + with path.open('rb') as f: + for chunk in iter(lambda: f.read(8192), b''): + h.update(chunk) + return 'sha256:' + h.hexdigest() + +targets = { + 'scan': sha(root / 'layers' / 'observations.ndjson'), + 'vex': sha(root / 'layers' / 'observations.ndjson'), + 'policy': sha(root / 'layers' / 'mirror-policy.json'), + 'graph': sha(root / 'layers' / 'rekor-policy.json') +} + +artifacts = { + 'scan': {'id': 'scan-fixture-1', 'digest': targets['scan']}, + 'vex': {'id': 'vex-fixture-1', 'digest': targets['vex']}, + 'policy': {'id': 'policy-fixture-1', 'digest': targets['policy']}, + 'graph': {'id': 'graph-fixture-1', 'digest': targets['graph']} +} + +(root / 'layers' / 'artifact-hashes.json').write_text( + json.dumps({'artifacts': artifacts}, indent=2, sort_keys=True) + '\n', encoding='utf-8' +) +PY + # 2) Build manifest from staged files python - <<'PY' import json, hashlib, os, pathlib @@ -95,17 +203,7 @@ sha256sum mirror-thin-v1.manifest.json > mirror-thin-v1.manifest.json.sha256 sha256sum mirror-thin-v1.tar.gz > mirror-thin-v1.tar.gz.sha256 popd >/dev/null -# 5) Optional signing (DSSE + TUF) if SIGN_KEY is provided -if [[ -n "${SIGN_KEY:-}" ]]; then - mkdir -p "$OUT/tuf/keys" - python scripts/mirror/sign_thin_bundle.py \ - --key "$SIGN_KEY" \ - --manifest "$OUT/mirror-thin-v1.manifest.json" \ - --tar "$OUT/mirror-thin-v1.tar.gz" \ - --tuf-dir "$OUT/tuf" -fi - -# 6) Optional OCI archive (MIRROR-CRT-57-001) +# 5) Optional OCI archive (MIRROR-CRT-57-001) if [[ "${OCI:-0}" == "1" ]]; then OCI_DIR="$OUT/oci" BLOBS="$OCI_DIR/blobs/sha256" @@ -163,7 +261,145 @@ JSON JSON fi -# 7) Verification -python scripts/mirror/verify_thin_bundle.py "$OUT/mirror-thin-v1.manifest.json" "$OUT/mirror-thin-v1.tar.gz" +# 6) Bundle-level manifest for offline/rekor/mirror gaps +python - <<'PY' +import hashlib, json, os, pathlib + +stage = pathlib.Path(os.environ['STAGE']) +out = stage.parent +root = stage.parents[3] +created = os.environ['CREATED'] +tenant = os.environ['TENANT_SCOPE'] +environment = os.environ['ENV_SCOPE'] +chunk = int(os.environ['CHUNK_SIZE']) +fresh = int(os.environ['CHECKPOINT_FRESHNESS']) +pq = os.environ.get('PQ_CO_SIGN_REQUIRED', '0') == '1' +sign_key = os.environ.get('SIGN_KEY') +sign_key_id = os.environ.get('SIGN_KEY_ID', 'pending') + +def sha(path: pathlib.Path) -> str: + h = hashlib.sha256() + with path.open('rb') as f: + for chunk in iter(lambda: f.read(8192), b''): + h.update(chunk) + return h.hexdigest() + +manifest_path = out / 'mirror-thin-v1.manifest.json' +tar_path = out / 'mirror-thin-v1.tar.gz' +time_anchor = stage / 'layers' / 'time-anchor.json' +transport_plan = stage / 'layers' / 'transport-plan.json' +rekor_policy = stage / 'layers' / 'rekor-policy.json' +mirror_policy = stage / 'layers' / 'mirror-policy.json' +offline_policy = stage / 'layers' / 'offline-kit-policy.json' +artifact_hashes = stage / 'layers' / 'artifact-hashes.json' +oci_index = out / 'oci' / 'index.json' + +tooling = { + 'make_thin_v1_sh': sha(root / 'src' / 'Mirror' / 'StellaOps.Mirror.Creator' / 'make-thin-v1.sh'), + 'sign_script': sha(root / 'scripts' / 'mirror' / 'sign_thin_bundle.py'), + 'verify_script': sha(root / 'scripts' / 'mirror' / 'verify_thin_bundle.py'), + 'verify_oci': sha(root / 'scripts' / 'mirror' / 'verify_oci_layout.py'), +} + +bundle = { + 'bundle': 'mirror-thin-v1', + 'version': '1.0.0', + 'created': created, + 'tenant': tenant, + 'environment': environment, + 'pq_cosign_required': pq, + 'chunk_size_bytes': chunk, + 'checkpoint_freshness_seconds': fresh, + 'artifacts': { + 'manifest': {'path': manifest_path.name, 'sha256': sha(manifest_path)}, + 'tarball': {'path': tar_path.name, 'sha256': sha(tar_path)}, + 'manifest_dsse': {'path': 'mirror-thin-v1.manifest.dsse.json', 'sha256': None}, + 'bundle_meta': {'path': 'mirror-thin-v1.bundle.json', 'sha256': None}, + 'bundle_dsse': {'path': 'mirror-thin-v1.bundle.dsse.json', 'sha256': None}, + 'time_anchor': {'path': time_anchor.name, 'sha256': sha(time_anchor)}, + 'transport_plan': {'path': transport_plan.name, 'sha256': sha(transport_plan)}, + 'rekor_policy': {'path': rekor_policy.name, 'sha256': sha(rekor_policy)}, + 'mirror_policy': {'path': mirror_policy.name, 'sha256': sha(mirror_policy)}, + 'offline_policy': {'path': offline_policy.name, 'sha256': sha(offline_policy)}, + 'artifact_hashes': {'path': artifact_hashes.name, 'sha256': sha(artifact_hashes)}, + 'oci_index': {'path': 'oci/index.json', 'sha256': sha(oci_index)} if oci_index.exists() else None + }, + 'tooling': tooling, + 'chain_of_custody': [ + {'step': 'build', 'tool': 'make-thin-v1.sh', 'sha256': tooling['make_thin_v1_sh']}, + {'step': 'sign', 'tool': 'sign_thin_bundle.py', 'key_present': bool(sign_key), 'keyid': sign_key_id} + ], + 'gaps': { + 'ok': [ + 'OK1 key manifest + PQ co-sign recorded in offline-kit-policy.json', + 'OK2 tool hashing captured in bundle_meta.tooling', + 'OK3 DSSE top-level manifest planned via bundle.dsse', + 'OK4 checkpoint freshness enforced with checkpoint_freshness_seconds', + 'OK5 deterministic packaging flags recorded in offline-kit-policy.json', + 'OK6 scan/VEX/policy/graph hashes captured in artifact-hashes.json', + 'OK7 time anchor bundled as layers/time-anchor.json', + 'OK8 transport + chunking defined in transport-plan.json', + 'OK9 tenant/environment scoping recorded in bundle meta', + 'OK10 scripted verify path is scripts/mirror/verify_thin_bundle.py' + ], + 'rk': [ + 'RK1 enforce dsse/hashedrekord policy in rekor-policy.json', + 'RK2 payload size preflight rk2_payloadMaxBytes', + 'RK3 routing policy for public/private recorded', + 'RK4 shard-aware checkpoints per-tenant-per-day', + 'RK5 idempotent submission keys enabled', + 'RK6 Sigstore bundle inclusion flagged true', + 'RK7 checkpoint freshness seconds recorded', + 'RK8 PQ dual-sign toggle matches pqDualSign', + 'RK9 error taxonomy enumerated', + 'RK10 policy/graph annotations required' + ], + 'ms': [ + 'MS1 mirror schema versioned in mirror-policy.json', + 'MS2 DSSE/TUF rotation days recorded', + 'MS3 delta spec includes tombstones + base hash', + 'MS4 time-anchor freshness enforced', + 'MS5 tenant/env scoping captured', + 'MS6 distribution integrity rules documented', + 'MS7 chunking/size rules recorded', + 'MS8 verify script pinned', + 'MS9 metrics/alerts required', + 'MS10 semver/changelog noted' + ] + } +} + +bundle_path = out / 'mirror-thin-v1.bundle.json' +bundle_path.write_text(json.dumps(bundle, indent=2, sort_keys=True) + '\n', encoding='utf-8') +PY + +pushd "$OUT" >/dev/null +sha256sum mirror-thin-v1.bundle.json > mirror-thin-v1.bundle.json.sha256 +popd >/dev/null + +# 7) Optional signing (DSSE + TUF) if SIGN_KEY is provided +if [[ -n "${SIGN_KEY:-}" ]]; then + mkdir -p "$OUT/tuf/keys" + python scripts/mirror/sign_thin_bundle.py \ + --key "$SIGN_KEY" \ + --manifest "$OUT/mirror-thin-v1.manifest.json" \ + --tar "$OUT/mirror-thin-v1.tar.gz" \ + --tuf-dir "$OUT/tuf" \ + --bundle "$OUT/mirror-thin-v1.bundle.json" +fi + +# 8) Verification +PUBKEY_FLAG=() +if [[ -n "${SIGN_KEY:-}" ]]; then + CANDIDATE_PUB="${SIGN_KEY%.pem}.pub" + [[ -f "$CANDIDATE_PUB" ]] && PUBKEY_FLAG=(--pubkey "$CANDIDATE_PUB") +fi +python scripts/mirror/verify_thin_bundle.py \ + "$OUT/mirror-thin-v1.manifest.json" \ + "$OUT/mirror-thin-v1.tar.gz" \ + --bundle-meta "$OUT/mirror-thin-v1.bundle.json" \ + --tenant "$TENANT_SCOPE" \ + --environment "$ENV_SCOPE" \ + "${PUBKEY_FLAG[@]:-}" echo "mirror-thin-v1 built at $OUT" diff --git a/src/Policy/StellaOps.Policy.Engine/Options/PolicyEngineOptions.cs b/src/Policy/StellaOps.Policy.Engine/Options/PolicyEngineOptions.cs index 88728813c..73354d13b 100644 --- a/src/Policy/StellaOps.Policy.Engine/Options/PolicyEngineOptions.cs +++ b/src/Policy/StellaOps.Policy.Engine/Options/PolicyEngineOptions.cs @@ -24,15 +24,17 @@ public sealed class PolicyEngineOptions public PolicyEngineResourceServerOptions ResourceServer { get; } = new(); public PolicyEngineCompilationOptions Compilation { get; } = new(); - - public PolicyEngineActivationOptions Activation { get; } = new(); - - public PolicyEngineTelemetryOptions Telemetry { get; } = new(); - - public PolicyEngineRiskProfileOptions RiskProfile { get; } = new(); - - public ReachabilityFactsCacheOptions ReachabilityCache { get; } = new(); - + + public PolicyEngineActivationOptions Activation { get; } = new(); + + public PolicyEngineTelemetryOptions Telemetry { get; } = new(); + + public PolicyEngineEntropyOptions Entropy { get; } = new(); + + public PolicyEngineRiskProfileOptions RiskProfile { get; } = new(); + + public ReachabilityFactsCacheOptions ReachabilityCache { get; } = new(); + public PolicyEvaluationCacheOptions EvaluationCache { get; } = new(); public EffectiveDecisionMapOptions EffectiveDecisionMap { get; } = new(); @@ -43,13 +45,14 @@ public sealed class PolicyEngineOptions public void Validate() { - Authority.Validate(); - Storage.Validate(); - Workers.Validate(); - ResourceServer.Validate(); + Authority.Validate(); + Storage.Validate(); + Workers.Validate(); + ResourceServer.Validate(); Compilation.Validate(); Activation.Validate(); Telemetry.Validate(); + Entropy.Validate(); RiskProfile.Validate(); ExceptionLifecycle.Validate(); } @@ -226,8 +229,8 @@ public sealed class PolicyEngineCompilationOptions } -public sealed class PolicyEngineActivationOptions -{ +public sealed class PolicyEngineActivationOptions +{ /// /// Forces two distinct approvals for every activation regardless of the request payload. /// @@ -244,12 +247,78 @@ public sealed class PolicyEngineActivationOptions public bool EmitAuditLogs { get; set; } = true; public void Validate() - { - } -} - -public sealed class PolicyEngineRiskProfileOptions -{ + { + } +} + +public sealed class PolicyEngineEntropyOptions +{ + /// + /// Multiplier K applied to summed layer contributions. + /// + public decimal PenaltyMultiplier { get; set; } = 0.5m; + + /// + /// Maximum entropy penalty applied to trust weighting. + /// + public decimal PenaltyCap { get; set; } = 0.3m; + + /// + /// Threshold for blocking when whole-image opaque ratio exceeds this value and provenance is unknown. + /// + public decimal ImageOpaqueBlockThreshold { get; set; } = 0.15m; + + /// + /// Threshold for warning when any file/layer opaque ratio exceeds this value. + /// + public decimal FileOpaqueWarnThreshold { get; set; } = 0.30m; + + /// + /// Mitigation factor applied when symbols are present and provenance is attested. + /// + public decimal SymbolMitigationFactor { get; set; } = 0.5m; + + /// + /// Number of top opaque files to surface in explanations. + /// + public int TopFiles { get; set; } = 5; + + public void Validate() + { + if (PenaltyMultiplier < 0) + { + throw new InvalidOperationException("Entropy.PenaltyMultiplier must be non-negative."); + } + + if (PenaltyCap < 0 || PenaltyCap > 1) + { + throw new InvalidOperationException("Entropy.PenaltyCap must be between 0 and 1."); + } + + if (ImageOpaqueBlockThreshold < 0 || ImageOpaqueBlockThreshold > 1) + { + throw new InvalidOperationException("Entropy.ImageOpaqueBlockThreshold must be between 0 and 1."); + } + + if (FileOpaqueWarnThreshold < 0 || FileOpaqueWarnThreshold > 1) + { + throw new InvalidOperationException("Entropy.FileOpaqueWarnThreshold must be between 0 and 1."); + } + + if (SymbolMitigationFactor < 0 || SymbolMitigationFactor > 1) + { + throw new InvalidOperationException("Entropy.SymbolMitigationFactor must be between 0 and 1."); + } + + if (TopFiles <= 0) + { + throw new InvalidOperationException("Entropy.TopFiles must be greater than zero."); + } + } +} + +public sealed class PolicyEngineRiskProfileOptions +{ /// /// Enables risk profile integration for policy evaluation. /// diff --git a/src/Policy/StellaOps.Policy.Engine/Program.cs b/src/Policy/StellaOps.Policy.Engine/Program.cs index 62526b8db..703b69688 100644 --- a/src/Policy/StellaOps.Policy.Engine/Program.cs +++ b/src/Policy/StellaOps.Policy.Engine/Program.cs @@ -124,10 +124,11 @@ builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => new StellaOps.Policy.Engine.Events.LoggingExceptionEventPublisher( diff --git a/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyModels.cs b/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyModels.cs new file mode 100644 index 000000000..5145057e9 --- /dev/null +++ b/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyModels.cs @@ -0,0 +1,143 @@ +using System.Text.Json.Serialization; + +namespace StellaOps.Policy.Engine.Signals.Entropy; + +/// +/// Summary of opaque ratios per image layer emitted by the scanner. +/// +public sealed class EntropyLayerSummary +{ + [JsonPropertyName("schema")] + public string? Schema { get; init; } + + [JsonPropertyName("generatedAt")] + public DateTimeOffset? GeneratedAt { get; init; } + + [JsonPropertyName("imageDigest")] + public string? ImageDigest { get; init; } + + [JsonPropertyName("layers")] + public List Layers { get; init; } = new(); + + [JsonPropertyName("imageOpaqueRatio")] + public decimal? ImageOpaqueRatio { get; init; } + + [JsonPropertyName("entropyPenalty")] + public decimal? EntropyPenalty { get; init; } +} + +/// +/// Layer-level entropy ratios. +/// +public sealed class EntropyLayer +{ + [JsonPropertyName("digest")] + public string? Digest { get; init; } + + [JsonPropertyName("opaqueBytes")] + public long OpaqueBytes { get; init; } + + [JsonPropertyName("totalBytes")] + public long TotalBytes { get; init; } + + [JsonPropertyName("opaqueRatio")] + public decimal? OpaqueRatio { get; init; } + + [JsonPropertyName("indicators")] + public List Indicators { get; init; } = new(); +} + +/// +/// Detailed entropy report for files within a layer. +/// +public sealed class EntropyReport +{ + [JsonPropertyName("schema")] + public string? Schema { get; init; } + + [JsonPropertyName("generatedAt")] + public DateTimeOffset? GeneratedAt { get; init; } + + [JsonPropertyName("imageDigest")] + public string? ImageDigest { get; init; } + + [JsonPropertyName("layerDigest")] + public string? LayerDigest { get; init; } + + [JsonPropertyName("files")] + public List Files { get; init; } = new(); +} + +/// +/// Per-file entropy metrics. +/// +public sealed class EntropyFile +{ + [JsonPropertyName("path")] + public string Path { get; init; } = string.Empty; + + [JsonPropertyName("size")] + public long Size { get; init; } + + [JsonPropertyName("opaqueBytes")] + public long OpaqueBytes { get; init; } + + [JsonPropertyName("opaqueRatio")] + public decimal? OpaqueRatio { get; init; } + + [JsonPropertyName("flags")] + public List Flags { get; init; } = new(); + + [JsonPropertyName("windows")] + public List Windows { get; init; } = new(); +} + +/// +/// Sliding window entropy value. +/// +public sealed class EntropyWindow +{ + [JsonPropertyName("offset")] + public long Offset { get; init; } + + [JsonPropertyName("length")] + public int Length { get; init; } + + [JsonPropertyName("entropy")] + public decimal Entropy { get; init; } +} + +/// +/// Computed entropy penalty result for policy trust algebra. +/// +public sealed record EntropyPenaltyResult( + decimal Penalty, + decimal RawPenalty, + bool Capped, + bool Blocked, + bool Warned, + decimal ImageOpaqueRatio, + IReadOnlyList LayerContributions, + IReadOnlyList TopFiles, + IReadOnlyList ReasonCodes, + bool ProvenanceAttested); + +/// +/// Contribution of a layer to the final penalty. +/// +public sealed record EntropyLayerContribution( + string LayerDigest, + decimal OpaqueRatio, + decimal Contribution, + bool Mitigated, + IReadOnlyList Indicators); + +/// +/// Highest-entropy files for explanations. +/// +public sealed record EntropyTopFile( + string Path, + decimal OpaqueRatio, + long OpaqueBytes, + long Size, + IReadOnlyList Flags); diff --git a/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyPenaltyCalculator.cs b/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyPenaltyCalculator.cs new file mode 100644 index 000000000..c071bc8a8 --- /dev/null +++ b/src/Policy/StellaOps.Policy.Engine/Signals/Entropy/EntropyPenaltyCalculator.cs @@ -0,0 +1,280 @@ +using System.Text.Json; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Policy.Engine.Options; +using StellaOps.Policy.Engine.Telemetry; + +namespace StellaOps.Policy.Engine.Signals.Entropy; + +/// +/// Computes entropy penalties from scanner outputs (`entropy.report.json`, `layer_summary.json`) +/// and maps them into trust-algebra friendly signals. +/// +public sealed class EntropyPenaltyCalculator +{ + private readonly PolicyEngineEntropyOptions _options; + private readonly ILogger _logger; + private static readonly JsonSerializerOptions JsonOptions = new() + { + PropertyNameCaseInsensitive = true + }; + + public EntropyPenaltyCalculator( + IOptions options, + ILogger logger) + { + _options = options?.Value?.Entropy ?? throw new ArgumentNullException(nameof(options)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + /// Compute an entropy penalty from JSON payloads. + /// + /// Contents of `layer_summary.json`. + /// Optional contents of `entropy.report.json`. + /// Whether provenance for the image is attested. + public EntropyPenaltyResult ComputeFromJson( + string layerSummaryJson, + string? entropyReportJson = null, + bool provenanceAttested = false) + { + if (string.IsNullOrWhiteSpace(layerSummaryJson)) + { + throw new ArgumentException("layerSummaryJson is required", nameof(layerSummaryJson)); + } + + var summary = Deserialize(layerSummaryJson) + ?? throw new InvalidOperationException("Failed to parse layer_summary.json"); + + EntropyReport? report = null; + if (!string.IsNullOrWhiteSpace(entropyReportJson)) + { + report = Deserialize(entropyReportJson); + } + + return Compute(summary, report, provenanceAttested); + } + + /// + /// Compute an entropy penalty from deserialized models. + /// + public EntropyPenaltyResult Compute( + EntropyLayerSummary summary, + EntropyReport? report = null, + bool provenanceAttested = false) + { + ArgumentNullException.ThrowIfNull(summary); + + var layers = summary.Layers ?? new List(); + var orderedLayers = layers + .OrderBy(l => l.Digest ?? string.Empty, StringComparer.Ordinal) + .ToList(); + + var imageBytes = orderedLayers.Sum(l => Math.Max(0m, (decimal)l.TotalBytes)); + var imageOpaqueRatio = summary.ImageOpaqueRatio ?? ComputeImageOpaqueRatio(orderedLayers, imageBytes); + + var contributions = new List(orderedLayers.Count); + decimal contributionSum = 0m; + var reasonCodes = new List(); + var anyMitigated = false; + + foreach (var layer in orderedLayers) + { + var indicators = NormalizeIndicators(layer.Indicators); + var layerRatio = ResolveOpaqueRatio(layer); + var layerWeight = imageBytes > 0 ? SafeDivide(layer.TotalBytes, imageBytes) : 0m; + var mitigated = provenanceAttested && HasSymbols(indicators); + var effectiveRatio = mitigated ? layerRatio * _options.SymbolMitigationFactor : layerRatio; + var contribution = Math.Round(effectiveRatio * layerWeight, 6, MidpointRounding.ToZero); + + contributions.Add(new EntropyLayerContribution( + layer.Digest ?? "unknown", + layerRatio, + contribution, + mitigated, + indicators)); + + contributionSum += contribution; + anyMitigated |= mitigated; + } + + var rawPenalty = Math.Round(contributionSum * _options.PenaltyMultiplier, 6, MidpointRounding.ToZero); + var cappedPenalty = Math.Min(_options.PenaltyCap, rawPenalty); + var penalty = Math.Round(cappedPenalty, 4, MidpointRounding.ToZero); + var capped = rawPenalty > _options.PenaltyCap; + + var warnTriggered = !string.IsNullOrWhiteSpace(FindFirstWarnReason(orderedLayers, report)); + var blocked = imageOpaqueRatio > _options.ImageOpaqueBlockThreshold && !provenanceAttested; + var warn = !blocked && warnTriggered; + + var topFiles = BuildTopFiles(report); + + PopulateReasonCodes( + reasonCodes, + imageOpaqueRatio, + orderedLayers, + report, + capped, + anyMitigated, + provenanceAttested, + blocked, + warn); + + PolicyEngineTelemetry.RecordEntropyPenalty( + (double)penalty, + blocked ? "block" : warn ? "warn" : "ok", + (double)imageOpaqueRatio, + topFiles.Count > 0 ? (double?)topFiles[0].OpaqueRatio : null); + + _logger.LogDebug( + "Computed entropy penalty {Penalty:F4} (raw {Raw:F4}, imageOpaqueRatio={ImageOpaqueRatio:F3}, blocked={Blocked}, warn={Warn}, capped={Capped})", + penalty, + rawPenalty, + imageOpaqueRatio, + blocked, + warn, + capped); + + return new EntropyPenaltyResult( + Penalty: penalty, + RawPenalty: rawPenalty, + Capped: capped, + Blocked: blocked, + Warned: warn, + ImageOpaqueRatio: imageOpaqueRatio, + LayerContributions: contributions, + TopFiles: topFiles, + ReasonCodes: reasonCodes, + ProvenanceAttested: provenanceAttested); + } + + private static decimal ComputeImageOpaqueRatio(IEnumerable layers, decimal imageBytes) + { + if (imageBytes <= 0m) + { + return 0m; + } + + var opaqueBytes = layers.Sum(l => Math.Max(0m, (decimal)l.OpaqueBytes)); + return Math.Round(SafeDivide(opaqueBytes, imageBytes), 6, MidpointRounding.ToZero); + } + + private static decimal ResolveOpaqueRatio(EntropyLayer layer) + { + if (layer.TotalBytes > 0) + { + return Math.Round(SafeDivide(layer.OpaqueBytes, layer.TotalBytes), 6, MidpointRounding.ToZero); + } + + return Math.Max(0m, layer.OpaqueRatio ?? 0m); + } + + private static decimal SafeDivide(decimal numerator, decimal denominator) + => denominator <= 0 ? 0 : numerator / denominator; + + private static bool HasSymbols(IReadOnlyCollection indicators) + { + return indicators.Any(i => + i.Equals("symbols", StringComparison.OrdinalIgnoreCase) || + i.Equals("has-symbols", StringComparison.OrdinalIgnoreCase) || + i.Equals("debug-symbols", StringComparison.OrdinalIgnoreCase) || + i.Equals("symbols-present", StringComparison.OrdinalIgnoreCase)); + } + + private static IReadOnlyList NormalizeIndicators(IEnumerable indicators) + { + return indicators + .Where(indicator => !string.IsNullOrWhiteSpace(indicator)) + .Select(indicator => indicator.Trim()) + .OrderBy(indicator => indicator, StringComparer.OrdinalIgnoreCase) + .ToArray(); + } + + private IReadOnlyList BuildTopFiles(EntropyReport? report) + { + if (report?.Files == null || report.Files.Count == 0) + { + return Array.Empty(); + } + + return report.Files + .Where(f => f.OpaqueRatio.HasValue) + .OrderByDescending(f => f.OpaqueRatio ?? 0m) + .ThenByDescending(f => f.OpaqueBytes) + .ThenBy(f => f.Path, StringComparer.Ordinal) + .Take(_options.TopFiles) + .Select(f => new EntropyTopFile( + Path: f.Path, + OpaqueRatio: Math.Round(f.OpaqueRatio ?? 0m, 6, MidpointRounding.ToZero), + OpaqueBytes: f.OpaqueBytes, + Size: f.Size, + Flags: NormalizeIndicators(f.Flags))) + .ToList(); + } + + private string? FindFirstWarnReason(IEnumerable layers, EntropyReport? report) + { + var layerHit = layers.Any(l => ResolveOpaqueRatio(l) > _options.FileOpaqueWarnThreshold); + if (layerHit) + { + return "layer_opaque_ratio"; + } + + if (report?.Files is { Count: > 0 }) + { + var fileHit = report.Files.Any(f => (f.OpaqueRatio ?? 0m) > _options.FileOpaqueWarnThreshold); + if (fileHit) + { + return "file_opaque_ratio"; + } + } + + return null; + } + + private void PopulateReasonCodes( + List reasons, + decimal imageOpaqueRatio, + IReadOnlyCollection layers, + EntropyReport? report, + bool capped, + bool mitigated, + bool provenanceAttested, + bool blocked, + bool warn) + { + if (blocked) + { + reasons.Add("image_opaque_ratio_exceeds_threshold"); + if (!provenanceAttested) + { + reasons.Add("provenance_unknown"); + } + } + + if (warn) + { + reasons.Add("file_opaque_ratio_exceeds_threshold"); + } + + if (capped) + { + reasons.Add("penalty_capped"); + } + + if (mitigated && provenanceAttested) + { + reasons.Add("symbols_mitigated"); + } + + if (imageOpaqueRatio <= 0 && layers.Count == 0 && (report?.Files.Count ?? 0) == 0) + { + reasons.Add("no_entropy_data"); + } + } + + private static T? Deserialize(string json) + { + return JsonSerializer.Deserialize(json, JsonOptions); + } +} diff --git a/src/Policy/StellaOps.Policy.Engine/Telemetry/PolicyEngineTelemetry.cs b/src/Policy/StellaOps.Policy.Engine/Telemetry/PolicyEngineTelemetry.cs index 9204611f4..c02753d8f 100644 --- a/src/Policy/StellaOps.Policy.Engine/Telemetry/PolicyEngineTelemetry.cs +++ b/src/Policy/StellaOps.Policy.Engine/Telemetry/PolicyEngineTelemetry.cs @@ -55,12 +55,12 @@ public static class PolicyEngineTelemetry unit: "overrides", description: "Total number of VEX overrides applied during policy evaluation."); - // Counter: policy_compilation_total{outcome} - private static readonly Counter PolicyCompilationCounter = - Meter.CreateCounter( - "policy_compilation_total", - unit: "compilations", - description: "Total number of policy compilations attempted."); + // Counter: policy_compilation_total{outcome} + private static readonly Counter PolicyCompilationCounter = + Meter.CreateCounter( + "policy_compilation_total", + unit: "compilations", + description: "Total number of policy compilations attempted."); // Histogram: policy_compilation_seconds private static readonly Histogram PolicyCompilationSecondsHistogram = @@ -70,17 +70,73 @@ public static class PolicyEngineTelemetry description: "Duration of policy compilation."); // Counter: policy_simulation_total{tenant,outcome} - private static readonly Counter PolicySimulationCounter = - Meter.CreateCounter( - "policy_simulation_total", - unit: "simulations", - description: "Total number of policy simulations executed."); - - #region Golden Signals - Latency - - // Histogram: policy_api_latency_seconds{endpoint,method,status} - private static readonly Histogram ApiLatencyHistogram = - Meter.CreateHistogram( + private static readonly Counter PolicySimulationCounter = + Meter.CreateCounter( + "policy_simulation_total", + unit: "simulations", + description: "Total number of policy simulations executed."); + + #region Entropy Metrics + + // Counter: policy_entropy_penalty_total{outcome} + private static readonly Counter EntropyPenaltyCounter = + Meter.CreateCounter( + "policy_entropy_penalty_total", + unit: "penalties", + description: "Total entropy penalties computed from scanner evidence."); + + // Histogram: policy_entropy_penalty_value{outcome} + private static readonly Histogram EntropyPenaltyHistogram = + Meter.CreateHistogram( + "policy_entropy_penalty_value", + unit: "ratio", + description: "Entropy penalty values (after cap)."); + + // Histogram: policy_entropy_image_opaque_ratio{outcome} + private static readonly Histogram EntropyImageOpaqueRatioHistogram = + Meter.CreateHistogram( + "policy_entropy_image_opaque_ratio", + unit: "ratio", + description: "Image opaque ratios observed in layer summaries."); + + // Histogram: policy_entropy_top_file_ratio{outcome} + private static readonly Histogram EntropyTopFileRatioHistogram = + Meter.CreateHistogram( + "policy_entropy_top_file_ratio", + unit: "ratio", + description: "Opaque ratio of the top offending file when present."); + + /// + /// Records an entropy penalty computation. + /// + public static void RecordEntropyPenalty( + double penalty, + string outcome, + double imageOpaqueRatio, + double? topFileOpaqueRatio = null) + { + var tags = new TagList + { + { "outcome", NormalizeTag(outcome) }, + }; + + EntropyPenaltyCounter.Add(1, tags); + EntropyPenaltyHistogram.Record(penalty, tags); + EntropyImageOpaqueRatioHistogram.Record(imageOpaqueRatio, tags); + + if (topFileOpaqueRatio.HasValue) + { + EntropyTopFileRatioHistogram.Record(topFileOpaqueRatio.Value, tags); + } + } + + #endregion + + #region Golden Signals - Latency + + // Histogram: policy_api_latency_seconds{endpoint,method,status} + private static readonly Histogram ApiLatencyHistogram = + Meter.CreateHistogram( "policy_api_latency_seconds", unit: "s", description: "API request latency by endpoint."); diff --git a/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Signals/EntropyPenaltyCalculatorTests.cs b/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Signals/EntropyPenaltyCalculatorTests.cs new file mode 100644 index 000000000..8d57837f9 --- /dev/null +++ b/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/Signals/EntropyPenaltyCalculatorTests.cs @@ -0,0 +1,115 @@ +using System.Collections.Generic; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Policy.Engine.Options; +using StellaOps.Policy.Engine.Signals.Entropy; +using Xunit; +using OptionsFactory = Microsoft.Extensions.Options.Options; + +namespace StellaOps.Policy.Engine.Tests.Signals; + +public sealed class EntropyPenaltyCalculatorTests +{ + private readonly EntropyPenaltyCalculator _calculator = new( + OptionsFactory.Create(new PolicyEngineOptions()), + NullLogger.Instance); + + [Fact] + public void ComputeFromJson_ComputesPenaltyAndBlock_WhenImageOpaqueHighAndProvenanceUnknown() + { + var summaryJson = """ + { + "schema": "stellaops.entropy/layer-summary@1", + "imageOpaqueRatio": 0.18, + "layers": [ + { "digest": "sha256:l1", "opaqueBytes": 2306867, "totalBytes": 10485760, "opaqueRatio": 0.22, "indicators": ["packed", "no-symbols"] }, + { "digest": "sha256:l2", "opaqueBytes": 0, "totalBytes": 1048576, "opaqueRatio": 0.0, "indicators": ["symbols"] } + ] + } + """; + + var reportJson = """ + { + "schema": "stellaops.entropy/report@1", + "files": [ + { "path": "/opt/app/libblob.so", "size": 5242880, "opaqueBytes": 1342177, "opaqueRatio": 0.25, "flags": ["stripped", "section:.UPX0"], "windows": [ { "offset": 0, "length": 4096, "entropy": 7.45 } ] }, + { "path": "/opt/app/ok.bin", "size": 1024, "opaqueBytes": 0, "opaqueRatio": 0.0, "flags": [] } + ] + } + """; + + var result = _calculator.ComputeFromJson(summaryJson, reportJson, provenanceAttested: false); + + Assert.True(result.Blocked); + Assert.False(result.Warned); + Assert.InRange(result.Penalty, 0.099m, 0.101m); // ~0.1 after K=0.5 + Assert.Contains("image_opaque_ratio_exceeds_threshold", result.ReasonCodes); + Assert.Contains(result.TopFiles, tf => tf.Path == "/opt/app/libblob.so" && tf.OpaqueRatio == 0.25m); + } + + [Fact] + public void Compute_AppliesMitigationAndCap_WhenSymbolsPresentAndProvenanceAttested() + { + var summary = new EntropyLayerSummary + { + ImageOpaqueRatio = 0.9m, + Layers = new List + { + new() + { + Digest = "sha256:layer", + OpaqueBytes = 900, + TotalBytes = 1000, + Indicators = new List { "symbols" } + } + } + }; + + var report = new EntropyReport + { + Files = new List + { + new() + { + Path = "/bin/high.bin", + Size = 1000, + OpaqueBytes = 900, + OpaqueRatio = 0.9m, + Flags = new List { "packed" } + } + } + }; + + var result = _calculator.Compute(summary, report, provenanceAttested: true); + + Assert.False(result.Blocked); // provenance attested suppresses block + Assert.False(result.Capped); + Assert.InRange(result.Penalty, 0.224m, 0.226m); // mitigation reduces below cap and stays under cap + Assert.Contains("symbols_mitigated", result.ReasonCodes); + Assert.DoesNotContain("penalty_capped", result.ReasonCodes); + } + + [Fact] + public void Compute_WarnsWhenLayerExceedsThresholdWithoutReport() + { + var summary = new EntropyLayerSummary + { + ImageOpaqueRatio = 0.05m, + Layers = new List + { + new() + { + Digest = "sha256:l1", + OpaqueBytes = 40, + TotalBytes = 100, + Indicators = new List { "packed" } + } + } + }; + + var result = _calculator.Compute(summary, report: null, provenanceAttested: false); + + Assert.False(result.Blocked); + Assert.True(result.Warned); + Assert.Contains("file_opaque_ratio_exceeds_threshold", result.ReasonCodes); + } +} diff --git a/src/Scanner/StellaOps.Scanner.Worker/Processing/Reachability/ReachabilityBuildStageExecutor.cs b/src/Scanner/StellaOps.Scanner.Worker/Processing/Reachability/ReachabilityBuildStageExecutor.cs index a04b699f5..02a6dc5a4 100644 --- a/src/Scanner/StellaOps.Scanner.Worker/Processing/Reachability/ReachabilityBuildStageExecutor.cs +++ b/src/Scanner/StellaOps.Scanner.Worker/Processing/Reachability/ReachabilityBuildStageExecutor.cs @@ -1,8 +1,6 @@ using System; using System.Collections.Generic; using System.Linq; -using System.Security.Cryptography; -using System.Text; using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Logging; @@ -37,33 +35,40 @@ public sealed class ReachabilityBuildStageExecutor : IScanStageExecutor } var nodeMap = entryTrace.Nodes.ToDictionary(n => n.Id); + var symbolMap = new Dictionary(nodeMap.Count); var unionNodes = new List(entryTrace.Nodes.Length); foreach (var node in entryTrace.Nodes) { - var symbolId = ComputeSymbolId("shell", node.DisplayName, node.Kind.ToString()); + var command = FormatCommand(node); + var symbolId = SymbolId.ForShell(node.DisplayName, command); + symbolMap[node.Id] = symbolId; var source = node.Evidence is null ? null : new ReachabilitySource("static", "entrytrace", node.Evidence.Path); + var attributes = new Dictionary + { + ["code_id"] = CodeId.FromSymbolId(symbolId) + }; + unionNodes.Add(new ReachabilityUnionNode( SymbolId: symbolId, Lang: "shell", Kind: node.Kind.ToString().ToLowerInvariant(), Display: node.DisplayName, - Source: source)); + Source: source, + Attributes: attributes)); } var unionEdges = new List(entryTrace.Edges.Length); foreach (var edge in entryTrace.Edges) { - if (!nodeMap.TryGetValue(edge.FromNodeId, out var fromNode) || !nodeMap.TryGetValue(edge.ToNodeId, out var toNode)) + if (!symbolMap.TryGetValue(edge.FromNodeId, out var fromId) || !symbolMap.TryGetValue(edge.ToNodeId, out var toId)) { continue; } - var fromId = ComputeSymbolId("shell", fromNode.DisplayName, fromNode.Kind.ToString()); - var toId = ComputeSymbolId("shell", toNode.DisplayName, toNode.Kind.ToString()); unionEdges.Add(new ReachabilityUnionEdge( From: fromId, To: toId, @@ -78,15 +83,13 @@ public sealed class ReachabilityBuildStageExecutor : IScanStageExecutor return ValueTask.CompletedTask; } - private static string ComputeSymbolId(string lang, string display, string kind) + private static string FormatCommand(EntryTraceNode node) { - using var sha = SHA256.Create(); - var input = Encoding.UTF8.GetBytes((display ?? string.Empty) + "|" + (kind ?? string.Empty)); - var hash = sha.ComputeHash(input); - var base64 = Convert.ToBase64String(hash) - .TrimEnd('=') - .Replace('+', '-') - .Replace('/', '_'); - return $"sym:{lang}:{base64}"; + if (!node.Arguments.IsDefaultOrEmpty && node.Arguments.Length > 0) + { + return string.Join(' ', node.Arguments); + } + + return node.Kind.ToString(); } } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/CodeId.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/CodeId.cs index ddb3cb6ff..6a757ce4f 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/CodeId.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/CodeId.cs @@ -25,6 +25,22 @@ public static class CodeId return Build("dotnet", tuple); } + /// + /// Creates a binary code-id using canonical address + length tuple. + /// This aligns with function-level evidence expectations for richgraph-v1. + /// + /// Binary format (elf, pe, macho). + /// Digest of the binary or object file. + /// Virtual address (hex or decimal). Normalized to 0x prefix and lower-case. + /// Optional length in bytes. + /// Optional section name. + /// Optional hash of the code block for stripped binaries. + public static string ForBinarySegment(string format, string fileHash, string address, long? lengthBytes = null, string? section = null, string? codeBlockHash = null) + { + var tuple = $"{Norm(format)}\0{Norm(fileHash)}\0{NormalizeAddress(address)}\0{NormalizeLength(lengthBytes)}\0{Norm(section)}\0{Norm(codeBlockHash)}"; + return Build("binary", tuple); + } + public static string ForNode(string packageName, string entryPath) { var tuple = $"{Norm(packageName)}\0{Norm(entryPath)}"; @@ -48,5 +64,48 @@ public static class CodeId return $"code:{lang}:{base64}"; } + private static string NormalizeAddress(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "0x0"; + } + + var addrText = value.Trim(); + var isHex = addrText.StartsWith("0x", StringComparison.OrdinalIgnoreCase); + if (isHex) + { + addrText = addrText[2..]; + } + + if (long.TryParse(addrText, isHex ? System.Globalization.NumberStyles.HexNumber : System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var addrValue)) + { + if (addrValue < 0) + { + addrValue = 0; + } + + return $"0x{addrValue:x}"; + } + + addrText = addrText.TrimStart('0'); + if (addrText.Length == 0) + { + addrText = "0"; + } + + return $"0x{addrText.ToLowerInvariant()}"; + } + + private static string NormalizeLength(long? value) + { + if (value is null or <= 0) + { + return "unknown"; + } + + return value.Value.ToString("D", System.Globalization.CultureInfo.InvariantCulture); + } + private static string Norm(string? value) => (value ?? string.Empty).Trim(); } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraph.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraph.cs index f29ad166e..ed878a257 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraph.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraph.cs @@ -2,6 +2,7 @@ using System; using System.Collections.Generic; using System.Collections.Immutable; using System.Linq; +using System.Security.Cryptography; namespace StellaOps.Scanner.Reachability; diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/SymbolId.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/SymbolId.cs index 957070134..82f70c3ca 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/SymbolId.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Reachability/SymbolId.cs @@ -132,14 +132,21 @@ public static class SymbolId } /// - /// Creates a binary symbol ID from ELF/PE/Mach-O components. + /// Creates a binary symbol ID from ELF/PE/Mach-O components (legacy overload). /// /// Binary build-id (GNU build-id, PE GUID, Mach-O UUID). /// Section name (e.g., ".text", ".dynsym"). /// Symbol name from symbol table. public static string ForBinary(string buildId, string section, string symbolName) + => ForBinaryAddressed(buildId, section, string.Empty, symbolName, "static", null); + + /// + /// Creates a binary symbol ID that includes file hash, section, address, and linkage. + /// Aligns with {file:hash, section, addr, name, linkage} tuple used by richgraph-v1. + /// + public static string ForBinaryAddressed(string fileHash, string section, string address, string symbolName, string linkage, string? codeBlockHash = null) { - var tuple = $"{Norm(buildId)}\0{Norm(section)}\0{Norm(symbolName)}"; + var tuple = $"{Norm(fileHash)}\0{Norm(section)}\0{NormalizeAddress(address)}\0{Norm(symbolName)}\0{Norm(linkage)}\0{Norm(codeBlockHash)}"; return Build(Lang.Binary, tuple); } @@ -219,6 +226,40 @@ public static class SymbolId return $"sym:{lang}:{hash}"; } + private static string NormalizeAddress(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "0x0"; + } + + var addrText = value.Trim(); + var isHex = addrText.StartsWith("0x", StringComparison.OrdinalIgnoreCase); + if (isHex) + { + addrText = addrText[2..]; + } + + if (long.TryParse(addrText, isHex ? System.Globalization.NumberStyles.HexNumber : System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var addrValue)) + { + if (addrValue < 0) + { + addrValue = 0; + } + + return $"0x{addrValue:x}"; + } + + // Fallback to normalized string representation + addrText = addrText.TrimStart('0'); + if (addrText.Length == 0) + { + addrText = "0"; + } + + return $"0x{addrText.ToLowerInvariant()}"; + } + private static string ComputeFragment(string tuple) { var bytes = Encoding.UTF8.GetBytes(tuple); diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphWriterTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphWriterTests.cs index f1aa6dffa..2b8fca9f1 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphWriterTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphWriterTests.cs @@ -16,8 +16,8 @@ public class RichGraphWriterTests var union = new ReachabilityUnionGraph( Nodes: new[] { - new ReachabilityUnionNode("sym:dotnet:B", "dotnet", "method", display: "B"), - new ReachabilityUnionNode("sym:dotnet:A", "dotnet", "method", display: "A") + new ReachabilityUnionNode("sym:dotnet:B", "dotnet", "method", "B"), + new ReachabilityUnionNode("sym:dotnet:A", "dotnet", "method", "A") }, Edges: new[] { diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/SymbolIdTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/SymbolIdTests.cs new file mode 100644 index 000000000..07ab22a24 --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/SymbolIdTests.cs @@ -0,0 +1,40 @@ +using StellaOps.Scanner.Reachability; +using Xunit; + +namespace StellaOps.Scanner.Reachability.Tests; + +public class SymbolIdTests +{ + [Fact] + public void ForBinaryAddressed_NormalizesAddressAndKeepsLinkage() + { + var id1 = SymbolId.ForBinaryAddressed("sha256:deadbeef", ".text", "0x0040", "foo", "weak"); + var id2 = SymbolId.ForBinaryAddressed("sha256:deadbeef", ".text", "0x40", "foo", "weak"); + + Assert.Equal(id1, id2); + + var id3 = SymbolId.ForBinaryAddressed("sha256:deadbeef", ".text", "40", "foo", "strong"); + Assert.NotEqual(id1, id3); + } + + [Fact] + public void CodeIdBinarySegment_NormalizesAddressAndLength() + { + var cid1 = CodeId.ForBinarySegment("elf", "sha256:abc", "0X0010", 64, ".text"); + var cid2 = CodeId.ForBinarySegment("elf", "sha256:abc", "16", 64, ".text"); + + Assert.Equal(cid1, cid2); + + var cid3 = CodeId.ForBinarySegment("elf", "sha256:abc", "0x20", 32, ".text"); + Assert.NotEqual(cid1, cid3); + } + + [Fact] + public void SymbolIdForShell_RemainsStableForSameCommand() + { + var id1 = SymbolId.ForShell("/entrypoint.sh", "python -m app"); + var id2 = SymbolId.ForShell("/entrypoint.sh", "python -m app"); + + Assert.Equal(id1, id2); + } +} diff --git a/src/Web/StellaOps.Web/src/app/core/api/console-status.client.spec.ts b/src/Web/StellaOps.Web/src/app/core/api/console-status.client.spec.ts index ee163c563..11defd26d 100644 --- a/src/Web/StellaOps.Web/src/app/core/api/console-status.client.spec.ts +++ b/src/Web/StellaOps.Web/src/app/core/api/console-status.client.spec.ts @@ -66,6 +66,8 @@ describe('ConsoleStatusClient', () => { const req = httpMock.expectOne('/console/status'); expect(req.request.method).toBe('GET'); expect(req.request.headers.get('X-StellaOps-Tenant')).toBe('tenant-dev'); + expect(req.request.headers.get('X-Stella-Trace-Id')).toBeTruthy(); + expect(req.request.headers.get('X-Stella-Request-Id')).toBeTruthy(); req.flush(sample); }); @@ -75,7 +77,8 @@ describe('ConsoleStatusClient', () => { expect(eventSourceFactory).toHaveBeenCalled(); const url = eventSourceFactory.calls.mostRecent().args[0]; - expect(url).toBe('/console/runs/run-123/stream?tenant=tenant-dev'); + expect(url).toContain('/console/runs/run-123/stream?tenant=tenant-dev'); + expect(url).toContain('traceId='); // Simulate incoming message const fakeSource = eventSourceFactory.calls.mostRecent().returnValue as unknown as FakeEventSource; diff --git a/src/Web/StellaOps.Web/src/app/core/api/console-status.client.ts b/src/Web/StellaOps.Web/src/app/core/api/console-status.client.ts index e648e5ec8..b45b57ad8 100644 --- a/src/Web/StellaOps.Web/src/app/core/api/console-status.client.ts +++ b/src/Web/StellaOps.Web/src/app/core/api/console-status.client.ts @@ -5,6 +5,7 @@ import { map } from 'rxjs/operators'; import { AuthSessionStore } from '../auth/auth-session.store'; import { ConsoleRunEventDto, ConsoleStatusDto } from './console-status.models'; +import { generateTraceId } from './trace.util'; export const CONSOLE_API_BASE_URL = new InjectionToken('CONSOLE_API_BASE_URL'); @@ -29,9 +30,15 @@ export class ConsoleStatusClient { /** * Poll console status (queue lag, backlog, run counts). */ - getStatus(tenantId?: string): Observable { + getStatus(tenantId?: string, traceId?: string): Observable { const tenant = this.resolveTenant(tenantId); - const headers = new HttpHeaders({ 'X-StellaOps-Tenant': tenant }); + const trace = traceId ?? generateTraceId(); + const headers = new HttpHeaders({ + 'X-StellaOps-Tenant': tenant, + 'X-Stella-Trace-Id': trace, + 'X-Stella-Request-Id': trace, + }); + return this.http.get(`${this.baseUrl}/status`, { headers }).pipe( map((dto) => ({ ...dto, @@ -50,9 +57,10 @@ export class ConsoleStatusClient { * Subscribe to streaming updates for a specific run via SSE. * Caller is responsible for unsubscribing to close the connection. */ - streamRun(runId: string, tenantId?: string): Observable { + streamRun(runId: string, tenantId?: string, traceId?: string): Observable { const tenant = this.resolveTenant(tenantId); - const params = new HttpParams().set('tenant', tenant); + const trace = traceId ?? generateTraceId(); + const params = new HttpParams().set('tenant', tenant).set('traceId', trace); const url = `${this.baseUrl}/runs/${encodeURIComponent(runId)}/stream?${params.toString()}`; return new Observable((observer) => { diff --git a/src/Web/StellaOps.Web/src/app/core/api/risk-http.client.ts b/src/Web/StellaOps.Web/src/app/core/api/risk-http.client.ts index b0aa61623..1a5d42c7b 100644 --- a/src/Web/StellaOps.Web/src/app/core/api/risk-http.client.ts +++ b/src/Web/StellaOps.Web/src/app/core/api/risk-http.client.ts @@ -1,6 +1,6 @@ -import { HttpClient, HttpHeaders, HttpParams } from '@angular/common/http'; +import { HttpClient, HttpErrorResponse, HttpHeaders, HttpParams } from '@angular/common/http'; import { Inject, Injectable, InjectionToken } from '@angular/core'; -import { Observable, map } from 'rxjs'; +import { Observable, catchError, map, throwError } from 'rxjs'; import { AuthSessionStore } from '../auth/auth-session.store'; import { RiskApi } from './risk.client'; @@ -9,6 +9,12 @@ import { generateTraceId } from './trace.util'; export const RISK_API_BASE_URL = new InjectionToken('RISK_API_BASE_URL'); +export class RateLimitError extends Error { + constructor(public readonly retryAfterMs?: number) { + super('rate-limit'); + } +} + @Injectable({ providedIn: 'root' }) export class RiskHttpClient implements RiskApi { constructor( @@ -35,7 +41,8 @@ export class RiskHttpClient implements RiskApi { ...page, page: page.page ?? 1, pageSize: page.pageSize ?? 20, - })) + }), + catchError((err) => throwError(() => this.normalizeError(err))) ); } @@ -50,10 +57,23 @@ export class RiskHttpClient implements RiskApi { map((stats) => ({ countsBySeverity: stats.countsBySeverity, lastComputation: stats.lastComputation ?? '1970-01-01T00:00:00Z', - })) + })), + catchError((err) => throwError(() => this.normalizeError(err))) ); } + private normalizeError(err: unknown): Error { + if (err instanceof RateLimitError) return err; + if (err instanceof HttpErrorResponse && err.status === 429) { + const retryAfter = err.headers.get('Retry-After'); + const retryAfterMs = retryAfter ? Number(retryAfter) * 1000 : undefined; + return new RateLimitError(Number.isFinite(retryAfterMs) ? retryAfterMs : undefined); + } + + if (err instanceof Error) return err; + return new Error('Risk API request failed'); + } + private buildHeaders(tenantId: string, projectId?: string, traceId?: string): HttpHeaders { let headers = new HttpHeaders({ 'X-Stella-Tenant': tenantId }); if (projectId) headers = headers.set('X-Stella-Project', projectId); diff --git a/src/Web/StellaOps.Web/src/app/core/api/risk.store.spec.ts b/src/Web/StellaOps.Web/src/app/core/api/risk.store.spec.ts index 63419ab7a..d5f6681c0 100644 --- a/src/Web/StellaOps.Web/src/app/core/api/risk.store.spec.ts +++ b/src/Web/StellaOps.Web/src/app/core/api/risk.store.spec.ts @@ -2,6 +2,7 @@ import { TestBed } from '@angular/core/testing'; import { of, throwError } from 'rxjs'; import { RISK_API } from './risk.client'; +import { RateLimitError } from './risk-http.client'; import { RiskQueryOptions, RiskResultPage, RiskStats } from './risk.models'; import { RiskStore } from './risk.store'; @@ -47,6 +48,14 @@ describe('RiskStore', () => { expect(store.error()).toBe('boom'); }); + it('reports rate limit errors with retry hint', () => { + apiSpy.list.and.returnValue(throwError(() => new RateLimitError(5000))); + + store.fetchList(defaultOptions); + + expect(store.error()).toContain('retry after 5s'); + }); + it('stores stats results', () => { const stats: RiskStats = { countsBySeverity: { none: 0, info: 0, low: 1, medium: 0, high: 1, critical: 0 }, diff --git a/src/Web/StellaOps.Web/src/app/core/api/risk.store.ts b/src/Web/StellaOps.Web/src/app/core/api/risk.store.ts index 7c4651462..ff06ed578 100644 --- a/src/Web/StellaOps.Web/src/app/core/api/risk.store.ts +++ b/src/Web/StellaOps.Web/src/app/core/api/risk.store.ts @@ -2,6 +2,7 @@ import { inject, Injectable, Signal, computed, signal } from '@angular/core'; import { finalize } from 'rxjs/operators'; import { RISK_API, RiskApi } from './risk.client'; +import { RateLimitError } from './risk-http.client'; import { RiskQueryOptions, RiskResultPage, RiskStats } from './risk.models'; @Injectable({ providedIn: 'root' }) @@ -47,6 +48,13 @@ export class RiskStore { } private normalizeError(err: unknown): string { + if (err instanceof RateLimitError) { + if (err.retryAfterMs && Number.isFinite(err.retryAfterMs)) { + const seconds = Math.ceil(err.retryAfterMs / 1000); + return `Rate limited; retry after ${seconds}s`; + } + return 'Rate limited; retry shortly'; + } if (err instanceof Error) return err.message; return 'Unknown error fetching risk data'; } diff --git a/src/Web/StellaOps.Web/src/app/core/console/console-status.service.spec.ts b/src/Web/StellaOps.Web/src/app/core/console/console-status.service.spec.ts new file mode 100644 index 000000000..49b4648d0 --- /dev/null +++ b/src/Web/StellaOps.Web/src/app/core/console/console-status.service.spec.ts @@ -0,0 +1,75 @@ +import { TestBed } from '@angular/core/testing'; +import { Subject, of } from 'rxjs'; + +import { ConsoleRunEventDto, ConsoleStatusDto } from '../api/console-status.models'; +import { ConsoleStatusClient } from '../api/console-status.client'; +import { ConsoleStatusService } from './console-status.service'; +import { ConsoleStatusStore } from './console-status.store'; + +class FakeConsoleStatusClient { + public streams: { subject: Subject; traceId?: string }[] = []; + + getStatus(): any { + const dto: ConsoleStatusDto = { + backlog: 0, + queueLagMs: 0, + activeRuns: 0, + pendingRuns: 0, + healthy: true, + lastCompletedRunId: null, + lastCompletedAt: null, + }; + return of(dto); + } + + streamRun(_runId: string, _tenantId?: string, traceId?: string) { + const subject = new Subject(); + this.streams.push({ subject, traceId }); + return subject.asObservable(); + } +} + +describe('ConsoleStatusService', () => { + let service: ConsoleStatusService; + let client: FakeConsoleStatusClient; + + beforeEach(() => { + TestBed.configureTestingModule({ + providers: [ + ConsoleStatusStore, + ConsoleStatusService, + { provide: ConsoleStatusClient, useClass: FakeConsoleStatusClient }, + ], + }); + + service = TestBed.inject(ConsoleStatusService); + client = TestBed.inject(ConsoleStatusClient) as unknown as FakeConsoleStatusClient; + jasmine.clock().install(); + }); + + afterEach(() => { + jasmine.clock().uninstall(); + }); + + it('reconnects when heartbeat is missed', () => { + const sub = service.subscribeToRun('run-1', { heartbeatMs: 5, maxRetries: 2, traceId: 'trace-heartbeat' }); + + expect(client.streams.length).toBe(1); + jasmine.clock().tick(6); + expect(client.streams.length).toBe(2); + + sub.unsubscribe(); + }); + + it('retries after stream errors with backoff', () => { + const sub = service.subscribeToRun('run-2', { maxRetries: 1, heartbeatMs: 50, traceId: 'trace-error' }); + + expect(client.streams.length).toBe(1); + client.streams[0].subject.error(new Error('boom')); + + jasmine.clock().tick(1001); + expect(client.streams.length).toBe(2); + + sub.unsubscribe(); + }); +}); diff --git a/src/Web/StellaOps.Web/src/app/core/console/console-status.service.ts b/src/Web/StellaOps.Web/src/app/core/console/console-status.service.ts index eed5efb44..d30bb541c 100644 --- a/src/Web/StellaOps.Web/src/app/core/console/console-status.service.ts +++ b/src/Web/StellaOps.Web/src/app/core/console/console-status.service.ts @@ -5,6 +5,14 @@ import { switchMap } from 'rxjs/operators'; import { ConsoleStatusClient } from '../api/console-status.client'; import { ConsoleRunEventDto, ConsoleStatusDto } from '../api/console-status.models'; import { ConsoleStatusStore } from './console-status.store'; +import { generateTraceId } from '../api/trace.util'; + +export interface RunStreamOptions { + heartbeatMs?: number; + maxRetries?: number; + traceId?: string; + tenantId?: string; +} @Injectable({ providedIn: 'root', @@ -58,14 +66,65 @@ export class ConsoleStatusService { /** * Subscribe to run stream events for a given run id. */ - subscribeToRun(runId: string): Subscription { + subscribeToRun(runId: string, options?: RunStreamOptions): Subscription { this.store.clearEvents(); - return this.client.streamRun(runId).subscribe({ - next: (evt: ConsoleRunEventDto) => this.store.appendRunEvent(evt), - error: (err) => { - console.error('console run stream error', err); - this.store.setError('Run stream disconnected'); - }, + + const traceId = options?.traceId ?? generateTraceId(); + const heartbeatMs = options?.heartbeatMs ?? 15000; + const maxRetries = options?.maxRetries ?? 3; + const tenantId = options?.tenantId; + + let retries = 0; + let heartbeatHandle: ReturnType | undefined; + let innerSub: Subscription | null = null; + let disposed = false; + + const clearHeartbeat = () => { + if (heartbeatHandle) { + clearTimeout(heartbeatHandle); + heartbeatHandle = undefined; + } + }; + + const scheduleHeartbeat = () => { + clearHeartbeat(); + heartbeatHandle = setTimeout(() => { + handleError(new Error('heartbeat-timeout')); + }, heartbeatMs); + }; + + const handleError = (err: unknown) => { + console.error('console run stream error', err); + this.store.setError('Run stream disconnected'); + if (disposed || retries >= maxRetries) { + return; + } + const delay = Math.min(1000 * Math.pow(2, retries), 30000); + retries += 1; + setTimeout(connect, delay); + }; + + const connect = () => { + if (disposed) return; + innerSub?.unsubscribe(); + const stream$ = this.client.streamRun(runId, tenantId, traceId); + innerSub = stream$.subscribe({ + next: (evt: ConsoleRunEventDto) => { + retries = 0; + this.store.appendRunEvent(evt); + scheduleHeartbeat(); + }, + error: handleError, + }); + scheduleHeartbeat(); + }; + + connect(); + + return new Subscription(() => { + disposed = true; + clearHeartbeat(); + innerSub?.unsubscribe(); }); } diff --git a/tools/cosign/cosign b/tools/cosign/cosign new file mode 120000 index 000000000..396f39d8b --- /dev/null +++ b/tools/cosign/cosign @@ -0,0 +1 @@ +v2.6.0/cosign-linux-amd64 \ No newline at end of file