feat: Add VEX compact fixture and implement offline verifier for Findings Ledger exports
- Introduced a new VEX compact fixture for testing purposes. - Implemented `verify_export.py` script to validate Findings Ledger exports, ensuring deterministic ordering and applying redaction manifests. - Added a lightweight stub `HarnessRunner` for unit tests to validate ledger hashing expectations. - Documented tasks related to the Mirror Creator. - Created models for entropy signals and implemented the `EntropyPenaltyCalculator` to compute penalties based on scanner outputs. - Developed unit tests for `EntropyPenaltyCalculator` to ensure correct penalty calculations and handling of edge cases. - Added tests for symbol ID normalization in the reachability scanner. - Enhanced console status service with comprehensive unit tests for connection handling and error recovery. - Included Cosign tool version 2.6.0 with checksums for various platforms.
This commit is contained in:
@@ -2,3 +2,9 @@
|
||||
# Findings Ledger
|
||||
|
||||
Start here for ledger docs.
|
||||
|
||||
## Quick links
|
||||
- FL1–FL10 remediation tracker: `gaps-FL1-FL10.md`
|
||||
- Schema catalog (events/projections/exports): `schema-catalog.md`
|
||||
- Merkle & external anchor policy: `merkle-anchor-policy.md`
|
||||
- Tenant isolation & redaction manifest: `tenant-isolation-redaction.md`
|
||||
|
||||
26
docs/modules/findings-ledger/dsse-policy-linkage.md
Normal file
26
docs/modules/findings-ledger/dsse-policy-linkage.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# DSSE & Policy Hash Linkage (FL6)
|
||||
|
||||
**Goal:** Every export, replay report, and anchor manifest is tied to the exact policy digest that produced it and is verifiable offline via DSSE.
|
||||
|
||||
## Binding rules
|
||||
1. **Policy digest:** `policyVersion` (SHA-256 over policy bundle) is mandatory in ledger events, projections, exports, and replay reports.
|
||||
2. **DSSE payload types**
|
||||
- `application/vnd.stella-ledger-export+json` — export manifests (hashlist + filtersHash).
|
||||
- `application/vnd.stella-ledger-anchor+json` — Merkle anchors (see `merkle-anchor-policy.md`).
|
||||
- `application/vnd.stella-ledger-harness+json` — replay harness report.
|
||||
3. **Hashlists:** export manifests contain `sha256` for each emitted NDJSON line (`lineDigest`), plus a dataset digest (`datasetSha256`) over concatenated line digests. Replay harness exposes `eventStreamChecksum` and `projectionChecksum`.
|
||||
4. **Policy linkage:** DSSE payload must include `policyHash` and `schemaVersion` to prevent replay under mismatched policy versions.
|
||||
|
||||
## Offline verification flow
|
||||
1. Verify DSSE signature (local key or Rekor transparency log if online).
|
||||
2. Recompute dataset checksum with `tools/LedgerReplayHarness/scripts/verify_export.py --input <export.ndjson> --expected <datasetSha256>`.
|
||||
3. Cross-check `policyHash` in payload matches policy bundle in use; mismatch → block import/export.
|
||||
|
||||
## File locations
|
||||
- Harness DSSE placeholder now embeds `policyHash` when `LEDGER_POLICY_HASH` env var is set.
|
||||
- Export manifests and checksums: `docs/modules/findings-ledger/golden-checksums.json`.
|
||||
- External anchors: `docs/modules/findings-ledger/merkle-anchor-policy.md` (DSSE template).
|
||||
- Set `LEDGER_POLICY_HASH` before running `tools/LedgerReplayHarness` to imprint the policy digest into the generated `.sig` file.
|
||||
|
||||
## Change management
|
||||
- Any change to payloadType or hash recipe bumps schema version in `schema-catalog.md` and requires new DSSE key roll announcement.
|
||||
28
docs/modules/findings-ledger/gaps-FL1-FL10.md
Normal file
28
docs/modules/findings-ledger/gaps-FL1-FL10.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Findings Ledger — FL1–FL10 Remediation (LEDGER-GAPS-121-009)
|
||||
|
||||
**Source advisory:** `docs/product-advisories/archived/27-Nov-2025-superseded/28-Nov-2025 - Findings Ledger and Immutable Audit Trail.md`
|
||||
**Created:** 2025-12-02 · **Owner:** Findings Ledger Guild
|
||||
|
||||
## Gap closure map
|
||||
|
||||
| ID | Gap summary | Remediation artefact(s) | Evidence / notes |
|
||||
| --- | ----------- | ----------------------- | ---------------- |
|
||||
| FL1 | Versioned ledger event schema (canonical JSON + hashes) | `docs/modules/findings-ledger/schema-catalog.md` §1; updated `docs/modules/findings-ledger/schema.md` canonical rules | Canonical envelope `v1.0.0` stamped; hash derivation pinned to `sha256(canonicalJson)` + `sha256(eventHash-sequence)`. |
|
||||
| FL2 | Projection schema versions + cycle hash determinism | `schema-catalog.md` §2; `schema.md` §4 | Projection `v1.0.0` with cycle-hash recipe and required fields; rebuild checksum guard in harness. |
|
||||
| FL3 | Export schema (canonical/compact) + filter hash versioning | `schema-catalog.md` §3; golden fixtures under `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/` | Canonical export shape tagged `export.v1.canonical`; compact tagged `export.v1.compact`; fixtures hashed. |
|
||||
| FL4 | Merkle + external anchor policy (Rekor/offline) | `docs/modules/findings-ledger/merkle-anchor-policy.md` | Anchoring cadence (1k/15m), Rekor/air-gap policy, anchor ref format, DSSE anchoring manifest. |
|
||||
| FL5 | Tenant isolation + redaction manifest for exports/logs | `docs/modules/findings-ledger/tenant-isolation-redaction.md`; manifest: `docs/modules/findings-ledger/redaction-manifest.yaml` | Per-tenant partitions, export field redaction (comments, actor ids), signed manifest checksum. |
|
||||
| FL6 | DSSE + policy hash linkage for exports and attestations | `docs/modules/findings-ledger/dsse-policy-linkage.md`; harness DSSE placeholder includes `policyHash` | Describes payloadType + bindings to policy digest and export hashlist. |
|
||||
| FL7 | Deterministic export fixtures (golden) | `fixtures/golden/*.ndjson` (findings, vex, advisories, sboms) | Each includes `filtersHash`, `cycleHash`, `policyVersion`; hashes logged in manifest. |
|
||||
| FL8 | Offline verifier script for bundles/exports | `tools/LedgerReplayHarness/scripts/verify_export.py` | Pure-Python, no deps; validates ordering, recomputes SHA-256 and optional expected hash file. |
|
||||
| FL9 | Replay/rebuild checksum guard | Harness update: `tools/LedgerReplayHarness/Program.cs` (`--expected-checksum`) | Computes event-stream and projection checksums; fails on mismatch; emitted in report. |
|
||||
| FL10 | Quotas/backpressure metrics and alerts | Metrics update: `Observability/LedgerMetrics.cs`; doc: `observability.md` §2/§4 | New counters `ledger_backpressure_applied_total`, gauge `ledger_quota_remaining`, alert guidance. |
|
||||
|
||||
## How to verify
|
||||
- Run `dotnet run --project tools/LedgerReplayHarness -- --fixture <path> --connection <conn> --tenant <tenant> --report out/report.json --metrics out/metrics.json --expected-checksum <baseline-checksums.json>` (use a file produced by a known-good run; template: `docs/modules/findings-ledger/replay-checksums.sample.json`).
|
||||
- Validate exports: `python tools/LedgerReplayHarness/scripts/verify_export.py --input fixtures/golden/findings-canonical.ndjson --schema export.v1.canonical`.
|
||||
- Check manifest hashes: `sha256sum docs/modules/findings-ledger/redaction-manifest.yaml fixtures/golden/*.ndjson`.
|
||||
|
||||
## Follow-ons
|
||||
- Integrate Rekor anchor publishing toggle into Helm/Compose overlays (tracked separately).
|
||||
- Mirror golden fixtures into Offline Kit once export pipeline emits real data.
|
||||
53
docs/modules/findings-ledger/golden-checksums.json
Normal file
53
docs/modules/findings-ledger/golden-checksums.json
Normal file
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"generatedAt": "2025-12-02T00:00:00Z",
|
||||
"policyHash": "sha256:policy-v1",
|
||||
"datasets": {
|
||||
"findings-canonical.ndjson": {
|
||||
"path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/findings-canonical.ndjson",
|
||||
"schema": "export.v1.canonical",
|
||||
"records": 2,
|
||||
"filtersHash": "a81d6c6d2bcf9c0e7cbb1fcd292e4b7cc21f6d5c4e3f2b1a0c9d8e7f6c5b4a3e",
|
||||
"sha256": "cd270235484748f2f4c871e9d574796e6f61b48df9cc65e009dab4ba0769dfa4"
|
||||
},
|
||||
"vex-compact.ndjson": {
|
||||
"path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/vex-compact.ndjson",
|
||||
"schema": "export.v1.compact",
|
||||
"records": 1,
|
||||
"filtersHash": "b5c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb998877665544",
|
||||
"sha256": "e786a12b4ee08776df73f7f2a97907280b5f8bb76cc7a901e2a680d3fe69e85e"
|
||||
},
|
||||
"advisories-canonical.ndjson": {
|
||||
"path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/advisories-canonical.ndjson",
|
||||
"schema": "export.v1.canonical",
|
||||
"records": 1,
|
||||
"filtersHash": "c6d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb99887766554433",
|
||||
"sha256": "6d5a2d522179b616c112c255c7dd06b3434ae0a4992009d25ea82f50144425ab"
|
||||
},
|
||||
"sboms-compact.ndjson": {
|
||||
"path": "src/Findings/StellaOps.Findings.Ledger/fixtures/golden/sboms-compact.ndjson",
|
||||
"schema": "export.v1.compact",
|
||||
"records": 1,
|
||||
"filtersHash": "d7e8f9a0b1c2d3e4f50617283940aa5544332211ffeeccbb9988776655443322",
|
||||
"sha256": "c89be7fcc511c4ef5a4a291c45061da1a7f4592506150e5b9bce92ba2bb5bbe2"
|
||||
}
|
||||
},
|
||||
"manifests": {
|
||||
"redaction-manifest.yaml": {
|
||||
"path": "docs/modules/findings-ledger/redaction-manifest.yaml",
|
||||
"schema": "redaction.v1",
|
||||
"sha256": "7c2f437a47c6514ad4688072b8b5e33b2e0cd0f9f289f15b49bf2f7def54a730"
|
||||
},
|
||||
"redaction-manifest.json": {
|
||||
"path": "docs/modules/findings-ledger/redaction-manifest.json",
|
||||
"schema": "redaction.v1",
|
||||
"sha256": "6965ea311f65482e6f51da0fd26cae1995997fcd456cea6dac84ab7b3354990a"
|
||||
}
|
||||
},
|
||||
"replay": {
|
||||
"sample": {
|
||||
"path": "docs/modules/findings-ledger/replay-checksums.sample.json",
|
||||
"schema": "ledger.harness.v1",
|
||||
"note": "replace with harness-produced checksums before enforcement"
|
||||
}
|
||||
}
|
||||
}
|
||||
50
docs/modules/findings-ledger/merkle-anchor-policy.md
Normal file
50
docs/modules/findings-ledger/merkle-anchor-policy.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Merkle & External Anchor Policy (FL4)
|
||||
|
||||
**Audience:** Findings Ledger Guild · DevOps · Compliance
|
||||
**Applies to:** `src/Findings/StellaOps.Findings.Ledger` (Merkle worker, anchoring jobs)
|
||||
|
||||
## Anchoring cadence
|
||||
- **Batch size:** 1,000 events or **15 minutes**, whichever is first (`LedgerServiceOptions:Merkle.BatchSize/WindowDuration`).
|
||||
- **Tree:** flat Merkle over `merkle_leaf_hash` (see `schema-catalog.md` §1). Root hashed with SHA-256; no salt.
|
||||
- **Partitions:** per-tenant batching only; no cross-tenant mixing.
|
||||
- **Ordering:** leaves ordered by `(sequence_no, recorded_at)`. Any deviation is a failure.
|
||||
|
||||
## Anchor references
|
||||
- `ledger_merkle_roots.anchor_reference` formats:
|
||||
- `rekor::<uuid>` when pushed to Rekor.
|
||||
- `airgap::<bundleId>` when sealed in offline bundle.
|
||||
- `none` (empty) for internal-only anchors.
|
||||
- External publication is optional but **must** include DSSE envelope with payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/vnd.stella-ledger-anchor+json",
|
||||
"payload": {
|
||||
"tenant": "<tenant>",
|
||||
"rootHash": "<sha256>",
|
||||
"leafCount": 1000,
|
||||
"windowStart": "2025-12-02T00:00:00Z",
|
||||
"windowEnd": "2025-12-02T00:15:00Z",
|
||||
"policyHash": "<policyVersion>",
|
||||
"schemaVersion": "ledger.event.v1"
|
||||
},
|
||||
"signatures": [...]
|
||||
}
|
||||
```
|
||||
|
||||
## Determinism & recovery
|
||||
- Anchor worker enforces stable ordering; replay harness recomputes Merkle roots and fails when root mismatch (FL9 guard).
|
||||
- Root hash + DSSE signature are stored alongside export bundles for offline verification.
|
||||
- External anchors **never** include tenant-identifying data beyond tenant id already present in ledger tables.
|
||||
|
||||
## Air-gap posture
|
||||
- Rekor publication optional; when disabled, anchors are sealed inside offline bundles with `anchor_reference=airgap::<bundleId>`.
|
||||
- Anchor manifest is bundled in Offline Kit under `offline/ledger/anchors/<tenant>/<anchorId>.json`.
|
||||
- No outbound network calls when `ExternalAnchoring:Enabled=false`.
|
||||
|
||||
## Monitoring
|
||||
- Metrics: `ledger_merkle_anchor_duration_seconds`, `ledger_merkle_anchor_failures_total`, `ledger_backpressure_applied_total{reason="anchoring"}`, `ledger_quota_remaining{kind="ingest"}`.
|
||||
- Alerts: see `observability.md` (AnchorFailure + new Backpressure alert).
|
||||
|
||||
## Change control
|
||||
- Any change to batch size/window or hash recipe requires bumping `ledger.event` schema minor version and updating `schema-catalog.md`.
|
||||
@@ -14,7 +14,10 @@
|
||||
| --- | --- | --- | --- |
|
||||
| `ledger_write_duration_seconds` | Histogram | `tenant`, `event_type`, `source` | End-to-end append latency (API ingress → persisted). P95 ≤ 120 ms. |
|
||||
| `ledger_events_total` | Counter | `tenant`, `event_type`, `source` (`policy`, `workflow`, `orchestrator`) | Incremented per committed event. Mirrors Merkle leaf count. |
|
||||
| `ledger_ingest_backlog_events` | Gauge | — | Number of events buffered in the writer/anchor queues. Alert when >5 000 for 5 min. |
|
||||
| `ledger_ingest_backlog_events` | Gauge | `tenant` | Number of events buffered in the writer/anchor queues. Alert when >5 000 for 5 min. |
|
||||
| `ledger_quota_remaining` | Gauge | `tenant` | Remaining ingest capacity before backpressure applies (defaults to 5 000 events). |
|
||||
| `ledger_backpressure_applied_total` | Counter | `tenant`, `reason`, `limit` | Incremented whenever backlog crosses quota threshold. |
|
||||
| `ledger_quota_rejections_total` | Counter | `tenant`, `reason` | Incremented when requests are actively rejected due to quotas. |
|
||||
| `ledger_projection_lag_seconds` | Gauge | `tenant` | Wall-clock difference between latest ledger event and projection tail. Target <30 s. |
|
||||
| `ledger_projection_rebuild_seconds` | Histogram | `tenant` | Duration of replay/rebuild operations triggered by LEDGER-29-008 harness. |
|
||||
| `ledger_projection_apply_seconds` | Histogram | `tenant`, `event_type`, `policy_version`, `evaluation_status` | Time to apply a single ledger event to projection. Target P95 <1 s. |
|
||||
@@ -43,6 +46,7 @@
|
||||
| --- | --- | --- |
|
||||
| **LedgerWriteSLA** | `ledger_write_latency_seconds` P95 > 1 s for 3 intervals | Check DB contention, review queue backlog, scale writer. |
|
||||
| **LedgerBacklogGrowing** | `ledger_ingest_backlog_events` > 5 000 for 5 min | Inspect upstream policy runs, ensure projector keeping up. |
|
||||
| **LedgerBackpressure** | `ledger_backpressure_applied_total` increases while `ledger_quota_remaining` < 0 | Throttle callers, raise quota or scale anchor worker. |
|
||||
| **ProjectionLag** | `ledger_projection_lag_seconds` > 30 s | Trigger rebuild, verify change streams. |
|
||||
| **AnchorFailure** | `ledger_merkle_anchor_failures_total` increase > 0 | Collect logs, rerun anchor, verify signing service. |
|
||||
| **AttachmentSecurityError** | `ledger_attachments_encryption_failures_total` increase > 0 | Audit attachments pipeline; check key material and storage endpoints. |
|
||||
|
||||
29
docs/modules/findings-ledger/redaction-manifest.json
Normal file
29
docs/modules/findings-ledger/redaction-manifest.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"schemaVersion": "redaction.v1",
|
||||
"generatedAt": "2025-12-02T00:00:00Z",
|
||||
"owner": "findings-ledger-guild",
|
||||
"rules": {
|
||||
"ledger.event": [
|
||||
{ "path": "$.actor.id", "action": "mask", "maskWith": "user:<realm>" },
|
||||
{ "path": "$.payload.comment", "action": "drop" },
|
||||
{ "path": "$.payload.ticket.url", "action": "drop" },
|
||||
{ "path": "$.payload.attachments[*].downloadUrl", "action": "drop" }
|
||||
],
|
||||
"export.canonical": [
|
||||
{ "path": "$.actorId", "action": "mask", "maskWith": "user:<realm>" },
|
||||
{ "path": "$.comment", "action": "drop" },
|
||||
{ "path": "$.attachments", "action": "drop" }
|
||||
],
|
||||
"export.compact": [
|
||||
{ "path": "$.actorId", "action": "drop" },
|
||||
{ "path": "$.comment", "action": "drop" },
|
||||
{ "path": "$.policyRationale", "action": "drop" },
|
||||
{ "path": "$.attachments", "action": "drop" },
|
||||
{ "path": "$.labels", "action": "drop" }
|
||||
],
|
||||
"observability": [
|
||||
{ "path": "$.event_body", "action": "drop" },
|
||||
{ "path": "$.actor_id", "action": "hash", "hashWith": "sha256" }
|
||||
]
|
||||
}
|
||||
}
|
||||
39
docs/modules/findings-ledger/redaction-manifest.yaml
Normal file
39
docs/modules/findings-ledger/redaction-manifest.yaml
Normal file
@@ -0,0 +1,39 @@
|
||||
schemaVersion: redaction.v1
|
||||
generatedAt: 2025-12-02T00:00:00Z
|
||||
owner: findings-ledger-guild
|
||||
rules:
|
||||
ledger.event:
|
||||
- path: $.actor.id
|
||||
action: mask
|
||||
maskWith: "user:<realm>"
|
||||
- path: $.payload.comment
|
||||
action: drop
|
||||
- path: $.payload.ticket.url
|
||||
action: drop
|
||||
- path: $.payload.attachments[*].downloadUrl
|
||||
action: drop
|
||||
export.canonical:
|
||||
- path: $.actorId
|
||||
action: mask
|
||||
maskWith: "user:<realm>"
|
||||
- path: $.comment
|
||||
action: drop
|
||||
- path: $.attachments
|
||||
action: drop
|
||||
export.compact:
|
||||
- path: $.actorId
|
||||
action: drop
|
||||
- path: $.comment
|
||||
action: drop
|
||||
- path: $.policyRationale
|
||||
action: drop
|
||||
- path: $.attachments
|
||||
action: drop
|
||||
- path: $.labels
|
||||
action: drop
|
||||
observability:
|
||||
- path: $.event_body
|
||||
action: drop
|
||||
- path: $.actor_id
|
||||
action: hash
|
||||
hashWith: sha256
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"eventStream": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
|
||||
"projection": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
|
||||
"notes": "Replace with real values from harness output before enforcing checksum guard."
|
||||
}
|
||||
75
docs/modules/findings-ledger/schema-catalog.md
Normal file
75
docs/modules/findings-ledger/schema-catalog.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Findings Ledger Schema Catalog (FL1–FL3)
|
||||
|
||||
**Scope:** Versioned canonical schemas for ledger events, projections, and exports.
|
||||
**Status:** v1.0.0 sealed (2025-12-02) — breaking changes require new minor/major version tags.
|
||||
|
||||
## 1) Ledger event envelope — `ledger.event.v1`
|
||||
|
||||
| Field | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `event.id` | `uuid` | V7 GUID allowed. |
|
||||
| `event.type` | `string` (`ledger_event_type`) | See `schema.md` §2.2. |
|
||||
| `event.tenant` | `string` | Partition key. |
|
||||
| `event.chainId` | `uuid` | Derived when absent (`tenant :: policyVersion`), see `workflow-inference.md`. |
|
||||
| `event.sequence` | `long` | Gapless per chain, starts at 1. |
|
||||
| `event.policyVersion` | `string` | SHA-256 digest of policy bundle; propagated into exports and DSSE. |
|
||||
| `event.finding` | object | `id`, `artifactId`, `vulnId`. |
|
||||
| `event.actor` | object | `id`, `type` (`system|operator|integration`). |
|
||||
| `event.occurredAt` | `string` (UTC ISO-8601 ms) | Domain clock. |
|
||||
| `event.recordedAt` | `string` (UTC ISO-8601 ms) | Service `TimeProvider`. |
|
||||
| `event.payload` | object | Mutation-specific body. |
|
||||
| `event.evidenceBundleRef` | `string?` | DSSE/capsule id (optional). |
|
||||
| `event.airgap.bundle` | object? | See `airgap-provenance.md`. |
|
||||
| `event_hash` | `char(64)` | `sha256(canonicalJson)` lower-hex. |
|
||||
| `previous_hash` | `char(64)` | All-zero for chain genesis. |
|
||||
| `merkle_leaf_hash` | `char(64)` | `sha256(event_hash || "-" || sequence)`. |
|
||||
|
||||
Canonicalisation: UTF-8, sorted keys, lower-case enums, ISO-8601 UTC with millisecond precision, arrays stable-order. Any field addition bumps minor version.
|
||||
|
||||
## 2) Finding projection — `ledger.projection.v1`
|
||||
|
||||
| Field | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `tenantId` | `string` | Partition key. |
|
||||
| `findingId` | `string` | Stable identity. |
|
||||
| `policyVersion` | `string` | Hash of active policy bundle. |
|
||||
| `status` | `string` | `affected|triaged|accepted_risk|resolved|unknown`. |
|
||||
| `severity` | `number` | 0–10, 3 decimal places. |
|
||||
| `riskScore` | `number` | 0–10, 3 decimal places. |
|
||||
| `riskSeverity` | `string` | `low|medium|high|critical|unknown`. |
|
||||
| `riskProfileVersion` | `string` | Version/hash from Risk Engine. |
|
||||
| `riskExplanationId` | `uuid?` | Links to explain bundle. |
|
||||
| `labels` | `json` | KEV/runtime flags, sorted keys. |
|
||||
| `currentEventId` | `uuid` | Source ledger event. |
|
||||
| `explainRef` | `string?` | Object storage / DSSE reference. |
|
||||
| `policyRationale` | `json` | Array of rationale refs. |
|
||||
| `updatedAt` | `string` UTC | Projection timestamp. |
|
||||
| `cycleHash` | `char(64)` | `sha256(canonicalProjectionJson)`; used in exports. |
|
||||
|
||||
Projection deterministic hash recipe: serialize projection record with sorted keys (excluding `updatedAt` jitter) and hash via SHA-256. The replay harness recomputes and compares.
|
||||
|
||||
## 3) Export payloads — `export.v1`
|
||||
|
||||
Shapes share headers: `policyVersion`, `projectionVersion` (cycle hash), `filtersHash`, `pageToken`, `observedAt`, `provenance` (`ledgerRoot`, `projectorVersion`, `policyHash`, optional `dsseDigest`).
|
||||
|
||||
### Canonical vs compact
|
||||
- **Canonical (`export.v1.canonical`)** — full provenance fields, evidence refs, DSSE linkage.
|
||||
- **Compact (`export.v1.compact`)** — drops verbose fields (`policyRationale`, comments, actor ids), keeps `cycleHash` + `filtersHash` for determinism; redaction manifest enforced.
|
||||
|
||||
### Record fields
|
||||
- Findings: `findingId`, `eventSequence`, `status`, `severity`, `risk`, `advisories[]`, `evidenceBundleRef`, `cycleHash`.
|
||||
- VEX: `vexStatementId`, `product`, `status`, `justification`, `knownExploited`, `cycleHash`.
|
||||
- Advisories: `advisoryId`, `source`, `cvss{version,vector,baseScore}`, `epss`, `kev`, `cycleHash`.
|
||||
- SBOMs: `sbomId`, `subject{digest,mediaType}`, `sbomFormat`, `componentsCount`, `materials[]`, `cycleHash`.
|
||||
|
||||
Filters hash: `sha256(sortedQueryString)`; stored alongside fixtures for replayability.
|
||||
|
||||
## 4) Versioning rules
|
||||
- Patch: backward-compatible field additions (new optional key) — bump patch digit.
|
||||
- Minor: additive required fields or canonical rule tweaks — bump minor.
|
||||
- Major: breaking change (field removal/rename, hash recipe) — bump major and keep prior schema frozen.
|
||||
|
||||
## 5) Reference artefacts
|
||||
- Golden fixtures: `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/*.ndjson`.
|
||||
- Checksum manifest: `docs/modules/findings-ledger/golden-checksums.json`.
|
||||
- Offline verifier: `tools/LedgerReplayHarness/scripts/verify_export.py`.
|
||||
@@ -119,6 +119,11 @@ Canonicalisation rules:
|
||||
5. Numbers use decimal notation; omit trailing zeros.
|
||||
6. Arrays maintain supplied order.
|
||||
|
||||
### 2.4 Versioning & DSSE linkage (FL1, FL6)
|
||||
- Canonical schema identifiers are catalogued in `schema-catalog.md` (`ledger.event.v1`, `ledger.projection.v1`, `export.v1.*`).
|
||||
- Any change to the envelope, hash recipe, or required fields bumps the catalog version; legacy versions remain frozen.
|
||||
- DSSE artefacts (anchors, exports, replay reports) **must** embed `policyVersion` and `schemaVersion` (see `dsse-policy-linkage.md`).
|
||||
|
||||
Hash pipeline:
|
||||
|
||||
```
|
||||
@@ -270,7 +275,7 @@ Ordering and pagination: `ORDER BY recorded_at ASC, attestation_id ASC` with cur
|
||||
1. Canonical serialize the envelope (§2.3).
|
||||
2. Compute `event_hash` and store along with `previous_hash`.
|
||||
3. Build Merkle tree per anchoring window using leaf hash `SHA256(event_hash || '-' || sequence_no)`.
|
||||
4. Persist root in `ledger_merkle_roots` and, when configured, submit to external transparency log (Rekor v2). Store receipt/UUID in `anchor_reference`.
|
||||
4. Persist root in `ledger_merkle_roots` and, when configured, submit to external transparency log (Rekor v2). Store receipt/UUID in `anchor_reference` (see `merkle-anchor-policy.md`).
|
||||
5. Projection rows compute `cycle_hash = SHA256(canonical_projection_json)` where canonical projection includes fields `{tenant_id, finding_id, policy_version, status, severity, labels, current_event_id}` with sorted keys.
|
||||
|
||||
Verification flow for auditors:
|
||||
@@ -284,6 +289,8 @@ Verification flow for auditors:
|
||||
- Initial migration script: `src/Findings/StellaOps.Findings.Ledger/migrations/001_initial.sql`.
|
||||
- Sample canonical event: `seed-data/findings-ledger/fixtures/ledger-event.sample.json` (includes pre-computed `eventHash`, `previousHash`, and `merkleLeafHash` values).
|
||||
- Sample projection row: `seed-data/findings-ledger/fixtures/finding-projection.sample.json` (includes canonical `cycleHash` for replay validation).
|
||||
- Golden export fixtures (FL7): `src/Findings/StellaOps.Findings.Ledger/fixtures/golden/*.ndjson` with checksums in `docs/modules/findings-ledger/golden-checksums.json`.
|
||||
- Redaction manifest (FL5): `docs/modules/findings-ledger/redaction-manifest.yaml` governs mask/drop rules for canonical vs compact exports.
|
||||
|
||||
Fixtures follow canonical key ordering and include precomputed hashes to validate tooling.
|
||||
|
||||
|
||||
28
docs/modules/findings-ledger/tenant-isolation-redaction.md
Normal file
28
docs/modules/findings-ledger/tenant-isolation-redaction.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Tenant Isolation & Redaction Manifest (FL5)
|
||||
|
||||
**Purpose:** Document how Findings Ledger enforces tenant boundaries and which fields are redacted in deterministic exports.
|
||||
|
||||
## Isolation controls
|
||||
- Storage: all ledger, projection, history, and merkle tables are **LIST-partitioned by `tenant_id`** (PostgreSQL). Cross-tenant queries are disallowed at repo level.
|
||||
- Queueing: Merkle batches and projector pipelines are keyed by `(tenant_id, chain_id)`; no mixing.
|
||||
- Exports: `/ledger/export/*` requires `X-Stella-Tenant`; service rejects multi-tenant requests.
|
||||
- Hashing: event/projection hashes include `tenant_id` as part of canonical envelope, preventing replay across tenants.
|
||||
|
||||
## Redaction policy
|
||||
- User-generated content (comments, attachments metadata) is excluded from compact exports and masked in canonical exports per manifest.
|
||||
- Actor identifiers are truncated to realm (`user:<realm>`); emails/PII never emitted.
|
||||
- Evidence bundle references are retained, but inline evidence payloads are not stored in ledger.
|
||||
|
||||
## Manifest
|
||||
- Path: `docs/modules/findings-ledger/redaction-manifest.yaml` (JSON twin: `redaction-manifest.json` for offline tooling).
|
||||
- Content: declarative list of fields redacted or truncated for each export shape.
|
||||
- The manifest is signed in checksum list `docs/modules/findings-ledger/golden-checksums.json`; sha256 must match before release.
|
||||
|
||||
### Applying the manifest
|
||||
- Canonical exports apply `redact: mask` rules only to PII (`actorId`, `comment`); compact exports drop (`drop: true`) the same fields plus verbose rationale arrays.
|
||||
- Log pipelines ensure `event_body` is never written to logs; only metadata/hashes appear (see `observability.md`).
|
||||
|
||||
## Validation steps
|
||||
1. `sha256sum docs/modules/findings-ledger/redaction-manifest.yaml` matches `golden-checksums.json`.
|
||||
2. Run `python tools/LedgerReplayHarness/scripts/verify_export.py --input fixtures/golden/findings-canonical.ndjson --schema export.v1.canonical --manifest docs/modules/findings-ledger/redaction-manifest.json` (script enforces mask/drop rules offline).
|
||||
3. Confirm export responses in staging omit masked fields for the requesting tenant.
|
||||
Reference in New Issue
Block a user