prep docs and service updates
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
This commit is contained in:
master
@@ -1,12 +1,22 @@
|
||||
# Ledger Observability Prep — PREP-LEDGER-OBS-54-001
|
||||
|
||||
Status: Draft (2025-11-20)
|
||||
Status: Prep complete (2025-11-20)
|
||||
Owners: Findings Ledger Guild · Provenance Guild
|
||||
Scope: Minimal API surface for `/ledger/attestations` and observability hooks.
|
||||
Scope: Minimal HTTP surface plus determinism/telemetry hooks for `/v1/ledger/attestations`.
|
||||
|
||||
## Needs
|
||||
- HTTP surface spec (routes, auth scopes) to host `/ledger/attestations`.
|
||||
- Telemetry fields to include provenance IDs.
|
||||
## Agreed contract (PREP-LEDGER-OBS-54-001)
|
||||
- HTTP surface published in `docs/modules/findings-ledger/prep/ledger-attestations-http.md`.
|
||||
- Endpoint: `GET /v1/ledger/attestations` with tenant header `X-Stella-Tenant` and bearer scope `ledger.attest.read` (or mTLS).
|
||||
- Filters: `artifactId`, `findingId`, `attestationId`, `status`, `sinceRecordedAt`, `untilRecordedAt`, `limit`.
|
||||
- Ordering/pagination: deterministic by `recordedAt ASC, attestationId ASC`; pagination token encodes `{recordedAt, attestationId, filtersHash}`.
|
||||
- Response shape (JSON or NDJSON): ids, verification status/time, DSSE digest, optional Rekor entry id, evidence bundle ref, source ledger event id, Merkle leaf + root hashes.
|
||||
- Offline posture: no live Rekor calls; all hashes lowercase SHA-256; times UTC; deterministic sort only.
|
||||
|
||||
## Telemetry hooks
|
||||
- Log events: `ledger.attestations.query` (tenant, filtersHash, limit, duration_ms, result_count).
|
||||
- Metrics: `ledger_attestations_queries_total{tenant,status}`; `ledger_attestations_failures_total{reason}`.
|
||||
- Tracing: span `ledger.attestations.query` with attributes `filtersHash`, `next_page_token_present`.
|
||||
|
||||
## Handoff
|
||||
Use as PREP artefact; update once API contract is drafted.
|
||||
- Use `docs/modules/findings-ledger/prep/ledger-attestations-http.md` as the binding prep artefact for LEDGER-OBS-54-001 / 55-001 implementation.
|
||||
- Service scaffolding and OAS wiring land in LEDGER-OBS-54-001 once the web-service handler is added.
|
||||
|
||||
@@ -1,9 +1,52 @@
|
||||
# Ledger Packs Snapshot Prep — PREP-LEDGER-PACKS-42-001
|
||||
|
||||
Status: Draft (2025-11-20)
|
||||
Status: Prep complete (2025-11-20)
|
||||
Owners: Findings Ledger Guild · Mirror Creator Guild
|
||||
Scope: Snapshot/time-travel contract for packs simulation.
|
||||
Scope: Snapshot/time-travel contract for packs simulation and offline CLI execution (PREP-LEDGER-PACKS-42-001).
|
||||
|
||||
## Needs
|
||||
- Snapshot format and bundle layout for pack simulation/time-travel.
|
||||
## Goals
|
||||
- Provide deterministic, tenant-scoped snapshots that let pack runners/CLI replay ledger state offline.
|
||||
- Allow “time-travel” queries (choose exact ledger sequence/cycle) to debug policy outcomes.
|
||||
- Reuse existing export shapes where possible and avoid redundant DB projections.
|
||||
|
||||
## Surfaces
|
||||
- `GET /v1/ledger/packs/snapshots`
|
||||
- Headers: `X-Stella-Tenant` (required), bearer scope `ledger.packs.read`.
|
||||
- Query: `atSequence` (long, optional), `atCycleHash` (string, optional), `sinceSequence` / `untilSequence` (long, optional), `page_size` (default 100, max 1000), `page_token`.
|
||||
- Returns: list of available snapshot descriptors (JSON or NDJSON) sorted by `sequence ASC`.
|
||||
- `GET /v1/ledger/packs/snapshots/{snapshotId}/download`
|
||||
- Streams a gzip tarball containing the snapshot bundle (see layout).
|
||||
- Supports `Accept: application/vnd.stella.pack-snapshot+tar` (default) or `application/x-ndjson` for manifest-only dry-run (no payload files) when `Prefer: return=representation` is absent.
|
||||
|
||||
## Snapshot descriptor fields
|
||||
- `snapshot_id` (uuid, stable)
|
||||
- `tenant`
|
||||
- `base_sequence` (long) — earliest ledger event included
|
||||
- `upper_sequence` (long) — last ledger event included (inclusive)
|
||||
- `cycle_hash` (string) — Merkle cycle hash at `upper_sequence`
|
||||
- `policy_version`, `projector_version`, `generator_version`
|
||||
- `created_at` (ISO-8601 UTC)
|
||||
- `approx_uncompressed_size_bytes`
|
||||
- `content` summary: counts for `findings`, `vex`, `advisories`, `sboms`
|
||||
|
||||
## Bundle layout (tar.gz)
|
||||
- `manifest.json`: descriptor above plus SHA-256 digests and lengths for each payload file.
|
||||
- `findings.ndjson`: canonical finding shape matching `/ledger/export/findings`.
|
||||
- `vex.ndjson`, `advisories.ndjson`, `sboms.ndjson`: same shapes/filters as export endpoints.
|
||||
- `indexes/`: optional bloom/filter helpers for fast CLI lookup (`component_purl`, `advisory_id`, `risk_profile_version`).
|
||||
- `provenance.json`: DSSE envelope with bundle hash, generator inputs (seed, source commit, policy version).
|
||||
|
||||
## Determinism and filters
|
||||
- Snapshot is deterministic for a given `(tenant, base_sequence, upper_sequence, cycle_hash, policy_version)`.
|
||||
- Page tokens: base64url JSON `{ "last": { "upper_sequence": long, "snapshot_id": uuid }, "filters_hash": sha256 }`.
|
||||
- When `atCycleHash` is provided, server resolves the closest <=matching cycle and emits one descriptor; otherwise uses `untilSequence` or latest committed.
|
||||
- No wall-clock dependence; `created_at` reflects generator runtime but is stored once and signed in provenance.
|
||||
|
||||
## Validation rules
|
||||
- Reject overwrite if snapshot with identical `(tenant, upper_sequence, cycle_hash)` already published (idempotent response with existing `snapshot_id`).
|
||||
- Reject if requested window crosses projector gap (missing sequences) with error `409` and `X-Stella-Gap-From/To`.
|
||||
- Enforce `page_size` consistency across tokens; 400 on mismatch.
|
||||
|
||||
## Artefact location
|
||||
- This prep: `docs/modules/findings-ledger/prep/2025-11-20-ledger-packs-42-001-prep.md`.
|
||||
- Bundle schema is derived from export shapes in `docs/modules/findings-ledger/export-http-surface.md`; SDK/OAS plumbing to be added in LEDGER-PACKS-42-001 implementation.
|
||||
|
||||
@@ -1,11 +1,50 @@
|
||||
# Ledger Risk Schema Prep — PREP-LEDGER-RISK-66-001/002
|
||||
|
||||
Status: Draft (2025-11-20)
|
||||
Status: Prep complete (2025-11-20)
|
||||
Owners: Findings Ledger Guild · Risk Engine Guild
|
||||
Scope: Contract + data model for PREP-LEDGER-RISK-66-001/002 (risk scoring fields and deterministic upsert).
|
||||
|
||||
## Needs
|
||||
- Risk engine schema/contract inputs: `risk_score`, `risk_severity`, `profile_version`, `explanation_id`, indexes.
|
||||
- Migration plan to add fields.
|
||||
## Field definitions (canonical finding projection)
|
||||
- `risk_score` (numeric, 0–100, 2dp) — monotonic per `(finding_id, profile_version)`; computed by Risk Engine.
|
||||
- `risk_severity` (enum) — derived mapping: `critical >= 90`, `high >= 70`, `medium >= 40`, `low >= 10`, `informational < 10`.
|
||||
- `risk_profile_version` (string) — semantic version of the scoring policy/profile; required.
|
||||
- `risk_explanation_id` (uuid/string) — pointer to Risk Engine explanation payload stored in Risk service (not duplicated in ledger).
|
||||
- `risk_event_sequence` (long) — ledger sequence of the last applied risk event; enforces monotonic updates.
|
||||
- `risk_updated_at` (ISO-8601 UTC) — when the score was last written.
|
||||
|
||||
## Storage and indexes (MongoDB)
|
||||
- Collection: `findings` (existing). Add fields above to the projection document.
|
||||
- Unique compound index: `{ tenant: 1, finding_id: 1, risk_profile_version: 1 }`.
|
||||
- Query helper index for exports/UI: `{ tenant: 1, risk_severity: 1, risk_score: -1, observed_at: -1 }`.
|
||||
- TTL: none; scores are historical but superseded by deterministic upsert described below.
|
||||
|
||||
## Deterministic upsert flow (LEDGER-RISK-66-002)
|
||||
1. Risk Engine emits `RiskScoreApplied` event with `{tenant, finding_id, profile_version, score, explanation_id, event_sequence}`.
|
||||
2. Handler loads current projection by `(tenant, finding_id)`; compares `(profile_version, event_sequence)`:
|
||||
- If incoming `event_sequence` < stored `risk_event_sequence` → ignore (idempotent).
|
||||
- If equal → idempotent update allowed only when score/severity unchanged.
|
||||
- If greater → write new values and set `risk_event_sequence = event_sequence`.
|
||||
3. All writes recorded in ledger append with same event_sequence for audit; projection updates deterministic by sequence ordering.
|
||||
4. Exports (`/ledger/export/findings`) surface these fields; snapshot bundles reuse the same shape.
|
||||
|
||||
## API/SDK contract hooks
|
||||
- OAS baseline will mark all four fields in the finding shapes (canonical + compact) as optional today, required once migrations finish.
|
||||
- `/ledger/export/findings` filters: `risk_profile_version` (already reserved), add `risk_severity` and `risk_score_min/max` in the next OAS bump.
|
||||
- UI/SDK must treat missing `risk_profile_version` as “not yet scored”.
|
||||
|
||||
## Migration/rollout plan (LEDGER-RISK-66-001)
|
||||
- Step 1: Add fields and indexes behind feature flag `RiskScoringEnabled` (default off).
|
||||
- Step 2: Backfill for latest profile per tenant using Risk Engine batch export; write via deterministic upsert to enforce ordering.
|
||||
- Step 3: Enable streaming ingestion of `RiskScoreApplied` events; monitor lag via metric `ledger_risk_score_apply_lag_seconds`.
|
||||
- Step 4: Flip default for `RiskScoringEnabled` to on after backfill success criteria:
|
||||
- 99.9% of existing findings have `risk_profile_version` populated.
|
||||
- No rejected events due to sequence regressions in the last 24h.
|
||||
- Step 5: Update OAS/SDK to mark fields required; notify UI/Export consumers.
|
||||
|
||||
## Observability
|
||||
- Log: `ledger.risk.apply` with tenant, finding_id, profile_version, score, event_sequence, applied (bool).
|
||||
- Metrics: `ledger_risk_apply_total{result}`; `ledger_risk_score_latest{severity}` gauges per tenant.
|
||||
- Tracing: span `ledger.risk.apply` tagging `profile_version`, `event_sequence`, `idempotent`.
|
||||
|
||||
## Handoff
|
||||
Use as PREP artefact; update when risk field definitions and rollout plan are available.
|
||||
- This document is the prep artefact for PREP-LEDGER-RISK-66-001/002. Implementation tasks wire schema + deterministic upsert and extend exports/OAS accordingly.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Verification Event Contract (attestations → ledger_attestations)
|
||||
|
||||
Status: Draft (2025-11-21)
|
||||
Owners: Provenance Guild · Findings Ledger Guild
|
||||
|
||||
Purpose: unblock LEDGER-OBS-54-001 by defining the ingestion event emitted by the verifier so we can populate `ledger_attestations`.
|
||||
|
||||
```
|
||||
event_type: verification.attestation.completed
|
||||
payload:
|
||||
tenant_id: string (required)
|
||||
attestation_id: uuid (required)
|
||||
artifact_id: string (required; OCI digest or SBOM id)
|
||||
finding_id: string (optional)
|
||||
verification_status: string enum [verified, failed, unknown] (required)
|
||||
verification_time: string (ISO-8601 UTC, required)
|
||||
dsse_digest: string (sha256, lowercase, required)
|
||||
rekor_entry_id: string (optional)
|
||||
evidence_bundle_ref: string (optional)
|
||||
merkle_leaf_hash: string (sha256, required)
|
||||
root_hash: string (sha256, required)
|
||||
cycle_hash: string (required)
|
||||
projection_version: string (required)
|
||||
```
|
||||
|
||||
Ordering/monotonicity:
|
||||
- Events are emitted with a ledger `sequence_no`. Ingestion must ignore any verification event with `sequence_no` less than the stored `risk_event_sequence` for the same `(tenant_id, attestation_id)`.
|
||||
|
||||
Determinism for ingestion:
|
||||
- Sort by `(sequence_no ASC, attestation_id ASC)` before upsert.
|
||||
- Upsert target: `ledger_attestations` (see `004_ledger_attestations.sql`).
|
||||
|
||||
Open question:
|
||||
- Should `verification_status` include `expired`/`revoked`? Need decision before marking schema final.
|
||||
|
||||
Next step:
|
||||
- Once the verifier confirms this payload, wire ingestion job to project into `ledger_attestations` and flip LEDGER-OBS-54-001 to DOING.
|
||||
@@ -35,4 +35,5 @@
|
||||
|
||||
## Artefact location
|
||||
- This prep doc: `docs/modules/findings-ledger/prep/ledger-attestations-http.md`.
|
||||
- Storage/view contract: `docs/modules/findings-ledger/prep/ledger-attestations-storage.md`.
|
||||
- Add path to OAS in a follow-on increment (LEDGER-OAS-61-002/63-001) once approved.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Ledger Attestations Storage & Query Contract (LEDGER-OBS-54-001)
|
||||
|
||||
Status: PrepComplete (2025-11-20)
|
||||
Owners: Findings Ledger Guild · Provenance Guild
|
||||
|
||||
## Goal
|
||||
Provide a deterministic storage/view contract so the `/v1/ledger/attestations` endpoint can be implemented without further design work.
|
||||
|
||||
## Table (proposed)
|
||||
- Name: `ledger_attestations`
|
||||
- Partitioning: tenant-scoped (same strategy as `ledger_events`).
|
||||
- Columns:
|
||||
- `tenant_id` (text, not null)
|
||||
- `attestation_id` (uuid, not null)
|
||||
- `artifact_id` (text, not null) — OCI digest or SBOM id
|
||||
- `finding_id` (text, null)
|
||||
- `verification_status` (text, not null; `verified|failed|unknown`)
|
||||
- `verification_time` (timestamptz, not null)
|
||||
- `dsse_digest` (text, not null; lowercase sha256)
|
||||
- `rekor_entry_id` (text, null)
|
||||
- `evidence_bundle_ref` (text, null)
|
||||
- `ledger_event_id` (uuid, not null) — source ledger event linking the attestation
|
||||
- `recorded_at` (timestamptz, not null)
|
||||
- `merkle_leaf_hash` (text, not null)
|
||||
- `root_hash` (text, not null)
|
||||
- `cycle_hash` (text, not null)
|
||||
- `projection_version` (text, not null)
|
||||
|
||||
## Indexes / ordering
|
||||
- PK: `(tenant_id, attestation_id)`
|
||||
- Paging index: `(tenant_id, recorded_at, attestation_id)` to back deterministic sort `recorded_at ASC, attestation_id ASC`.
|
||||
- Lookup indexes:
|
||||
- `(tenant_id, artifact_id, recorded_at DESC)`
|
||||
- `(tenant_id, finding_id, recorded_at DESC)`
|
||||
- `(tenant_id, verification_status, recorded_at DESC)`
|
||||
|
||||
## Query contract for `/v1/ledger/attestations`
|
||||
- Filters map to indexed columns: `artifactId`, `findingId`, `attestationId`, `status`, `sinceRecordedAt`, `untilRecordedAt`.
|
||||
- Pagination token encodes `{ recordedAt, attestationId, filtersHash }`; server must reject mismatched hash.
|
||||
- Response fields align 1:1 with columns above; no joins required.
|
||||
- Determinism: sort strictly by `(recorded_at ASC, attestation_id ASC)`; no server clocks in payload.
|
||||
|
||||
## Migration notes
|
||||
- Add table and indexes in the same migration (see `src/Findings/StellaOps.Findings.Ledger/migrations/004_ledger_attestations.sql`).
|
||||
- Backfill from existing provenance/verification store (if present) into this table with recorded_at = original verification timestamp.
|
||||
- Ensure writes/coalescing happen via ledger projections to keep `ledger_event_id`/`cycle_hash` consistent.
|
||||
|
||||
## Observability
|
||||
- Logs: `ledger.attestations.query` (tenant, filtersHash, limit, duration_ms, result_count).
|
||||
- Metrics: `ledger_attestations_queries_total{tenant,status}`, `ledger_attestations_failures_total{reason}`; reuse endpoint spans already defined in prep doc.
|
||||
|
||||
## Artefact location
|
||||
- Storage contract: `docs/modules/findings-ledger/prep/ledger-attestations-storage.md`
|
||||
- HTTP contract: `docs/modules/findings-ledger/prep/ledger-attestations-http.md`
|
||||
Reference in New Issue
Block a user