up
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled

This commit is contained in:
StellaOps Bot
2025-12-03 00:10:19 +02:00
parent ea1d58a89b
commit 37cba83708
158 changed files with 147438 additions and 867 deletions

View File

@@ -24,5 +24,6 @@
- Check manifest hashes: `sha256sum docs/modules/findings-ledger/redaction-manifest.yaml fixtures/golden/*.ndjson`.
## Follow-ons
- Keep lightweight test stub `HarnessRunner` (unit-only) to avoid heavy harness bootstrap during fast tests; revisit once harness logic is extracted into a reusable library.
- Integrate Rekor anchor publishing toggle into Helm/Compose overlays (tracked separately).
- Mirror golden fixtures into Offline Kit once export pipeline emits real data.

View File

@@ -0,0 +1,39 @@
# Graph Analytics Gaps (GA1GA10) Remediation Plan
**Sprint:** 0207-0001-0001 (Experience & SDKs 180.C)
**Artifacts produced:** schemas + samples for analytics results/bundles; governance rules; test/fixture expectations.
## Objectives (mapped to GA1GA10)
- **GA1 — Versioned analytics schemas:** `analytics-result.schema.json` defines versioned result payloads with `schemaVersion` + `algorithmVersion`.
- **GA2 — Deterministic seeds/rerun-hash CI:** every job records `seed`, `rerunHash = sha256(inputs+seed+algorithmVersion)`, and must replay to identical outputs.
- **GA3 — Privacy/tenant redaction:** results require `tenant` field; redaction rules apply before export (`redactions[]` logged).
- **GA4 — Baseline datasets/fixtures:** ship minimal deterministic fixture set under `src/Graph/__Tests/Fixtures/analytics-baseline/` (TODO when code added) and sample bundle here.
- **GA5 — Performance budgets/quotas:** default budgets captured in schema (`budgetSeconds`, `maxNodes`, `maxEdges`); jobs failing budgets emit `status=budget_exceeded`.
- **GA6 — Explainability metadata:** include `inputs`, `seed`, `algorithmVersion`, `parameters`, `provenance` (source hashes) for replay.
- **GA7 — Checksums + DSSE for exports:** bundle schema carries per-file SHA-256 plus optional DSSE signature envelope reference.
- **GA8 — Algorithm versioning:** `algorithmVersion` semver and `changeLogUrl` required; breaking changes bump MAJOR.
- **GA9 — Offline analytics bundle schema:** `analytics-bundle.schema.json` documents offline package with manifest, dataset hashes, redactions, and optional signatures.
- **GA10 — SemVer/change-log governance:** bundles must cite `changeLogUrl`; release notes must link to signed manifests; exports failing SemVer gating are rejected.
## Schemas & Samples
- `docs/modules/graph/analytics/analytics-result.schema.json`
- `docs/modules/graph/analytics/analytics-bundle.schema.json`
- Sample bundle: `docs/modules/graph/analytics/samples/analytics-bundle.sample.json`
## Rules of Engagement
1. **Determinism:** fixed `seed`; stable ordering of nodes/edges; `rerunHash` must match across runs given same inputs/seed.
2. **Redaction before export:** `redactions[]` enumerates removed fields per tenant policy; exports lacking redaction entries are invalid for multi-tenant bundles.
3. **Signatures (optional but encouraged):** DSSE/JWS envelopes over `bundle.manifest` and `resultHash` using offline keys; record under `signatures[]`.
4. **Offline readiness:** no network fetch during analysis or validation; datasets referenced by hash + relative path.
5. **Performance budgets:** defaults—`budgetSeconds: 30`, `maxNodes: 50000`, `maxEdges: 200000`; overridable per job but must be logged.
## Implementation Hooks
- API/Indexer must emit analytics results conforming to `analytics-result.schema.json`.
- Export jobs must validate bundles against `analytics-bundle.schema.json` and attach DSSE refs when available.
- CI: add rerun-hash check in analytics test pipeline using fixture bundle; fail on drift.
## Open Follow-ups
- Add real fixtures under `src/Graph/__Tests/Fixtures/analytics-baseline/` mirrored in Offline Kit.
- Wire DSSE signing in release pipeline once signing keys for Graph are provisioned.
## Evidence
- Schemas + sample committed in this sprint. Link in sprint Decisions & Risks. Tests to follow in analytics pipeline PR.***

View File

@@ -0,0 +1,115 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.local/graph/analytics-bundle.schema.json",
"title": "Graph Analytics Bundle (Offline)",
"type": "object",
"additionalProperties": false,
"required": [
"bundleId",
"tenant",
"schemaVersion",
"analyticsResults",
"datasets",
"manifest",
"hashes",
"createdAt"
],
"properties": {
"bundleId": { "type": "string", "pattern": "^analytics-bundle:[A-Za-z0-9._:-]+$" },
"tenant": { "type": "string", "minLength": 1 },
"schemaVersion": { "type": "string", "pattern": "^1\\.\\d+\\.\\d+$" },
"createdAt": { "type": "string", "format": "date-time" },
"analyticsResults": {
"type": "array",
"items": {
"type": "object",
"required": ["analysisId", "resultPath", "resultHash", "algorithmVersion", "schemaVersion"],
"additionalProperties": false,
"properties": {
"analysisId": { "type": "string" },
"resultPath": { "type": "string" },
"resultHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" },
"schemaVersion": { "type": "string" },
"algorithmVersion": { "type": "string" },
"rerunHash": { "type": "string", "pattern": "^[a-f0-9]{64}$" }
}
},
"uniqueItems": true
},
"datasets": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "path", "hash"],
"additionalProperties": false,
"properties": {
"id": { "type": "string" },
"path": { "type": "string" },
"hash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" },
"redactions": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }
}
},
"uniqueItems": true
},
"manifest": {
"type": "array",
"items": {
"type": "object",
"required": ["path", "sha256", "size"],
"additionalProperties": false,
"properties": {
"path": { "type": "string" },
"sha256": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" },
"size": { "type": "integer", "minimum": 0 },
"contentType": { "type": "string" }
}
},
"uniqueItems": true
},
"hashes": {
"type": "object",
"additionalProperties": false,
"properties": {
"bundleSha256": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" },
"manifestSha256": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" }
}
},
"signatures": {
"type": "array",
"items": {
"type": "object",
"required": ["type", "keyId", "signature"],
"additionalProperties": false,
"properties": {
"type": { "type": "string", "enum": ["dsse", "jws-detached"] },
"keyId": { "type": "string" },
"signature": { "type": "string" },
"envelopeDigest": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }
}
}
},
"metadata": {
"type": "object",
"additionalProperties": false,
"properties": {
"budgets": {
"type": "object",
"additionalProperties": false,
"properties": {
"budgetSeconds": { "type": "number", "minimum": 0, "default": 30 },
"maxNodes": { "type": "integer", "minimum": 0, "default": 50000 },
"maxEdges": { "type": "integer", "minimum": 0, "default": 200000 }
}
},
"offline": {
"type": "object",
"additionalProperties": false,
"properties": {
"sealed": { "type": "boolean", "default": true },
"provenance": { "type": "string" }
}
}
}
}
}
}

View File

@@ -0,0 +1,80 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://stellaops.local/graph/analytics-result.schema.json",
"title": "Graph Analytics Result",
"type": "object",
"additionalProperties": false,
"required": [
"schemaVersion",
"algorithmVersion",
"analysisId",
"tenant",
"inputs",
"seed",
"rerunHash",
"metrics",
"result",
"createdAt"
],
"properties": {
"schemaVersion": { "type": "string", "pattern": "^1\\.\\d+\\.\\d+$" },
"algorithmVersion": { "type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$" },
"changeLogUrl": { "type": "string", "format": "uri" },
"analysisId": { "type": "string", "minLength": 1 },
"tenant": { "type": "string", "minLength": 1 },
"inputs": {
"type": "object",
"additionalProperties": false,
"properties": {
"snapshotIds": { "type": "array", "items": { "type": "string" }, "uniqueItems": true },
"filters": { "type": "object" },
"datasetHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }
}
},
"seed": { "type": "integer", "minimum": 0 },
"parameters": { "type": "object" },
"rerunHash": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
"metrics": {
"type": "object",
"additionalProperties": false,
"properties": {
"durationSeconds": { "type": "number", "minimum": 0 },
"budgetSeconds": { "type": "number", "minimum": 0 },
"maxNodes": { "type": "integer", "minimum": 0 },
"maxEdges": { "type": "integer", "minimum": 0 },
"nodesProcessed": { "type": "integer", "minimum": 0 },
"edgesProcessed": { "type": "integer", "minimum": 0 },
"status": { "type": "string", "enum": ["ok", "budget_exceeded", "failed"] }
},
"required": ["durationSeconds", "budgetSeconds", "status"]
},
"result": {
"type": "object",
"description": "Algorithm-specific payload (centrality, community detection, reachability, etc.).",
"additionalProperties": true
},
"provenance": {
"type": "object",
"additionalProperties": false,
"properties": {
"inputsHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" },
"resultHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" },
"manifestHash": { "type": "string", "pattern": "^sha256:[A-Fa-f0-9]{64}$" }
}
},
"redactions": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"field": { "type": "string" },
"reason": { "type": "string" },
"policy": { "type": "string" }
},
"required": ["field", "reason"]
}
},
"createdAt": { "type": "string", "format": "date-time" }
}
}

View File

@@ -0,0 +1,62 @@
{
"$schema": "../analytics-bundle.schema.json",
"bundleId": "analytics-bundle:graph:2025-12-02T00-00Z",
"tenant": "default",
"schemaVersion": "1.0.0",
"createdAt": "2025-12-02T00:00:00Z",
"analyticsResults": [
{
"analysisId": "centrality-2025-12-02",
"resultPath": "results/centrality.ndjson",
"resultHash": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd",
"schemaVersion": "1.0.0",
"algorithmVersion": "2.1.0",
"rerunHash": "29d58b9fdc5c4e65b26c03f3bd9f442ff0c7f8514b8a9225f8b6417ffabc0101"
}
],
"datasets": [
{
"id": "snapshot-2025-12-01",
"path": "datasets/graph-snapshot-2025-12-01.tzst",
"hash": "sha256:fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210",
"redactions": ["user.email", "org.internalNotes"]
}
],
"manifest": [
{
"path": "results/centrality.ndjson",
"sha256": "89abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567",
"size": 104857,
"contentType": "application/x-ndjson"
},
{
"path": "datasets/graph-snapshot-2025-12-01.tzst",
"sha256": "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210",
"size": 2097152,
"contentType": "application/octet-stream"
}
],
"hashes": {
"bundleSha256": "0f0e0d0c0b0a09080706050403020100ffeeddccbbaa99887766554433221100",
"manifestSha256": "aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55aa55"
},
"signatures": [
{
"type": "dsse",
"keyId": "graph-analytics-dev-pub",
"signature": "MEQCIDevGraphSig==",
"envelopeDigest": "sha256:bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66bb66"
}
],
"metadata": {
"budgets": {
"budgetSeconds": 30,
"maxNodes": 50000,
"maxEdges": 200000
},
"offline": {
"sealed": true,
"provenance": "offline-kit:graph-analytics:2025-12"
}
}
}

View File

@@ -112,7 +112,7 @@ Key notes:
| **DSL Compiler** (`Dsl/`) | Parse, canonicalise, IR generation, checksum caching. | Uses Roslyn-like pipeline; caches by `policyId+version+hash`. |
| **Selection Layer** (`Selection/`) | Batch SBOM ↔ advisory ↔ VEX joiners; apply equivalence tables; support incremental cursors. | Deterministic ordering (SBOM → advisory → VEX). |
| **Evaluator** (`Evaluation/`) | Execute IR with first-match semantics, compute severity/trust/reachability weights, record rule hits. | Stateless; all inputs provided by selection layer. |
| **Signals** (`Signals/`) | Normalizes reachability, trust, entropy, uncertainty, runtime hits into a single dictionary passed to Evaluator; supplies default `unknown` values when signals missing. Entropy penalties are derived from Scanner `layer_summary.json`/`entropy.report.json` (K=0.5, cap=0.3, block at image opaque ratio > 0.15 w/ unknown provenance) and exported via `policy_entropy_penalty_value` / `policy_entropy_image_opaque_ratio`. | Aligns with `signals.*` namespace in DSL. |
| **Signals** (`Signals/`) | Normalizes reachability, trust, entropy, uncertainty, runtime hits into a single dictionary passed to Evaluator; supplies default `unknown` values when signals missing. Entropy penalties are derived from Scanner `layer_summary.json`/`entropy.report.json` (K=0.5, cap=0.3, block at image opaque ratio > 0.15 w/ unknown provenance) and exported via `policy_entropy_penalty_value` / `policy_entropy_image_opaque_ratio`; SPL scope `entropy.*` exposes `penalty`, `image_opaque_ratio`, `blocked`, `warned`, `capped`, `top_file_opaque_ratio`. | Aligns with `signals.*` namespace in DSL. |
| **Materialiser** (`Materialization/`) | Upsert effective findings, append history, manage explain bundle exports. | Mongo transactions per SBOM chunk. |
| **Orchestrator** (`Runs/`) | Change-stream ingestion, fairness, retry/backoff, queue writer. | Works with Scheduler Models DTOs. |
| **API** (`Api/`) | Minimal API endpoints, DTO validation, problem responses, idempotency. | Generated clients for CLI/UI. |

View File

@@ -0,0 +1,138 @@
# Contract: POLICY-CONSOLE-23-001 — Console findings/export & simulation surfaces
**Status:** Draft → Proposed (2025-12-02)
**Scope**
- Provide deterministic, tenant-scoped APIs from Policy Engine to StellaOps Console for findings browse/export and simulation/explain experiences.
- Replace legacy ad-hoc Console queries with cursor-based, RBAC-aware endpoints that expose provenance and aggregation hints.
- Keep all responses deterministic (stable ordering, explicit timestamps, no wall-clock/default time windows).
## Versioning & Compatibility
- `schemaVersion`: `console-policy-23-001` (bumped on breaking changes).
- Media type: `application/vnd.stellaops.console-policy-23-001+json` (clients MUST send `Accept` and SHOULD send `Content-Type`).
- Backward-compatible additions follow additive fields; ordering and cursor format remain stable.
## Authentication & RBAC
- Required scopes: `policy:read`, `effective:read`, `explain:read` (all tenant-scoped).
- Optional `findings:export` to enable NDJSON bulk export.
- All endpoints require `X-Tenant-Id`; server enforces tenant filter and rejects cross-tenant cursor reuse.
## Determinism Rules
- Ordering: `policyVersion DESC`, `artifactDigest ASC`, `purl ASC`, `ruleId ASC`, `findingId ASC`.
- Cursor: opaque, URL-safe base64 of the last tuple above; contains `policyVersion|artifactDigest|purl|ruleId|findingId` plus `schemaVersion`. No server clocks in cursors.
- Timestamps: clients MUST provide `evaluationTimestamp` or `timeWindowStart/End`; server never injects `DateTime.UtcNow` defaults.
- Randomness/network access disallowed; sampling ratios must be provided by the client or policy config.
## Endpoints
### 1) List findings (paged)
- **GET** `/policy/console/findings`
- **Query params**
- `cursor` (string, optional)
- `limit` (int, 1500, default 100)
- `severityBand[]` (enum: critical|high|medium|low|unknown)
- `ruleId[]`, `policyId`, `policyVersion`
- `artifactDigest[]`, `purl[]`, `namespace[]`
- `advisoryId[]`, `vexStatement[]`
- `state[]` (open|waived|fixed|not_applicable)
- `timeWindowStart`, `timeWindowEnd` (ISO-8601, optional)
- `sort` (one of `default`, `severity_desc`, `artifact`, `rule`); default respects deterministic tuple above.
- **Response**
```json
{
"schemaVersion": "console-policy-23-001",
"items": [
{
"findingId": "ulid",
"policyVersion": "2025.11.24",
"artifactDigest": "sha256:...",
"purl": "pkg:maven/org.example/foo@1.2.3",
"ruleId": "RULE-1234",
"severity": "high",
"state": "open",
"explainSummary": {
"hitRules": ["RULE-1234"],
"traceSampleId": "ulid",
"rationale": ["package matches advisory CVE-2025-1234"]
},
"provenance": {
"evaluationTimestamp": "2025-11-28T00:00:00Z",
"effectiveFindingHash": "be...",
"source": "materialized"
}
}
],
"cursor": { "next": "b64...", "prev": "b64..." },
"aggregates": {
"countsBySeverity": {"critical": 1, "high": 5, "medium": 12, "low": 3, "unknown": 0},
"countsByRule": [{"ruleId": "RULE-1234", "count": 4}],
"countsByPolicyVersion": [{"policyVersion": "2025.11.24", "count": 25}]
}
}
```
### 2) Finding explain trace (summary)
- **GET** `/policy/console/findings/{findingId}/explain`
- Returns deterministic trace summary for UI drawer (no full trace fan-out): hit rules, key facts, sampled trace token, policyVersion, evaluationTimestamp, hashes.
- Optional `format` (`json` default, `markdown` for UI preview); output ordering stable.
### 3) Simulation/export diff (used by POLICY-CONSOLE-23-002)
- **POST** `/policy/console/simulations/diff`
- **Body**
```json
{
"baselinePolicyVersion": "2025.11.24",
"candidatePolicyVersion": "2025.12.02",
"artifactScope": [{"artifactDigest": "sha256:..."}],
"budget": {"maxFindings": 2000, "maxExplainSamples": 50},
"filters": {"severityBand": ["high","critical"]}
}
```
- **Response**
```json
{
"schemaVersion": "console-policy-23-001",
"summary": {
"before": {"total": 120, "severity": {"critical":4,"high":30,"medium":60,"low":26}},
"after": {"total": 98, "severity": {"critical":3,"high":22,"medium":55,"low":18}},
"delta": {"added":12,"removed":34,"regressed":2}
},
"ruleImpact": [
{"ruleId":"RULE-1234","added":3,"removed":10,"severityShift":{"high→medium":6}},
{"ruleId":"RULE-2000","added":1,"removed":0}
],
"samples": {
"explain": ["trace-token-1","trace-token-2"],
"findings": ["finding-ulid-1","finding-ulid-2"]
},
"provenance": {
"baselinePolicyVersion": "2025.11.24",
"candidatePolicyVersion": "2025.12.02",
"evaluationTimestamp": "2025-12-02T00:00:00Z"
}
}
```
- Ordering of ruleImpact array: `ruleId ASC`; samples ordered by hash.
### 4) Bulk export (NDJSON)
- **POST** `/policy/console/findings/export`
- Body accepts same filters as list endpoint plus `format` (`ndjson` only) and `maxRows` (hard cap 50k).
- Response streams NDJSON of finding records in deterministic ordering with content hashes.
## Error Model
- 400 with machine-readable code (`invalid_filter`, `unsupported_schemaVersion`, `budget_exceeded`).
- 401/403 for auth/scope failures; 409 when `schemaVersion` mismatch.
- 429 when budget limits tripped; include `retryAfterSeconds` but never implicit sleep in server.
## Non-Goals
- No mutable state or approvals exposed here; status transitions remain in Console backend via existing endpoints.
- No live wall-clock filtering; clients must pass explicit windows.
## Testing Hooks
- Provide `X-Dry-Run: true` to validate filters and budgets without executing evaluation.
- `X-Debug-Sampling: <0..1>` allowed in non-production tenants only; otherwise rejected.
## Implementation Notes
- Reuse batch evaluation pipeline for simulation diff; reuse materialized `effective_finding_*` collections for listing/export.
- Enforce deterministic `evaluationTimestamp` supplied by caller; reject missing timestamp when `baselinePolicyVersion != candidatePolicyVersion`.
- All aggregates computed in-memory over deterministically ordered result sets; no sampling unless explicitly requested.

View File

@@ -14,11 +14,19 @@ Planned Evidence Locker paths (to fill post-signing):
- `evidence-locker/signals/heuristics/2025-12-01/fixtures/` (golden inputs/outputs)
Pending steps:
1) Sign each artifact with its predicate:
1) Sign each artifact with its predicate (cosign v3.0.2 in `/usr/local/bin`, use `--bundle`; v2.6.0 fallback in `tools/cosign` also works with `--output-signature`):
- `stella.ops/confidenceDecayConfig@v1`
- `stella.ops/unknownsScoringManifest@v1`
- `stella.ops/heuristicCatalog@v1`
Example (replace KEY):
Example (v3, replace KEY):
```bash
cosign sign-blob \
--key cosign.key \
--predicate-type stella.ops/confidenceDecayConfig@v1 \
--bundle confidence_decay_config.sigstore.json \
decay/confidence_decay_config.yaml
```
v2.6.0 fallback (if PATH prefixed with `tools/cosign`):
```bash
cosign sign-blob \
--key cosign.key \
@@ -26,7 +34,9 @@ Pending steps:
--output-signature confidence_decay_config.dsse \
decay/confidence_decay_config.yaml
```
2) Attach SHA256 from `SHA256SUMS` in DSSE headers/annotations.
2) Record SHA256 from `SHA256SUMS` in DSSE annotations (or bundle metadata); keep canonical filenames:
- v3: `confidence_decay_config.sigstore.json`, `unknowns_scoring_manifest.sigstore.json`, `heuristics_catalog.sigstore.json`
- v2 fallback: `.dsse` signatures.
3) Place signed envelopes + checksums in the Evidence Locker paths above; update sprint tracker Delivery Tracker rows 57 and Decisions & Risks with the final URIs.
4) Add signer/approver IDs to the sprint Execution Log once signatures are complete.

View File

@@ -21,6 +21,12 @@ Public key copy: `docs/modules/zastava/kit/ed25519.pub`.
- `evidence-locker/zastava/2025-12-02/zastava-kit.tzst.dsse`
- `evidence-locker/zastava/2025-12-02/SHA256SUMS`
Local staging: all files above are present under `evidence-locker/zastava/2025-12-02/` in the repo root, ready for locker upload/mirroring.
## CI delivery note
- Locker upload in CI requires a write credential (e.g., `CI_EVIDENCE_LOCKER_TOKEN`) with access to the `evidence-locker/zastava/` namespace.
- If the secret is absent, perform a manual upload from the staged folder and record the locker URI in the sprint log.
## Signing template (Python, ed25519)
```bash
python - <<'PY'